Why LLM integration needs a different approach
Integrating LLMs into legacy software systems is not a matter of inserting a chatbot into an old application. It is an architecture and risk-management project: the model must work with ageing databases, undocumented business rules, batch jobs, identity systems, and interfaces that were never designed for probabilistic outputs.
The safest approach is to keep the system of record intact and introduce LLM capabilities at carefully controlled boundaries. Use the model for tasks such as document extraction, search, summarisation, drafting, classification, and assisted workflows—while leaving transactions, permissions, calculations, and final approvals to deterministic software.
For Indian organisations, this distinction matters in sectors such as banking, insurance, healthcare, government, logistics, and manufacturing, where auditability and continuity are as important as user experience.
Start with a system and use-case assessment
Before selecting a model, map the legacy application and rank possible use cases. Document:
- The application’s languages, frameworks, databases, queues, batch processes, and integration protocols.
- Where business rules live: source code, stored procedures, configuration files, spreadsheets, or staff knowledge.
- Data classifications, retention requirements, access controls, and cross-border processing constraints.
- Latency, uptime, volume, and peak-load requirements for each workflow.
- The cost of an incorrect answer and whether a human must approve the result.
Choose an initial use case with high operational value and bounded risk. Summarising support tickets, extracting fields from invoices, searching internal manuals, or drafting responses is usually safer than allowing an LLM to approve payments or update customer records without review.
Define measurable acceptance criteria before building. Examples include extraction accuracy, citation coverage, response latency, escalation rate, cost per task, and reduction in manual handling time.
Use an integration layer, not direct model calls
Avoid connecting a legacy client or database directly to a model provider. Put an integration service between them. This layer should handle authentication, prompt construction, input validation, model routing, retries, rate limits, output schemas, logging, and policy enforcement.
A practical request flow is:
1. The legacy application sends a structured request to an internal API.
2. The integration service authenticates the caller and checks authorisation.
3. A retrieval component fetches only the records the user is allowed to see.
4. The service builds a versioned prompt and calls the selected model.
5. The model returns structured output, preferably validated against a schema.
6. The service records evidence, confidence signals, latency, and cost.
7. The legacy application displays the result or routes it for approval.
Teams building a small pilot can use patterns from integrating LLM APIs in Python web apps, then harden the service for production. For older systems that cannot consume modern REST APIs, use an adapter, message queue, scheduled export, or a small sidecar service rather than rewriting the core platform immediately.
Choose the right model and deployment pattern
Model selection should follow the task, not brand familiarity. Compare models on quality, context length, structured-output support, latency, availability, privacy controls, and total cost. A smaller model may be sufficient for classification or extraction, while a stronger model may be justified for complex reasoning or multilingual drafting.
Common deployment patterns include:
- Managed API: Fastest to launch, with less infrastructure ownership. Review data-use terms, regional availability, retention, and outage behaviour.
- Private cloud or dedicated endpoint: Offers stronger isolation and predictable capacity, but requires more operational expertise.
- Self-hosted open-weight model: Useful for sensitive workloads or offline environments, with added responsibility for GPUs, patching, monitoring, and model updates.
- Hybrid routing: Sends low-risk or routine tasks to a smaller model and escalates difficult cases to a stronger or privately hosted model.
For sensitive workloads, consider a local-first design and minimise what leaves the organisation. Mask personal data before inference, use short-lived credentials, and separate production data from experimentation.
Connect enterprise data with retrieval, not indiscriminate fine-tuning
Most legacy integrations need reliable access to current documents and records more than they need model retraining. Retrieval-augmented generation (RAG) can index approved manuals, policies, product data, or case histories and insert relevant passages into each request.
Build retrieval with access control in mind. Every document should carry ownership, department, geography, retention, and permission metadata. Filter before retrieval or immediately afterwards; do not rely on the model to enforce authorisation. Return citations or source identifiers so users can verify answers.
Fine-tuning can help with consistent formatting, classification, terminology, or style, but it does not reliably teach a model fast-changing facts. Follow best practices for fine-tuning LLMs on custom data, especially around representative datasets, held-out evaluation, redaction, and rollback.
Make legacy data and outputs dependable
Legacy systems commonly expose data through fixed-width files, SOAP services, COBOL copybooks, stored procedures, or inconsistent database fields. Create canonical schemas at the integration boundary and translate formats there. Do not force the model to infer undocumented field meanings.
Require structured responses for any downstream action. JSON schema validation, enumerations, type checks, range checks, and duplicate detection should happen outside the model. If validation fails, retry with a constrained prompt or send the item to a human queue. Never interpret free-form model text as an executable instruction.
Keep write operations separate from recommendations. A useful pattern is propose, validate, approve, commit: the LLM proposes a change, deterministic rules validate it, an authorised person or service approves it, and only then does the legacy system commit the transaction.
Security, privacy, and Indian compliance considerations
Treat prompts, retrieved documents, model outputs, and traces as potentially sensitive data. Establish controls for:
- Prompt injection and malicious content inside retrieved documents.
- Excessive permissions granted to tools or internal APIs.
- Sensitive-data leakage through prompts, logs, analytics, or vendor dashboards.
- Insecure deserialisation, poisoned documents, and unvalidated tool calls.
- Model denial-of-service, runaway token usage, and provider outages.
Apply least-privilege service accounts, encryption in transit and at rest, tenant isolation, secret rotation, network controls, and immutable audit logs. Align processing with applicable Indian requirements, including the Digital Personal Data Protection Act and sector-specific rules. Define retention and deletion policies before production launch, and involve legal, security, compliance, and domain owners early.
Evaluate before and after release
A demo is not an evaluation. Build a test set from real but appropriately redacted examples, including difficult cases, regional language variants, missing fields, adversarial inputs, and known failure modes. Measure both model quality and system behaviour:
- Accuracy, groundedness, citation correctness, and refusal quality.
- False approvals, missed escalations, and harmful or discriminatory outputs.
- Latency percentiles, availability, token consumption, and cost per workflow.
- Human override frequency and user satisfaction.
Use shadow mode first: generate outputs without affecting production decisions and compare them with human outcomes. Then release to a small group, add feature flags, maintain a rollback path, and expand only when metrics remain within agreed limits. Monitor drift as documents, policies, models, and user behaviour change.
A phased implementation plan
A practical rollout can follow four stages:
1. Discovery: Map dependencies, classify data, select one workflow, and define success metrics.
2. Pilot: Build an integration API, retrieval or extraction pipeline, evaluation set, and human review interface.
3. Controlled production: Add observability, cost limits, security testing, incident procedures, and a limited user cohort.
4. Scale: Introduce model routing, reusable connectors, governance reviews, and a catalogue of approved prompts and tools.
Keep the legacy core stable wherever possible. Modernise only the interfaces and services required for the selected workflow. If the programme later needs coordinated automation across several systems, study building distributed systems with AI agents carefully—but start with narrow, observable workflows rather than unrestricted agents.
What success looks like
A successful integration does not merely produce fluent answers. It preserves business continuity, respects permissions, exposes evidence, controls cost, and gives people a clear way to correct the system. In 2026, the strongest implementations treat LLMs as governed capabilities around the legacy platform—not as replacements for the deterministic systems that run the business.
For Indian founders building secure AI products around existing enterprise infrastructure, AI Grants India offers funding, mentorship, and ecosystem support for turning validated pilots into scalable solutions.