Why LLM platforms need a different architecture
Building scalable LLM applications in India is not simply a matter of connecting a chatbot to a model API. Production systems must manage variable traffic, long-running generation requests, retrieval quality, model failures, sensitive data, and inference costs. They also need to work across English, Indian languages, code-mixed queries, uneven connectivity, and industry-specific workflows.
The strongest approach is to treat the LLM as one service inside a broader application platform. Keep business logic, identity, permissions, retrieval, observability, and data storage under your control. This reduces vendor lock-in and makes it easier to change models as quality, pricing, and availability evolve.
Before writing code, define the job the system must perform. A support copilot, document-review tool, voice agent, and autonomous workflow have different latency, accuracy, and safety requirements. A narrow, measurable use case is usually easier to scale than a general-purpose assistant.
Start with a measurable product contract
Write down the operating targets for the first production release:
- Users and traffic: expected daily active users, peak requests per second, and concurrency during business or exam hours.
- Latency: time to first token and total response time for short and long requests.
- Quality: grounded-answer rate, task completion, citation accuracy, refusal quality, and escalation rate.
- Cost: maximum cost per interaction, tenant, or completed workflow.
- Availability: acceptable downtime and behaviour when a model, vector database, or external API fails.
- Data boundaries: what can be stored, where it can be processed, and how long it should be retained.
These targets guide model selection and infrastructure decisions. For example, a low-latency FAQ assistant may use a small model with retrieval and caching, while a legal review workflow may justify a slower model, human approval, and stronger audit trails.
Use a modular reference architecture
A practical platform usually includes the following layers:
- Client and API layer: web, mobile, WhatsApp, or voice interfaces connected through authenticated APIs.
- Application layer: tenant management, workflow orchestration, prompt templates, permissions, billing, and human hand-off.
- Model gateway: one internal interface for multiple providers and open-weight models, with routing, retries, quotas, and fallbacks.
- Knowledge layer: document ingestion, parsing, chunking, embeddings, metadata filters, hybrid search, and reranking.
- Data layer: transactional storage for application state, object storage for files, and separate logs for traces and evaluations.
- Operations layer: metrics, tracing, cost dashboards, alerting, red-team tests, and release controls.
Separating these layers makes it possible to replace a model without rewriting the product. It also prevents provider-specific SDKs from spreading through every service. Teams scaling beyond an early prototype should plan their backend infrastructure for AI applications around queues, autoscaling, rate limits, and graceful degradation rather than adding compute reactively.
Choose models for the workload, not the benchmark
Compare models using your own representative tasks. Evaluate answer quality, Indian-language performance, structured output reliability, context-window needs, tool use, latency, and total cost. A larger model is not automatically better if retrieval is weak or the workflow needs predictable JSON.
A model gateway should support:
- Routing: send simple classification and extraction tasks to smaller models and complex reasoning to stronger ones.
- Fallbacks: switch providers or regions when quotas, outages, or latency thresholds are breached.
- Versioning: record the model, prompt, retrieval sources, and tool calls for every important response.
- Budget controls: enforce per-user, per-tenant, and per-workflow token limits.
- Caching: reuse safe, deterministic results for repeated questions and expensive retrieval operations.
Open-weight models can improve control and data residency, but self-hosting adds GPU procurement, serving, patching, capacity planning, and evaluation work. Managed APIs may be the better starting point for a small team. Reassess this trade-off when volume, privacy, or unit economics change.
Design retrieval and data pipelines carefully
Most enterprise LLM failures originate in the knowledge pipeline, not the model. Build ingestion as a repeatable process: identify source ownership, extract text and tables, preserve document structure, remove duplicates, attach permissions, and record source versions.
Use metadata such as language, department, geography, effective date, and access level. Apply permission filters before retrieval, not after generation. Hybrid search—combining keyword and vector retrieval—often works better than embeddings alone for Indian names, product codes, legal clauses, and mixed-language queries. Reranking can improve relevance when the candidate set is large.
For multilingual products, test transliteration, spelling variation, regional vocabulary, and code-mixing explicitly. Do not assume that an English benchmark predicts performance in Hindi, Tamil, Bengali, Marathi, or other target languages. Teams building local-language products can also study high-performance AI applications with open-source tools for practical approaches to model serving and optimization.
Make reliability and cost visible
LLM traffic is bursty and responses are variable in length. Put generation jobs behind queues where possible, stream responses to the client, and use timeouts for every external dependency. Add circuit breakers, idempotency keys, backpressure, and dead-letter queues for failed jobs. Maintain a degraded mode—for example, search-only answers or human escalation—when generation is unavailable.
Track more than uptime. Monitor:
- p50, p95, and p99 time to first token and completion latency;
- input and output tokens by model and tenant;
- retrieval hit rate and citation coverage;
- timeout, refusal, and fallback rates;
- hallucination and task-failure samples from human review; and
- infrastructure cost per successful workflow.
Batch embeddings, trim unnecessary context, summarise long histories, and cap output length. Store full prompts and responses only when justified; redact personal and financial information from logs. Cost dashboards should be visible to product and engineering teams, not only finance.
Build security, privacy, and governance in
Indian businesses often handle identity documents, health records, financial information, education data, or proprietary company material. Classify data before sending it to a model. Use encryption in transit and at rest, tenant isolation, role-based access, secrets management, and retention policies. Log administrative actions and model changes for auditability.
Treat retrieved documents and user input as untrusted content. Defend against prompt injection, data exfiltration, unsafe tool calls, and indirect instructions embedded in files. Restrict tools with explicit schemas and permissions; never let a model directly execute arbitrary database queries or shell commands.
For high-impact decisions, keep a human review step and provide an explanation of the source material used. Run adversarial tests before each major release, including multilingual and code-mixed cases. If the product uses voice, evaluate the full pipeline—speech recognition, intent detection, model response, and speech synthesis. The architecture lessons in telephony infrastructure for scalable voice agents are relevant when building call-centre or outbound systems.
Deploy in stages
A sensible path from prototype to platform is:
1. Prototype: prove one workflow with a small evaluation set and clear human review.
2. Pilot: add authentication, tenant separation, feedback capture, quotas, and trace logging.
3. Production: introduce model routing, queues, autoscaling, incident response, security testing, and cost controls.
4. Scale: standardise the model gateway, automate evaluations, add regional capacity, and publish service-level objectives.
Use feature flags and canary releases for prompt, retrieval, and model changes. Maintain a golden test set drawn from real, consented examples. Evaluate every release against quality, latency, cost, safety, and language coverage—not just a single accuracy score.
For teams considering agentic workflows, start with bounded tools and observable state transitions. Multi-agent designs can help with complex work, but they also multiply latency, cost, and failure modes. Patterns from building distributed systems with AI agents are useful only when each agent has a clear responsibility and the workflow can recover from partial failure.
A practical checklist for Indian builders
Before launch, confirm that you can answer these questions:
- Which users, languages, and workflows are supported—and which are explicitly out of scope?
- What happens when the model is wrong, unavailable, or over budget?
- Can every answer be traced to a prompt version, model version, and source document?
- Are sensitive fields redacted, access-controlled, and covered by retention rules?
- Do you have load tests that reflect Indian peak patterns and network conditions?
- Can a product manager see quality and cost without querying raw logs?
- Is there a human escalation path for ambiguous or high-risk requests?
The winning platform is rarely the one with the most elaborate model stack. It is the one that delivers dependable outcomes at a sustainable unit cost, while adapting to India’s languages, users, regulations, and infrastructure constraints.