A working LLM demo can hide the hardest engineering work. Production systems must deliver predictable latency, protect tenant data, survive traffic spikes, and keep cost per task within a viable margin. These demands make scalability challenges in large language model applications different from those in conventional web software: every request may consume substantial GPU memory, generate tokens sequentially, and trigger several additional services such as retrieval, moderation, tools, and logging.
For Indian builders, the constraints are sharper. GPU availability and bandwidth vary by region, users may switch between English and Indic languages, and price-sensitive customers punish inefficient inference. A scalable design therefore starts with workload measurement and model selection—not with automatically provisioning the largest available GPU.
Define scalability before choosing infrastructure
Start by writing down service-level targets for each user journey. A customer-support chatbot, batch document processor, and voice assistant have different requirements.
Track at least:
- Time to first token (TTFT): how quickly streaming begins.
- Time per output token: the generation speed after the first token.
- End-to-end latency: including retrieval, tool calls, moderation, and post-processing.
- Requests and tokens per second: the throughput your system sustains.
- Error, timeout, and cancellation rates: especially during traffic spikes.
- Cost per successful task: not merely cost per API call.
Load-test with realistic prompt lengths, output limits, concurrency, and language mixes. A benchmark using short English prompts will not expose the memory pressure caused by long Hindi, Tamil, or mixed-language conversations. For the wider service layer, pair inference metrics with the practices in Scaling Backend Infrastructure for AI Applications.
Inference latency and GPU utilisation
LLM generation is autoregressive: each new token depends on the preceding sequence. Prompt processing can be parallelised more effectively, but decoding remains sensitive to memory bandwidth and scheduling. Under load, requests queue, TTFT rises, and users experience a system that appears slow even when average GPU utilisation looks healthy.
Use continuous batching so new requests join an active batch as sequences finish. Separate prompt processing from decoding where the serving stack supports it, and stream output to improve perceived responsiveness. Set explicit maximum input and output tokens; unlimited generation is a reliability and cost risk.
Batching is not automatically beneficial. Large batches improve throughput but can increase TTFT and cause long requests to hold resources needed by short ones. Measure p50, p95, and p99 latency by workload class rather than relying on one global average. Admission control, request priorities, cancellation, and queue timeouts are essential when demand exceeds capacity.
VRAM, KV cache, and context windows
The model weights are only one part of inference memory. During generation, the key-value (KV) cache stores attention states for every active sequence. Memory use grows with context length, concurrent requests, layers, and model dimensions. A few long-context users can therefore crowd out many short requests.
Practical controls include:
- Enforcing per-route context and output limits.
- Using paged KV-cache management to reduce fragmentation.
- Evicting or compressing inactive conversation state.
- Prefix-caching repeated system prompts and document prefixes.
- Reserving capacity for high-priority or latency-sensitive traffic.
- Routing long-context jobs to an asynchronous queue.
Quantisation can reduce weight memory and enable cheaper hardware, but validate quality, tool use, multilingual performance, and numerical stability on your own evaluation set. Do not assume that a smaller checkpoint automatically lowers total cost: poor quality may increase retries, human review, and context length.
Control compute cost with routing and model tiers
The most effective optimisation is often avoiding an expensive generation. Classify requests first and route simple tasks—intent detection, extraction, formatting, and routine support—to a smaller model. Reserve a larger reasoning model for cases that genuinely need it. Cache deterministic results, embeddings, and reusable prompt prefixes where privacy and freshness permit.
Compare models using cost per successful outcome, including failed calls, retries, retrieval, storage, observability, and evaluation. Open-weight models can improve control and data residency, but self-hosting adds GPU operations, patching, capacity planning, and on-call responsibility. Managed APIs offer elastic capacity but introduce vendor pricing, rate limits, and model-change risk.
For constrained deployments, optimisation techniques such as quantisation, speculative decoding, and distillation are useful only after profiling. A distilled or fine-tuned smaller model may be the right choice for a narrow workflow. Builders working with Indian languages can also evaluate open-source small language models for Hindi and specialised fine-tuning approaches for Indian regional languages, rather than defaulting to a general-purpose frontier model.
RAG is a distributed system, not a plug-in
Retrieval-augmented generation reduces the need to place an entire corpus in the prompt, but it creates additional scaling paths. Ingestion, parsing, chunking, embedding, indexing, filtering, reranking, and generation must all remain reliable as data changes.
Separate ingestion from query serving. Build an idempotent pipeline with document versioning, deletion handling, retry queues, and freshness indicators. For search, benchmark approximate nearest-neighbour indexes against your actual corpus and filter patterns. Hybrid lexical-plus-vector retrieval often performs better for product codes, names, and legal or government terminology than vector search alone.
Tenant isolation must be enforced before generation. Apply authorization-aware filters at retrieval time, test them with adversarial cases, and avoid relying on the model to ignore documents it should never have received. Monitor empty-result rates, retrieval recall, reranker latency, citation correctness, and index freshness. For Indic use cases, invest in language-aware tokenisation, transliteration handling, and evaluation data; low-resource Indic NLP guidance covers the data and modelling implications.
Conversation state and workflow reliability
Do not resend an unlimited chat transcript on every turn. Store durable user facts separately from ephemeral conversation messages, summarise older turns with versioned summaries, and retrieve only relevant history. Treat summaries as fallible data: allow correction, retain provenance, and never let an automatically generated memory override explicit permissions.
Tool-using agents add another failure surface. Set budgets for steps, tokens, retries, and wall-clock time. Make tools idempotent where possible, require confirmation for irreversible actions, and persist workflow state so a worker can resume after failure. Use asynchronous jobs for document processing, bulk translation, and long-running analysis instead of holding an HTTP request open.
Observability, evaluation, and rollout safety
At scale, logs must explain both quality and infrastructure behaviour. Record model and prompt versions, token counts, cache hits, retrieval identifiers, tool calls, latency phases, and failure reasons—while redacting personal and confidential information. Track quality by language, customer segment, route, and model version rather than only overall averages.
Build a regression set from real, permissioned examples. Include multilingual queries, code-switching, ambiguous names, long documents, refusal cases, and adversarial retrieval tests. Automated judges can help triage, but combine them with deterministic checks, human review, pairwise comparisons, and task-specific metrics. A canary rollout, shadow traffic, and an immediate rollback path are safer than switching every user at once.
A practical 2026 scaling plan
1. Measure the current path: break latency into queue, prompt processing, retrieval, decoding, tools, and post-processing.
2. Set budgets: define token, latency, error, and cost limits for each workflow.
3. Reduce unnecessary work: tighten prompts, cap outputs, cache prefixes, and route simple tasks to smaller models.
4. Stabilise serving: add continuous batching, admission control, cancellation, and autoscaling based on queue depth and token throughput.
5. Harden retrieval: enforce tenant filters, version indexes, and test freshness and recall.
6. Validate quality: evaluate by language and workflow before increasing traffic.
7. Expand capacity gradually: use canaries and keep an asynchronous fallback for non-urgent jobs.
The goal is not maximum GPU utilisation at any cost. It is a dependable system that meets user-facing targets, protects data, and produces enough value per rupee to scale. AI Grants India supports founders building that kind of infrastructure; learn more about applying.