AI systems do not scale like ordinary CRUD applications. A web API can often add CPU replicas when traffic rises; an AI service may be constrained by GPU memory, model load time, token generation, queue depth, or data-transfer costs. The right architecture therefore separates business workflows from inference and treats latency, capacity, reliability, and model quality as one operating problem.
For Indian teams, the design must also account for uneven network quality, multilingual inputs, data-protection obligations, limited access to premium GPUs, and the economics of serving users across multiple cities. This guide explains how to build scalable microservices for AI without turning every feature into an independently deployed service.
Start with workload and service boundaries
Do not begin with Kubernetes. Begin by measuring the workload you need to support:
- Traffic: requests per second, peak bursts, daily volume, and concurrency.
- Latency: p50, p95, and p99 targets, including time to first token for streaming LLM responses.
- Payloads: image sizes, audio duration, document length, and expected output size.
- Model behaviour: GPU memory, cold-start time, context-window use, and average inference duration.
- Reliability: which operations can be retried, delayed, degraded, or abandoned.
A sensible first boundary usually includes an API or gateway service, an orchestration service, one or more inference services, and asynchronous workers. Keep authentication, billing, tenant limits, and workflow state out of the model container. This lets you scale CPU-heavy application logic separately from expensive GPU inference.
Do not split every preprocessing step into its own network hop. A small team may be better served by a modular monolith plus one dedicated inference service. Extract a separate service when it needs different hardware, release cadence, security controls, or scaling behaviour. This is especially relevant for agent workflows: the patterns in building distributed systems with AI agents can help when planning retries, tool calls, and durable state, but they should not be applied to a simple prediction endpoint.
Design the inference contract first
An inference service should expose a stable contract rather than leak framework-specific details. A useful request includes a model or capability identifier, tenant context, input reference, priority, deadline, and idempotency key. The response should include a request ID, model version, usage metrics, and a clear error category.
Support both synchronous and asynchronous modes:
- Use synchronous requests for short predictions with a strict latency target.
- Return a job ID for document processing, image generation, long audio, batch scoring, and multi-step agents.
- Stream partial output only when clients benefit from early results and your gateway can enforce timeouts and cancellation.
- Store large inputs and outputs in object storage; pass signed references through the queue instead of copying megabytes between services.
For voice products, inference is only one part of the path. Streaming audio, interruption handling, speech recognition, language-model calls, and text-to-speech require different latency budgets. The architecture discussed in real-time voice agents with fast barge-in is a useful reference for separating these real-time components.
Select a serving stack based on the model
There is no universally best serving framework. Match the tool to the model and team:
- vLLM is well suited to high-throughput LLM serving, continuous batching, and efficient KV-cache management.
- NVIDIA Triton is useful when serving multiple models or backends, including TensorRT, ONNX, PyTorch, and TensorFlow.
- BentoML helps package Python models and dependencies into deployable services with a developer-friendly workflow.
- Ray Serve fits composed pipelines and deployments that need flexible scaling across several model components.
- Managed endpoints can be appropriate for early validation, provided you track per-request cost, data handling, rate limits, and portability.
Benchmark the complete service, not only raw model tokens per second. Include tokenisation, preprocessing, network transfer, batching delay, post-processing, and serialization. Compare quality at each optimisation level: a faster quantized model that harms critical outputs is not a production saving.
Use queues and batching deliberately
AI workloads often arrive in bursts. A durable queue absorbs spikes and prevents a slow model from exhausting web-server threads. Kafka, RabbitMQ, cloud queues, or Redis-backed workers can all work; choose based on ordering, replay, delivery guarantees, operational skills, and expected volume.
Design jobs to be idempotent. Persist status transitions such as queued, running, succeeded, failed, and expired, and include retry counts and deadlines. Retries should not duplicate payments, notifications, or irreversible actions. Use a dead-letter queue for poison messages and expose cancellation where the underlying model supports it.
Dynamic batching combines compatible requests after a short wait. Tune the maximum batch size and batching window against p95 latency, GPU utilisation, and output quality. LLM serving needs additional controls for prompt length, generation limits, and KV-cache pressure. Image, audio, and document workloads may need separate queues because their resource profiles differ.
Make Kubernetes GPU-aware
Kubernetes is useful once you have repeatable deployments and more than one workload profile. Configure it intentionally:
- Use node labels, taints, and tolerations to keep GPU workloads on suitable nodes.
- Request GPU resources explicitly and reserve CPU and memory for tokenisation and preprocessing.
- Scale on queue depth, in-flight requests, latency, and GPU memory—not CPU alone.
- Use KEDA for event-driven workers and custom metrics for inference deployments.
- Use topology-aware scheduling and warm replicas where cold starts threaten the SLA.
- Consider MIG or time-sharing only after measuring interference and isolation requirements.
Separate model versions through distinct deployments or traffic routing rules. Canary a new checkpoint on a small percentage of requests, compare quality and latency, then expand gradually. Keep rollback to the previous image, model artefact, and configuration as one tested operation.
Reduce cost before adding GPUs
Optimisation should follow measurement. Quantisation, distillation, pruning, shorter prompts, output limits, and smaller task-specific models can reduce both memory and cost. Route requests by complexity: a compact model can handle classification or extraction while a larger model handles ambiguous cases.
Cache deterministic results using a request fingerprint that includes model version and relevant configuration. Semantic caching can help for repeated knowledge queries, but it needs similarity thresholds, tenant isolation, expiry, and a policy for sensitive data. Never let a cache return one customer’s result to another.
For Indian deployments, compare cloud GPUs with reserved capacity, regional providers, and colocated hardware using total cost: idle time, egress, storage, support, failover, and engineering effort. A cheaper GPU is not cheaper if it causes queue growth or frequent model reloads.
Build observability around user impact
Use traces across the gateway, queue, preprocessing, model server, and post-processing stages. Track:
- request success, timeout, cancellation, and retry rates;
- queue wait, batch wait, model execution, and end-to-end latency;
- GPU utilisation, VRAM pressure, OOM events, and model load time;
- tokens per second, input/output tokens, and cost per successful request;
- quality signals such as groundedness, extraction accuracy, human review, and drift.
Log request IDs and model versions, not raw prompts or documents by default. Redact personal data, define retention periods, restrict operator access, and encrypt data in transit and at rest. For systems processing Indian user data, map data flows and access controls to your legal and contractual requirements, including obligations under the DPDP framework.
A production path for Indian builders
A practical sequence is:
1. Build one modular application and one inference endpoint.
2. Add load tests using realistic payloads and peak traffic patterns.
3. Separate synchronous and asynchronous workloads.
4. Introduce a queue, idempotency, timeouts, and dead-letter handling.
5. Add model metrics, cost dashboards, and quality evaluation.
6. Move to GPU-aware orchestration only when utilisation or reliability justifies it.
7. Add regional failover, model fallbacks, and disaster-recovery tests.
If your product is a voice assistant, research tool, or agent platform, borrow specialised patterns rather than forcing all traffic through one generic model API. For example, AI research assistant architecture covers retrieval and tool orchestration concerns that should remain distinct from the core inference server.
FAQ
Should every AI feature be a microservice? No. Split services when independent scaling, ownership, deployment, or security requirements provide a clear benefit. Otherwise, modular code inside one deployable application is faster and cheaper.
Is serverless suitable for AI inference? It can handle lightweight preprocessing, webhooks, and queue triggers. Large models with long load times or GPU requirements usually need persistent workers or managed model endpoints.
What should be tested before launch? Test burst traffic, long inputs, concurrent tenants, GPU exhaustion, queue retries, model reloads, dependency failures, cancellation, rollback, and degraded-mode behaviour—not just average latency.
How should teams control GPU spend? Set per-tenant quotas, enforce deadlines and output limits, right-size replicas, scale from queue and latency metrics, and review cost per successful outcome weekly.
A scalable AI microservice is not defined by the number of containers. It is defined by predictable behaviour under load, transparent cost, recoverable failure, and measurable model quality. Start with the smallest architecture that isolates the expensive path, then add orchestration and optimisation in response to evidence.