LLM applications fail under load for reasons that ordinary API designs often miss. A single request may consume a few hundred tokens or an entire long context, inference time varies by model, and a retry can multiply demand precisely when a provider is throttling you. For Indian startups, these problems are compounded by uneven traffic, INR-denominated budgets, regional latency, and dependence on multiple model providers.
The right goal is not to reject as many requests as possible. It is to protect availability, predictable latency, cost, and fairness while preserving a useful experience for legitimate users.
What rate limiting must control
Traditional APIs commonly limit requests per second. LLM gateways need a broader policy:
- Requests per minute (RPM): Controls call frequency and protects connection, queue, and orchestration layers.
- Tokens per minute (TPM): Controls the actual model workload, including input and generated output tokens.
- Concurrent requests: Prevents too many long-running generations from occupying workers.
- Daily or monthly spend: Protects a startup from accidental loops, abuse, and budget overruns.
- Tenant and user quotas: Stops one customer, API key, IP address, or workflow from starving others.
A request limit without a token limit is easy to evade: ten large prompts can consume more capacity than hundreds of short ones. Conversely, a token-only policy can allow excessive connection churn. Use a layered policy and document which limit caused a rejection.
Teams building structured AI workflows should also treat tool calls, embeddings, reranking, and moderation as separate capacity classes. A useful AI-generated API specification guide can help define these contracts before implementation.
The core LLM rate limiting challenges
1. Token demand is unpredictable
Input size changes with conversation history, retrieved documents, system prompts, and tool results. Output length is also uncertain. A chatbot that appears to make one request may send a large context window and generate a long answer.
Estimate tokens before dispatch, reserve capacity, and reconcile the estimate with actual usage after completion. If the request exceeds a tenant’s remaining budget, return a clear response or offer a smaller model, shorter context, or streamed summary.
2. Provider limits are multidimensional
Model providers may enforce separate quotas by model, organisation, region, endpoint, and billing tier. A successful request can still be followed by a 429 response because another quota—such as TPM or concurrency—was exhausted.
Your gateway should read provider headers when available, maintain per-provider state, and distinguish:
- 429: temporary capacity or quota pressure;
- 401/403: authentication or permission failure;
- 400: invalid payload or unsupported model;
- 5xx and timeouts: provider or network instability.
Do not retry permanent errors. For temporary failures, use bounded exponential backoff with jitter and a maximum retry count.
3. Bursts are normal, not exceptional
Indian consumer products may see traffic spikes around exam deadlines, campaigns, cricket matches, salary days, or a new feature launch. A fixed per-minute limit can either waste capacity during quiet periods or create a wall during bursts.
A token-bucket limiter allows controlled bursts while enforcing an average rate. A leaky-bucket or queue smooths work more aggressively. Choose based on the product: interactive voice agents need low waiting time, while document processing can tolerate a queue. For voice products, compare architectural trade-offs in Vapi vs Retell for voice agent development.
4. Fairness is difficult in multi-tenant systems
A global limit is simple but unfair. One enterprise batch job can consume the budget needed by smaller tenants. Apply limits hierarchically:
1. Global platform ceiling.
2. Provider and model ceiling.
3. Organisation or paid-plan quota.
4. User, API key, and IP safeguards.
5. Workflow-specific concurrency and spend limits.
Use weighted fair queuing or reserved capacity for important tenants. Anonymous traffic should have tighter limits, stronger caching, and possibly a lower-cost model. Never rely on IP address alone: mobile networks, offices, and carrier-grade NAT can place many legitimate users behind one address.
5. Retries and agents amplify load
An agent may call a model several times, invoke tools, summarise history, and retry a failed step. A small user action can therefore create a request cascade. Track a request or trace ID across every downstream call, enforce a per-workflow budget, and set a maximum step count.
Retries must be centralised rather than independently implemented by every service. Otherwise, five layers may each retry three times, creating up to 243 attempts. Add idempotency keys for operations that trigger side effects.
A practical architecture for 2026
Put a rate-limiting gateway between clients and model providers. The gateway should authenticate requests, classify tenants, estimate token demand, check quotas, and place eligible work into a queue.
A robust flow looks like this:
1. Validate the API key, tenant, model, region, and payload size.
2. Apply cheap protections first: IP, key, and request-frequency limits.
3. Estimate input plus maximum output tokens.
4. Check global, tenant, model, concurrency, and spend budgets.
5. Admit immediately, queue, downgrade, or reject with a retry hint.
6. Dispatch using a provider-aware scheduler.
7. Record actual tokens, latency, status, cost, and retry count.
8. Refund unused reservations where the policy permits.
Keep the fast decision path in Redis or another low-latency store, but do not make it the only source of billing truth. Persist usage events to durable storage and reconcile them periodically.
Designing useful failure responses
A rate limit is part of your product interface. Return a consistent error body with an internal code, human-readable explanation, request ID, and Retry-After value when appropriate. For streaming responses, define what happens if a connection is interrupted and ensure partial output is not silently billed twice.
Offer graceful degradation instead of a blank error page:
- switch to a smaller or faster model;
- reduce retrieved context;
- disable optional tool calls;
- serve a cached answer for identical, safe queries;
- place batch work into an asynchronous job queue;
- show an honest estimated wait time.
Do not promise a retry will succeed at a precise time unless you control the capacity. Clients should implement exponential backoff with jitter and respect server guidance.
Observability and testing
Measure more than 429 counts. Build dashboards for:
- admitted, queued, rejected, and timed-out requests;
- input and output TPM by model and tenant;
- queue age and p50/p95/p99 latency;
- retry amplification and timeout rates;
- cost per successful task and per customer;
- quota utilisation and unused reserved capacity;
- fallback frequency and answer-quality impact.
Test with load profiles that reflect real usage: steady traffic, sudden bursts, long prompts, many concurrent streams, provider failures, Redis delays, and a noisy tenant. Run failure-injection tests before a public launch. If your application is built as a larger enterprise system, the guidance on enterprise AI app development platforms in India provides useful context for governance and deployment choices.
India-specific operating considerations
Set budgets in INR for internal reporting, while retaining the provider’s billing currency for reconciliation. Monitor cross-region data transfer, data-residency requirements, and latency between Indian users, your application region, and the model endpoint. A fallback provider may improve resilience, but routing prompts across providers can create privacy, contractual, and quality issues.
For regulated workloads, record which model handled each request, where data was processed, retention settings, and whether fallback occurred. Rate limiting should support auditability—not just protection from overload.
Implementation checklist
Before production, confirm that you have:
- RPM, TPM, concurrency, tenant, and spend limits;
- token estimation and post-request reconciliation;
- bounded retries with jitter and idempotency;
- queue priorities and fairness rules;
- model fallback and graceful degradation;
- standard 429 responses and
Retry-Afterhandling; - dashboards, alerts, and durable usage records;
- load, burst, dependency-failure, and abuse tests;
- documented quotas that customers can understand.
Rate limiting is successful when users experience a stable product, finance sees controlled spend, and engineers can explain every rejection. Treat it as a capacity-management system spanning models, queues, tenants, and budgets—not as a single middleware setting.