Start with the workload, not the GPU
Affordable high throughput LLM infrastructure for startups begins with a workload model. Buying a powerful GPU before measuring traffic, context length, output length, and latency requirements is one of the fastest ways to inflate burn.
Record these metrics for at least a representative week:
- Requests per minute at average and peak load
- Input and output tokens per request
- Time to first token (TTFT) and time per output token
- Concurrent users and queue depth
- Model size, context window, and tool-calling frequency
- Availability, data-residency, and compliance requirements
Throughput is not simply requests per second. For LLM applications, track tokens per second per GPU, cost per million tokens, and p95 latency. A system that serves many short requests but struggles with long contexts may look efficient in a basic benchmark while producing poor economics in production.
This measurement discipline should sit alongside the wider practices in scalable machine learning infrastructure for developers, particularly around observability, deployment repeatability, and capacity planning.
Choose a model and serving mode that match demand
The cheapest GPU is the one that can serve the required model at the required quality and latency. Startups should test a small model, a quantised production candidate, and a larger fallback model against real business tasks. Generic benchmark scores are useful, but they do not reveal whether a model extracts Indian addresses correctly, handles code-mixed prompts, or follows your tool schema reliably.
Use a tiered serving strategy:
- Small models: classification, routing, extraction, summarisation, and routine support queries
- Medium models: general chat, document analysis, and tool use with moderate context
- Large models: difficult reasoning, escalation, or high-value workflows only
- External APIs: overflow capacity, rare tasks, or early validation before utilisation justifies dedicated GPUs
Route requests by complexity. A lightweight classifier or rules layer can send routine traffic to a smaller model and reserve expensive capacity for cases that benefit from it. For products involving sensitive records, combine this with the controls described in data veracity infrastructure for high stakes AI.
Select hardware by memory and utilisation
GPU selection should follow model memory, KV-cache requirements, and expected concurrency—not brand preference. Weight memory is only the starting point. Long prompts, large batches, and concurrent generations consume additional memory through the KV cache.
A practical starting framework is:
- L4 or comparable inference GPUs: Efficient for smaller and medium models, embeddings, extraction, and moderate chat workloads
- L40S-class GPUs: A strong option when more VRAM and throughput are needed without moving immediately to premium accelerators
- A100 or H100: Justified for large models, demanding latency targets, high concurrency, or workloads that can keep the card busy
- Consumer GPUs: Useful for development and controlled internal workloads, but less suitable for production because of reliability, support, cooling, and fleet-management constraints
- CPU inference: Appropriate for embeddings, reranking, batch jobs, and small quantised models where latency is not strict
Benchmark the complete serving configuration rather than comparing theoretical TFLOPS. Measure tokens per second, concurrent requests, p95 latency, startup time, failure recovery, and actual hourly cost. In India, compare Mumbai, Hyderabad, and other nearby regions with overseas GPU clouds. A cheaper hourly rate can be offset by cross-region network charges, data-transfer costs, or poor latency for Indian users.
Use modern inference engines
A basic Transformers generation loop is rarely suitable for a busy production endpoint. Engines such as vLLM, SGLang, and TensorRT-LLM improve utilisation through continuous batching, efficient KV-cache management, kernel optimisation, and request scheduling. vLLM is often the simplest starting point because it supports OpenAI-compatible APIs and a broad range of open models.
Key techniques include:
- Continuous batching: Admit new requests as others finish instead of waiting for an entire batch
- PagedAttention or equivalent cache management: Reduce memory fragmentation and support more concurrent sequences
- Quantisation: Use AWQ, GPTQ, FP8, or other tested formats to lower memory use and raise batch capacity
- Speculative decoding: Pair a small draft model with a larger target model when acceptance rates justify the added complexity
- Prefix caching: Reuse repeated system prompts, policy text, or document prefixes where supported
- Streaming: Improve perceived responsiveness, while still monitoring total generation time and connection overhead
Quantisation is not automatically safe. Test factual accuracy, structured-output validity, multilingual behaviour, tool calls, and refusal behaviour on your own evaluation set. Keep an unquantised or higher-precision fallback for cases where quality degradation affects revenue or safety.
For teams building the application layer around these components, building high performance AI applications with open source tools offers a useful complement to the serving discussion.
Build an elastic architecture
Separate the application gateway, request router, inference workers, model storage, and observability stack. Expose an OpenAI-compatible internal interface so that the product can switch between a dedicated GPU pool, a managed endpoint, and an external provider without rewriting business logic.
Use different capacity pools for different traffic patterns:
- Baseline pool: Reserved or committed GPUs for predictable daily demand
- Burst pool: On-demand or serverless capacity for campaigns and sudden peaks
- Batch pool: Spot instances for offline enrichment, evaluations, and backfills
- Fallback provider: A tested external endpoint for incidents or regional shortages
Autoscaling should consider queue depth, waiting time, active sequences, KV-cache utilisation, and tokens per second—not CPU utilisation alone. Scale-up time matters: downloading a multi-gigabyte model during a traffic spike is not a reliable failover plan. Pre-warm instances, cache model artefacts, and maintain a clear degradation mode such as a smaller model, delayed processing, or a bounded queue.
This is part of the broader problem covered in scaling backend infrastructure for AI applications, where inference is one component of a dependable product rather than the entire platform.
Control costs with unit economics
Create a cost dashboard that connects infrastructure to customer value. At minimum, report:
- Cost per million input and output tokens
- GPU utilisation and idle time
- Cost per successful workflow, not only per request
- Cache-hit rate and average prompt length
- Retries, failed generations, and fallbacks
- Gross margin by customer, plan, and model route
Reduce waste by trimming unnecessary context, summarising conversation history, caching stable results, limiting maximum output tokens, and moving embeddings or reranking to cheaper hardware. Do not optimise away quality blindly: a lower token bill is not a saving if it increases human review, support tickets, or failed transactions.
For India-focused startups, factor in GST treatment, foreign-exchange exposure, cross-border data transfers, egress, support contracts, and procurement lead times. Grants or shared compute programmes can help with experimentation, but production planning should assume that the startup will eventually pay for reliable capacity.
A practical production checklist
Before moving beyond an MVP, confirm that you can:
- Reproduce the serving environment from version-controlled configuration
- Benchmark each model and quantisation format on representative prompts
- Detect saturation before latency breaches the SLA
- Drain and replace unhealthy GPU workers safely
- Roll back models and engine versions independently
- Enforce per-tenant quotas, timeout limits, and maximum context sizes
- Keep audit logs without storing sensitive prompts unnecessarily
- Test Hindi, regional languages, code-mixed inputs, and Indian formats where relevant
Start with one well-instrumented model and a clear fallback. Add multi-GPU sharding, speculative decoding, or custom kernels only when measurements show that they improve unit economics. A modest fleet with high utilisation, predictable failure handling, and strong routing will usually outperform an elaborate platform built before demand is proven.
For voice products, inference capacity is only one dependency; latency and concurrency also depend on telephony and media pipelines. Review telephony infrastructure for scalable voice agents before committing to an architecture for high-volume calling.
FAQ
What is the most cost-effective GPU for an Indian AI startup?
There is no universal answer. L4-class GPUs often work well for smaller models, while L40S-class hardware can offer better value for higher-memory workloads. Compare measured tokens per rupee, not hourly price alone.
Should a startup self-host or use an API?
Use an API while traffic is uncertain or highly spiky. Self-host when baseline demand keeps dedicated hardware busy and the savings justify operations, monitoring, and reliability work. A hybrid design is usually the safest transition.
Is 4-bit quantisation suitable for production?
Often, but only after task-specific evaluation. Check quality, structured outputs, multilingual performance, and safety behaviour before routing all traffic to a quantised model.
How much utilisation is enough to buy dedicated GPUs?
Look at sustained utilisation and margin, not a single peak. Dedicated capacity becomes attractive when predictable baseline traffic keeps the fleet busy and API pricing exceeds the fully loaded cost of operating it.