Large language models do not have one fixed compute requirement. A small, quantised model running on a developer laptop, a fine-tuned model serving thousands of users, and a frontier-scale pre-training run are entirely different infrastructure problems. For Indian startups and research teams, the right question is not “How many GPUs do we need?” but what workload are we funding, what quality do we need, and how will usage grow?
This guide breaks down LLM compute needs for experimentation, fine-tuning, evaluation, and inference, with practical choices for teams operating under India-specific budget, availability, and data-residency constraints.
Start by defining the workload
Compute planning becomes clearer when the project is separated into four activities:
- Inference: Running a trained model to generate responses. This is usually the recurring production cost.
- Fine-tuning: Adapting an existing model to a domain, language, style, or task. Parameter-efficient methods can reduce the hardware requirement substantially.
- Pre-training: Teaching a model from raw text and other data. This requires large datasets, distributed training, high-speed networking, and serious capital.
- Evaluation and data processing: Cleaning corpora, generating embeddings, running benchmarks, and testing safety. These workloads are often overlooked in budgets.
Most Indian startups should begin with an existing open-weight or hosted model, establish product-market fit, and only then consider fine-tuning or pre-training. Teams building developer products can also compare their infrastructure choices with enterprise AI app development platforms in India before committing to a custom stack.
The main drivers of LLM compute needs
Model size and precision
A model’s parameter count is an important starting point, but it is not the whole story. A model with billions of parameters needs memory for weights, temporary activations, runtime buffers, and the key-value cache used during generation.
Reducing numerical precision can make deployment more affordable:
- FP32 is accurate but memory-intensive and rarely necessary for routine inference.
- FP16 or BF16 is common for training and higher-throughput inference.
- INT8 and INT4 quantisation can substantially reduce memory use, often with an acceptable quality trade-off for specialised applications.
Quantisation should be tested against representative Indian languages, code, domain terms, and long-context prompts. A lower-cost model that fails on production inputs may create greater support and rework costs than a larger model.
Context length and traffic
Long prompts increase memory and latency, particularly during prefill, when the system processes the input context. Long outputs increase decoding time. Concurrent users also increase the key-value cache requirement, even when the model weights remain unchanged.
Estimate at least these metrics before selecting hardware:
- Average and maximum input tokens per request
- Average and maximum output tokens
- Requests per second at peak
- Acceptable time to first token
- Acceptable time between generated tokens
- Number of simultaneous sessions
- Percentage of requests requiring long context
A chatbot with modest traffic may run comfortably on one accelerator, while a document-analysis service can become memory-bound because every request carries a large source file.
Hardware: GPU, CPU, or accelerator?
GPUs remain the default for serious LLM training and inference because they combine parallel compute with high memory bandwidth. The relevant specification is not only raw compute; available accelerator memory and memory bandwidth often determine whether a model fits and how quickly it runs.
- Consumer GPUs: Useful for prototyping, local evaluation, and smaller quantised models. They can offer strong value but may have limited VRAM and warranty support.
- Data-centre GPUs: Better suited to sustained workloads, multi-GPU training, reliability, and production serving.
- Cloud accelerators: Useful when demand is uncertain or capital is limited, though hourly pricing must be tracked carefully.
- CPUs: Suitable for preprocessing, orchestration, retrieval, small models, and low-volume inference. CPU-only serving is usually too slow for demanding interactive workloads.
For many teams, the practical path is hybrid: local or subsidised compute for development, rented accelerators for bursts, and a production endpoint selected around measured traffic rather than theoretical peak capacity.
Training and fine-tuning requirements
Pre-training a language model from scratch involves repeated passes over large datasets. The compute bill includes accelerator time, data storage, networking, experiment failures, checkpointing, and engineering support. Multi-node jobs need fast interconnects so that GPUs spend less time waiting for synchronisation.
Fine-tuning is more accessible. Full fine-tuning updates every parameter and can still require substantial memory. Parameter-efficient approaches such as LoRA and QLoRA update a smaller set of parameters, allowing many domain adaptations to run on a single high-memory GPU or a modest multi-GPU setup.
Before fine-tuning, establish a baseline using prompting and retrieval-augmented generation. If the problem is stale or inaccessible knowledge, better document retrieval may be more effective than changing model weights. If the problem is output format, tone, or a repeatable task, fine-tuning may be justified.
Inference architecture and cost control
Production inference is where compute costs become recurring operating expenses. A robust serving design usually includes:
- Continuous batching to combine compatible requests and improve accelerator utilisation
- Quantised models where quality testing supports them
- Paged or managed attention to control key-value cache memory
- Streaming responses to improve perceived latency
- Autoscaling based on queue depth, token throughput, and latency rather than CPU utilisation alone
- Caching for repeated prompts, embeddings, or deterministic responses
- Model routing so simple requests use smaller models and complex requests use stronger ones
Track cost per successful task, not merely cost per request. A low-cost model that requires retries, human review, or lengthy prompts may be less economical. Teams building voice products should also account for speech-to-text, text-to-speech, and real-time latency; the trade-offs are illustrated in Vapi vs Retell for voice agent development.
Storage, networking, and data operations
Compute is only one part of the system. Training datasets, checkpoints, tokenised files, evaluation results, logs, and vector indexes can quickly occupy terabytes.
Use fast local NVMe storage for active workloads and durable object storage for datasets and checkpoints. Keep multiple checkpoint versions only when they serve a clear rollback or comparison purpose. Data pipelines should validate duplicates, contaminated evaluation examples, personally identifiable information, and licensing restrictions before expensive training begins.
For distributed training, high-bandwidth, low-latency networking is essential. For inference, network placement matters: keeping the application, vector database, and model server close together can reduce latency and data-transfer charges. Indian teams handling sensitive enterprise or public-sector data should confirm where data is processed, logged, and backed up.
Choosing cloud, colocation, or institutional compute
Cloud infrastructure is attractive for fast experiments and variable demand. Set budgets, quotas, automatic shutdowns, and per-project tagging before launching large jobs. Reserved capacity can reduce cost for stable workloads, while spot or preemptible instances may suit fault-tolerant training.
Owned servers or colocation can be economical for predictable utilisation, but they require power, cooling, hardware maintenance, replacement planning, and security operations. Universities, incubators, and public programmes may offer shared capacity. Partnerships with research institutions can be especially valuable for evaluation and experimentation, not only for access to hardware.
When selecting a provider, compare:
- Accelerator model, VRAM, and availability
- Storage and interconnect performance
- Data-centre location and compliance controls
- Billing granularity and egress charges
- Support for the serving framework your team uses
- Ability to scale down without losing data
A practical planning sequence for Indian builders
1. Measure the workload: Collect token counts, concurrency, latency, and quality requirements from real or carefully simulated traffic.
2. Set a baseline: Test a hosted model and at least one open-weight model using the same evaluation set.
3. Optimise before scaling: Improve prompts, retrieval, batching, quantisation, and caching before adding accelerators.
4. Separate environments: Keep development, evaluation, and production budgets and credentials distinct.
5. Stress-test peak demand: Include long contexts, retries, outages, and traffic spikes.
6. Calculate unit economics: Report cost per user, document, conversation, or completed task.
7. Plan for fallback: Maintain a smaller model or asynchronous queue for capacity shortages and provider outages.
Students and early builders can practise these decisions through machine learning projects for computer science students, while teams working with multimodal data should separately benchmark vision and video workloads.
Common mistakes to avoid
- Buying hardware before measuring token demand
- Treating parameter count as the only sizing metric
- Ignoring key-value cache memory for long-context applications
- Training on unclean, duplicated, or poorly licensed data
- Leaving cloud instances running after experiments finish
- Selecting a model solely by benchmark score
- Failing to test latency and quality across Indian languages and accents
- Logging sensitive prompts and responses without a retention policy
Final takeaway
LLM compute needs should be managed as an engineering and finance problem together. Start with the smallest model and simplest serving setup that meets the quality target, measure real usage, and scale only where evidence supports it. For Indian startups, disciplined evaluation, parameter-efficient fine-tuning, quantisation, and careful cloud controls can make capable language systems viable without frontier-scale infrastructure.
If compute is the constraint holding back an AI research or product project, apply to AI Grants India to explore funding support for development, evaluation, and infrastructure.