Apply for AI Grants India

Start with the training objective

“LLM training” covers very different workloads. GPU needs for a 7B model fine-tuned on a domain corpus are not comparable to pretraining a foundation model from scratch. Define the job before choosing hardware:

• Inference or evaluation: usually the lightest requirement; quantisation can make a capable model usable on a single workstation or modest cloud instance.
• Supervised fine-tuning (SFT): updates a pretrained model on labelled examples. Full-parameter SFT is memory-intensive, while LoRA or QLoRA can reduce the hardware requirement substantially.
• Continued pretraining: adapts a base model to additional tokens, such as Indian legal, financial, technical, or multilingual data. This sits between fine-tuning and pretraining in cost and complexity.
• Pretraining from scratch: requires a coordinated multi-GPU cluster, high-speed networking, reliable storage, and an operations team.

For Indian teams, the right first step is often a small reproducible experiment rather than purchasing a large cluster. A documented open-source AI model training script can help establish tokens per second, memory headroom, checkpoint size, and expected convergence before scaling.

How to estimate GPU memory

GPU memory, or VRAM, is usually the first constraint. During training, memory holds more than model weights. A practical estimate includes:

• Weights: approximately 2 bytes per parameter in FP16 or BF16.
• Gradients: commonly another 2 bytes per parameter in mixed-precision training.
• Optimizer states: often 8 bytes or more per parameter with Adam-style optimisers, depending on precision and implementation.
• Activations: affected by sequence length, batch size, hidden dimensions, and checkpointing.
• Temporary buffers: required by attention kernels, communication, and the framework.

As a rough full-parameter training estimate, the model, gradients, and optimiser states can require 12–16 bytes per parameter before activations. A 7B model can therefore exceed 80–100GB in a conventional setup. Sharding, ZeRO or FSDP, activation checkpointing, 8-bit optimisers, and parameter-efficient fine-tuning can change this substantially, but they do not eliminate engineering trade-offs.

For a first experiment, a 24GB GPU may support QLoRA on a smaller model with a short context window. A 48GB or 80GB accelerator gives considerably more room for larger batches, longer sequences, full-parameter adapters, and debugging. Treat these as planning bands, not guarantees: framework overhead and dataset shape can move the requirement materially.

GPU classes and what they are good for

Select accelerators by memory, memory bandwidth, interconnect, software support, and hourly cost—not by core count alone.

• 24GB-class GPUs: useful for prototyping, quantised fine-tuning, evaluation, and smaller language models. They are a sensible entry point for an individual builder or early-stage startup.
• 40–48GB GPUs: suitable for more demanding adapter training and some full-parameter experiments with careful sharding and checkpointing.
• 80GB data-centre GPUs: the practical baseline for serious multi-GPU training. They reduce fragmentation, support larger context windows, and work well with mature distributed-training stacks.
• Newer high-memory accelerators: valuable when training throughput, memory capacity, and inter-GPU bandwidth justify their premium. Benchmark the exact workload rather than assuming a newer card automatically reduces total cost.

NVIDIA remains widely used because CUDA, NCCL, PyTorch, and vendor libraries are deeply integrated. AMD and other accelerators can be viable where ROCm or another software stack supports the exact model and kernels, but validate compatibility before committing. For teams planning longer-term infrastructure, research into energy-efficient AI training chips is relevant because electricity, cooling, and rack capacity increasingly shape the economics of training.

Compute, tokens, and scaling

GPU count should follow the amount of training compute, not the model label. A commonly used planning approximation for dense decoder-only pretraining is that training compute scales roughly with 6 × parameters × training tokens. This is an estimate, not a bill: architecture, sequence packing, utilisation, repeated data, and training schedule all affect the result.

Measure a pilot run and record:

• tokens processed per second per GPU;
• achieved utilisation and time spent waiting for communication;
• loss improvement per unit of compute;
• checkpoint and restart time; and
• total cost per billion training tokens.

A cluster that is twice as expensive but delivers three times the useful tokens per hour may be the cheaper option. Conversely, adding GPUs to a poorly tuned data pipeline can increase the bill without improving progress.

Distributed training: the hidden requirements

Large runs combine several forms of parallelism:

• Data parallelism replicates the model and divides batches across workers. It is straightforward until model states no longer fit on each device.
• Fully sharded data parallelism distributes parameters, gradients, and optimiser states to reduce per-GPU memory.
• Tensor parallelism splits matrix operations across GPUs and benefits from fast interconnects.
• Pipeline parallelism places model layers on stages, but can introduce pipeline bubbles and balancing challenges.
• Sequence or context parallelism helps with long-context workloads but adds implementation complexity.

For multi-node training, high-bandwidth networking is not optional. Weak interconnects can leave expensive GPUs idle during all-reduce and checkpoint operations. Plan for topology, NCCL or equivalent configuration, node failures, timeouts, and elastic recovery. Use a small cluster benchmark before signing a long reservation or building a private installation.

Storage, data, and reproducibility

Training performance depends on the pipeline feeding the GPUs. Keep tokenised data in formats that support sequential and distributed reads, use local NVMe caches where appropriate, and ensure the storage layer can serve every worker concurrently. Budget for multiple checkpoints, optimiser states, tokenizer files, logs, and evaluation artefacts; a run can need several times the final model size in storage.

Data quality is equally important. Deduplication, licensing, contamination checks, language balance, and PII controls can save more compute than an expensive accelerator upgrade. Teams working with Indian languages should consider low-resource language datasets for AI training in India, especially when deciding whether to train a multilingual model or continue-pretrain a strong base model.

Maintain dataset hashes, configuration files, code revisions, random seeds, and evaluation snapshots. If claims about data provenance matter, approaches such as cryptographic proof for AI training datasets may be worth evaluating for high-stakes or externally funded projects.

A practical sizing workflow for Indian teams

1. Define the target: model size, context length, languages, quality threshold, and deadline.
2. Choose the least expensive valid method: prompting, retrieval augmentation, SFT, QLoRA, continued pretraining, or scratch training.
3. Run a fixed pilot: use representative data and measure tokens per second, peak VRAM, throughput, and quality.
4. Project the full run: include failed experiments, evaluation, checkpoint storage, networking, and engineering time.
5. Compare procurement options: Indian cloud regions, GPU rental providers, research clusters, and owned hardware may have different availability, taxes, egress, and support terms.
6. Set stop conditions: stop scaling when validation quality plateaus or the marginal cost per quality gain becomes unjustifiable.

A grant-funded or early-stage team should generally preserve cash and iteration speed. Renting GPUs can be preferable until utilisation is predictable; owning hardware becomes more compelling with sustained workloads, stable software, and dependable power and cooling.

Common mistakes to avoid

• sizing from parameter count alone;
• ignoring sequence length and activation memory;
• buying GPUs without checking software and kernel support;
• underestimating interconnect and storage throughput;
• training on unverified or poorly deduplicated data;
• comparing cloud prices without including idle time, storage, egress, and failed runs; and
• assuming a larger model will outperform a smaller, better-trained model on Indian use cases.

Bottom line

The answer to llm training GPU needs is a measured system design, not a single GPU recommendation. Start with a representative pilot, use memory-saving methods where they preserve quality, benchmark distributed scaling, and price the complete pipeline. For most builders, a disciplined fine-tuning or continued-pretraining plan delivers more value than attempting foundation-model pretraining before data, evaluation, and operations are ready.

FAQ

Can I train an LLM on one GPU?
Yes, for small models, quantised fine-tuning, LoRA, QLoRA, and evaluation. Full pretraining of a modern foundation model is a multi-GPU workload.

Is 24GB VRAM enough?
It can be enough for adapter-based fine-tuning with quantisation and short sequences. It is rarely enough for comfortable full-parameter training of a multi-billion-parameter model.

How many GPUs do I need for a 7B model?
There is no universal number. One high-memory GPU may handle some adapter workloads; full-parameter training may require sharding across several GPUs, depending on precision, optimiser, batch size, and context length.

Should an Indian startup buy or rent GPUs?
Rent first when workload and utilisation are uncertain. Consider buying only after repeated benchmarks show sustained demand and you can operate power, cooling, scheduling, maintenance, and software reliably.

How can I reduce GPU costs?
Improve data quality, use parameter-efficient fine-tuning, pack sequences, apply mixed precision, checkpoint selectively, cache data locally, and benchmark utilisation before adding hardware.