Small language models are attractive because they can be cheaper to train, easier to deploy, and more controllable than frontier systems. But “small” is not a compute specification. A 50-million-parameter encoder fine-tuned for classification is a very different project from pretraining a 1-billion-parameter decoder on billions of tokens.
The right estimate depends on parameter count, training tokens, sequence length, objective, precision, hardware utilisation, and the number of experiments. This guide gives a practical way to estimate those requirements in 2026, with ranges relevant to Indian researchers, startups, and student teams.
Start by defining the project
Before choosing a GPU, clarify which of these jobs you actually need to perform:
- Inference: running an already-trained model for predictions or generation.
- Fine-tuning: adapting an open model to a domain, language, or task.
- Continued pretraining: teaching a model more domain or language-specific text.
- Pretraining from scratch: learning the model’s weights from randomly initialised parameters.
Most small teams should begin with fine-tuning or parameter-efficient adaptation. Pretraining from scratch makes sense when you need a permissive licence, a specialised vocabulary, strong control over data, or better coverage of an under-represented Indian language. For Indic projects, data quality and text normalisation often matter more than adding another GPU; the low-resource Indic NLP builder’s guide is a useful companion for planning that work.
A practical compute estimate
For decoder-only language models, a common first-order estimate for training compute is:
Training FLOPs ≈ 6 × parameters × training tokens
This is an approximation, not a quotation. It excludes or undercounts data preparation, evaluation, checkpointing, failed runs, hyperparameter sweeps, and inefficient hardware utilisation.
Illustrative estimates:
| Model size | Training tokens | Approx. training compute | Practical interpretation |
|---|---:|---:|---|
| 50M parameters | 1B | 0.3 exaFLOPs | Feasible for a small research cluster or rented GPU time |
| 125M parameters | 2B | 1.5 exaFLOPs | A serious multi-day project on one high-end GPU or a short multi-GPU run |
| 350M parameters | 5B | 10.5 exaFLOPs | Requires careful distributed training and a meaningful budget |
| 1B parameters | 10B | 60 exaFLOPs | No longer a casual single-GPU experiment |
Compute-efficient training also follows a data trade-off: a small model trained on too few tokens will often underperform, while excessive data can waste money once validation loss stops improving. Use a held-out validation set and track loss per token rather than relying only on total training steps.
These figures describe pretraining, not fine-tuning. A 100M-parameter model may be fine-tuned on a few million tokens using a single 16–24 GB GPU. With LoRA or QLoRA, adapting a much larger base model may also fit on one consumer or cloud GPU, depending on sequence length and batch size.
Hardware: what is enough?
CPU-only work
CPUs are suitable for tokenisation, cleaning, evaluation, small classical language models, and very small prototypes. They are generally a poor choice for transformer pretraining beyond toy datasets. CPU fine-tuning can work for compact encoders, but expect long runtimes and less iteration speed.
One GPU
A single 16 GB GPU can handle many small encoder models, lightweight decoder models, and parameter-efficient fine-tuning jobs. A 24 GB card gives more room for longer context, larger batches, and experimentation. Use gradient accumulation when the desired effective batch does not fit in memory.
Multi-GPU setups
Multiple GPUs become useful when the model, context window, or batch cannot fit on one device, or when reducing wall-clock time is important. They also introduce communication overhead, storage bandwidth requirements, distributed-training complexity, and potentially higher cloud costs. For a first build, one well-utilised GPU is often better than four poorly configured GPUs.
Memory is not determined by parameter count alone. Training typically needs space for weights, gradients, optimiser states, activations, and temporary buffers. Adam-style optimisers can require several times the model’s raw weight memory. Mixed precision, activation checkpointing, FlashAttention-style kernels, and 8-bit optimisers can reduce the footprint, but each has trade-offs.
Inference is a separate budget
A model that is affordable to train may still be expensive to serve at scale. Estimate inference using:
- Weight memory: roughly parameter count multiplied by bytes per parameter.
- KV cache: grows with context length, batch size, and generated tokens for decoder models.
- Throughput target: requests per second or tokens per second.
- Latency target: especially important for voice, search, and interactive assistants.
- Quantisation: 8-bit or 4-bit weights can make local and edge deployment practical.
For example, a 1B-parameter model stored in 4-bit format needs roughly 0.5 GB for raw weights, but the complete runtime needs additional memory for the framework, activations, cache, and batching. Benchmark the actual serving stack rather than budgeting from weight size alone. If the model must run on a phone or low-cost device, review this 2026 guide to AI model optimisation for mobile devices.
How to reduce compute without weakening the project
- Use an existing base model: Fine-tuning is usually cheaper than pretraining.
- Start with LoRA or QLoRA: Train a small set of adapter parameters before updating all weights.
- Deduplicate and filter data: Low-quality or repeated text wastes tokens and can damage generalisation.
- Pack sequences efficiently: Reduce padding by grouping examples of similar lengths.
- Use mixed precision: BF16 is often a robust choice on supported accelerators; FP16 may require loss scaling.
- Checkpoint selectively: Frequent checkpoints improve recovery but increase storage and I/O overhead.
- Run cheap evaluations early: Stop configurations that fail on a small validation slice.
- Distil after proving value: A compact student model is easier to deploy once the target behaviour is understood.
- Track utilisation and cost: GPU time lost to data loading, CPU bottlenecks, or network storage is paid-for compute.
For teams building multimodal or regional-language systems, compute planning should include data licensing, annotation, and evaluation. A smaller model with carefully curated Marathi, Tamil, Bengali, or Hindi data may be more useful than a larger English-centric model. If your roadmap includes multimodal inputs, compare the constraints with open-source vision-language models for Indian languages.
A sensible 2026 project plan
Use staged experiments instead of committing immediately to a large run:
1. Prototype: Train or fine-tune on 1–5% of the dataset using one GPU.
2. Validate the data: Check language balance, duplicates, leakage, toxicity, and licence terms.
3. Measure scaling: Compare loss and task metrics as tokens and model size increase.
4. Run a pilot: Estimate tokens per second, GPU utilisation, checkpoint size, and cost.
5. Scale only after evidence: Add GPUs when the pilot shows a real quality or schedule benefit.
6. Reserve evaluation budget: Include human review and domain-specific tests in the plan.
For an Indian startup, the cheapest option is not always the lowest hourly GPU price. Include data transfer, storage, taxes, idle time, engineering effort, and the cost of repeating a failed run. Students and early builders can also use machine learning project ideas for computer science students to scope a meaningful prototype before attempting full pretraining.
Bottom line
For a genuinely small language model, one capable GPU is often enough for fine-tuning and early experiments. Pretraining from scratch can range from a manageable single-GPU research project for tens of millions of parameters to a multi-GPU engineering project once you reach hundreds of millions or billions of parameters. Estimate from tokens and parameters, then add a realistic allowance for experiments, evaluation, and inefficiency.
The most reliable strategy is to prove the data and task on a small run, measure real throughput, and scale only when the validation results justify it.