Transformer optimisation is no longer only a question of buying a larger GPU. For teams building products in India, the real constraints are usually GPU availability, inference cost, latency on uneven networks, and the need to support multiple languages or domains. A good optimisation workflow measures these constraints first, then changes the model, data, or serving stack with evidence.
GitHub is useful because it puts code, configuration, benchmarks, issues, and deployment manifests in one place. Treat each repository as an engineering record—not just a collection of notebooks.
Start with a measurable baseline
Before changing the model, record a reproducible baseline. Pin the model revision, dataset version, tokenizer, software environment, hardware, and random seed. Measure more than validation accuracy:
- Quality: task-specific scores, calibration, hallucination rate, and performance by language or demographic group.
- Speed: tokens per second, time to first token, end-to-end latency, and throughput at realistic concurrency.
- Memory: peak GPU RAM, CPU RAM, model size, and KV-cache usage during generation.
- Cost: cost per 1,000 requests or tokens, including storage, orchestration, and idle capacity.
Keep benchmark scripts under version control and run them in CI for important pull requests. A smaller model is not an improvement if it saves memory but misses your latency target or degrades Marathi, Hindi, or other target-language performance.
Choose the smallest suitable starting model
Begin with a model whose architecture, licence, context window, and language coverage match the product. A compact encoder may be preferable for classification or search, while a decoder model is appropriate for generation. Avoid selecting by parameter count alone: tokenizer efficiency, attention implementation, and serving support can matter more.
For domain adaptation, first evaluate prompting and retrieval before full fine-tuning. When training is justified, use a parameter-efficient method such as LoRA or QLoRA and maintain a held-out test set that reflects real Indian usage, including code-switching, spelling variation, and low-resource language inputs. Our guide to fine-tuning LLMs on custom data covers dataset hygiene and evaluation decisions in greater depth.
Optimise the data and training loop
Training inefficiency often begins in the data pipeline. Deduplicate examples, remove corrupt records, group similar sequence lengths, and use dynamic padding. Bucketing by length reduces wasted computation from padding, particularly for conversational datasets.
Useful training techniques include:
- Mixed precision: Use BF16 where hardware supports it; use FP16 with loss scaling when necessary. Check for overflow and quality regressions rather than assuming faster is better.
- Gradient accumulation: Simulate a larger batch when GPU memory is limited, while tracking the effect on optimisation and wall-clock time.
- Gradient checkpointing: Recompute selected activations to reduce memory, accepting additional compute.
- Distributed training: Use data or fully sharded parallelism only after profiling a single device. Communication overhead can outweigh the benefit for small jobs.
- Efficient input delivery: Cache tokenised data, use multiple workers carefully, and monitor whether GPUs are waiting on storage or preprocessing.
Record effective batch size, learning-rate schedule, warm-up, sequence length, and number of tokens processed. These details make GitHub experiments reproducible and make regressions easier to diagnose.
Profile attention, memory, and inference
Use a profiler before applying optimisation techniques. Identify whether the bottleneck is attention, matrix multiplication, data movement, tokenisation, network overhead, or generation length. Inference performance is often dominated by long prompts and output tokens rather than model loading alone.
For decoder models, manage the KV cache and cap context according to the product requirement. Batch requests when throughput matters, but use continuous batching for variable-length traffic. Separate prefill and decode metrics: a model can have acceptable first-token latency but poor generation speed, or the reverse.
Modern kernels and runtimes can provide substantial gains. Evaluate PyTorch compiled execution, FlashAttention-compatible implementations, ONNX Runtime, TensorRT-LLM, or specialised CPU runtimes against your actual workload. Keep the original implementation available as a correctness reference, and test numerical differences after every runtime change.
Compress the model carefully
Compression should be driven by a target—such as a 30% latency reduction or a memory limit—not by a technique checklist.
- Quantisation: Compare INT8, weight-only INT4, and supported activation-aware methods. Test multilingual quality, long-context behaviour, and rare-token handling.
- Distillation: Train a smaller student using teacher logits, generated explanations where appropriate, or task labels. Distillation is especially useful when a large teacher is too expensive for production.
- Pruning: Prefer structured pruning when the serving stack can exploit removed heads, channels, or layers. Unstructured sparsity may reduce parameter count without reducing real latency.
- Vocabulary and sequence controls: Review tokenizer efficiency and maximum sequence length; these can reduce computation without altering core weights.
Save each compressed artefact with its calibration data, evaluation report, and compatible runtime version. Never replace the baseline checkpoint without preserving a rollback path.
Make GitHub the reproducibility layer
A production-ready repository should include a clear README, licence notes, environment lockfile, training and evaluation commands, model-card information, and a small smoke test. Store large checkpoints through an appropriate registry or Git LFS rather than committing them directly. Use GitHub Actions to run linting, unit tests, data-schema checks, and a lightweight inference benchmark.
Track experiments with structured configuration files instead of hidden notebook state. Issues and pull requests should state the hardware, model revision, dataset slice, metric change, and cost impact. Teams exploring open-source options can also review AI projects for beginners on GitHub to learn repository conventions and testing patterns.
For Indian teams, add language and deployment tests that reflect the intended users. Include Unicode normalisation, transliterated text, mixed English usage, regional names, and privacy-sensitive examples. If the model serves vision-language workloads, compare performance across scripts and image quality; resources on open-source vision-language models for Indian languages provide a useful starting point.
Validate deployment, not just the checkpoint
Benchmark the complete path from request ingress to response. Test cold starts, autoscaling, concurrency spikes, network transfer, batching, and failure recovery. A quantised model that fits on one GPU may still be impractical if loading takes minutes or if the serving container cannot scale economically.
For cloud deployments, define resource requests, health checks, timeouts, and observability from the beginning. Teams using Google Cloud can compare their serving design with this guide to deploying deep learning models on GKE. For local or edge use cases, measure CPU fallback, thermal limits, and offline behaviour rather than relying on desktop GPU results.
Monitor quality after release. Sample requests safely, track latency by input length, and create alerts for token spikes, error rates, and drift. Keep a canary version so a faster model can be rolled back when real-world quality falls.
A practical optimisation sequence
Use this order to avoid premature complexity:
1. Establish quality, latency, memory, and cost baselines.
2. Improve data cleaning, sequence packing, and input delivery.
3. Select a smaller or better-suited architecture.
4. Profile and optimise kernels, batching, and KV-cache use.
5. Apply mixed precision, quantisation, distillation, or pruning.
6. Re-run multilingual, safety, and regression evaluations.
7. Deploy with reproducible GitHub automation and monitor production metrics.
The best practice is not a single library or compression setting. It is a disciplined loop: measure, change one important variable, validate quality and systems performance, document the result, and keep a reversible release path.