Quantization is one of the most effective ways to make an AI model cheaper and faster to run. By representing weights and activations with fewer bits—such as INT8, INT4, or FP8—teams can reduce memory traffic, fit models on smaller devices, and increase throughput. But simply converting a model to a lower-precision format is not enough. Quantized model inference optimization requires a coordinated approach across calibration, operator support, runtime selection, hardware, and evaluation.
For Indian startups and research teams, the payoff is especially practical: lower cloud bills, more affordable on-device products, and better performance on constrained networks and hardware. The right workflow can support anything from a Hindi speech assistant to a computer-vision system deployed in factories, clinics, or field-service applications.
What quantization changes
A floating-point model commonly stores values in FP32 or FP16. Quantization maps those values to a smaller numerical range, often using integer arithmetic. A typical affine mapping is defined by a scale and zero point:
- Scale controls the relationship between the original and quantized ranges.
- Zero point represents the floating-point value mapped to integer zero.
- Per-tensor quantization uses one scale for an entire tensor.
- Per-channel quantization uses separate scales, usually for output channels in convolutional or linear layers.
INT8 is generally the safest starting point for production. INT4 can deliver larger memory and throughput gains for large language models, but it is more sensitive to outliers and hardware support. FP16 and BF16 are not integer quantization formats, yet they remain important mixed-precision options when INT8 or INT4 causes unacceptable quality loss.
The benefits are measurable:
- Smaller model files and lower RAM or VRAM requirements
- Reduced memory bandwidth and potentially lower latency
- Higher batch throughput on supported accelerators
- Lower energy use for edge and mobile inference
- Lower cost per request in cloud deployments
These gains are not automatic. A quantized model can be slower than its FP16 version if the runtime inserts frequent dequantization operations or falls back to inefficient kernels.
Choose the right quantization strategy
Post-training quantization
Post-training quantization (PTQ) is the quickest path for an already-trained model. Weight-only PTQ is common for language models because it reduces storage and bandwidth while keeping activations in a higher-precision format. Weight-and-activation quantization can produce greater speedups, particularly for CNNs and smaller transformer models.
PTQ depends heavily on a representative calibration set. For an Indian-language application, calibration data should reflect the target scripts, accents, code-switching, noisy audio, and regional vocabulary—not only clean English benchmark samples. Use a few hundred to a few thousand carefully selected examples, depending on model size and task complexity.
Quantization-aware training
Quantization-aware training (QAT) simulates quantization during training, allowing the model to adapt to rounding and clipping effects. Use QAT when PTQ causes a meaningful quality drop, especially in detection, segmentation, speech, OCR, and safety-critical classification. QAT needs additional training infrastructure and a reproducible dataset pipeline, but it can recover accuracy that calibration alone cannot.
Mixed precision and selective quantization
Do not treat every layer identically. Keep sensitive components—such as embeddings, attention projections, normalisation layers, output heads, or early vision layers—in FP16 or BF16 while quantizing robust layers to INT8 or INT4. This mixed-precision approach often provides a better accuracy-to-latency trade-off than an aggressively quantized full model.
For large language models, compare weight-only INT4, INT8, and FP8 against an FP16 baseline. For vision models, start with per-channel INT8 and inspect layers with high activation variance before considering lower precision.
A production workflow that works
1. Freeze a baseline. Record accuracy, task-specific quality, p50 and p95 latency, peak memory, throughput, power, and cost per request using the unquantized model.
2. Inspect the deployment target. Confirm whether the CPU, GPU, NPU, or accelerator supports the desired data type and kernels. Check operator coverage before converting the model.
3. Prepare calibration data. Match production traffic by language, class distribution, image conditions, audio quality, sequence length, and prompt style.
4. Quantize incrementally. Test FP16 or BF16 first, then INT8, and only then INT4 or more aggressive schemes. Keep an unquantized fallback.
5. Validate numerical behaviour. Compare layer outputs, activation ranges, saturation rates, and error distributions. Large deviations identify candidates for higher precision.
6. Compile for the actual runtime. Export to the format required by the serving stack, such as ONNX, TensorRT, TensorFlow Lite, or a mobile accelerator runtime. Avoid benchmarking an unoptimised reference implementation.
7. Benchmark realistic traffic. Test warm and cold starts, variable sequence lengths, concurrent requests, batch sizes, and memory pressure.
8. Monitor after release. Track quality drift, fallback rates, latency percentiles, device thermals, and user-visible failures.
Teams optimising models for phones should also read this AI model optimization guide for mobile devices, while teams serving models on Kubernetes can compare the workflow with deploying deep learning models on GKE.
Runtime and hardware considerations
The runtime determines whether theoretical compression becomes real-world performance. Use kernels that match the target accelerator and avoid unnecessary format conversions between operators. Fuse compatible operations—such as linear layers with bias or activation functions—when the compiler supports it. For autoregressive language models, measure time to first token separately from per-token decode latency; quantization may help decode more than prompt processing because decode is often memory-bandwidth bound.
For edge devices, measure sustained performance rather than a short peak benchmark. Thermal throttling, shared memory, battery state, and background processes can change results. On servers, test whether increased throughput creates a queueing penalty that harms p95 latency. A smaller model may permit larger batches, but excessive batching can make interactive applications feel slow.
Local deployment is another useful option when data residency, connectivity, or cost matters. The practical decisions involved in deploying large language models locally overlap closely with quantized model selection, runtime compatibility, storage, and update management.
Accuracy, safety, and evaluation
Accuracy should be measured at the task level, not inferred from file size or a single benchmark. Build an evaluation suite covering:
- Core quality metrics such as F1, exact match, BLEU, WER, mAP, or task-specific scores
- Indian languages, transliteration, code-mixing, names, addresses, and local entities where relevant
- Long-tail inputs and adversarial or malformed requests
- Calibration, abstention, and confidence behaviour
- Safety filters and refusal behaviour for generative systems
- Regression cases discovered in production
For language models, evaluate response quality and factuality at multiple sequence lengths. Quantization can amplify errors in long-context attention, rare tokens, and multilingual generation. For voice or vision systems, test noisy environments and lower-quality camera or microphone inputs rather than relying only on curated datasets. If the application uses a small Hindi model, compare quantized quality against relevant open-source small language models for Hindi, not only against an English-centric baseline.
Common failure modes
- Using an unrepresentative calibration set: The model works in testing but fails on real accents, scripts, or image conditions.
- Quantizing unsupported operators: Runtime fallbacks add conversions and erase the expected speedup.
- Optimising average latency only: p95 and p99 latency reveal queueing, thermal, and memory problems.
- Ignoring activation outliers: A few extreme values can severely damage INT8 or INT4 quality.
- Comparing different serving configurations: Keep batch size, prompt length, hardware, thread count, and sampling settings consistent.
- Removing the fallback path: Retain FP16 or FP32 execution for sensitive inputs and controlled rollback.
A decision framework for builders
Choose INT8 PTQ when you need a fast, low-risk deployment improvement and have good calibration data. Choose QAT when the model is sensitive or quality requirements are strict. Choose weight-only INT4 when memory capacity is the main constraint and the runtime has mature kernels. Choose mixed precision when a small number of layers dominate quality loss. In every case, select the format based on measured end-to-end performance—not the advertised bit width.
Quantized model inference optimization is ultimately an engineering discipline: define the target, measure a baseline, quantize selectively, compile for the hardware, and validate against real users. For Indian AI builders, that discipline can turn a model that is too expensive or slow to deploy into a viable product without sacrificing the language, domain, and safety performance that users depend on.
Apply for AI Grants India
If your team is building efficient AI infrastructure, multilingual models, edge applications, or lower-cost inference systems in India, explore AI Grants India for potential grant support and programme information.