Quantization reduces the number of bits used to store and compute model values. A model trained in FP32 may run with FP16, INT8, or 4-bit weights and activations, lowering memory use and often improving inference speed. The trade-off is quantization error: predictions can change because continuous values are mapped to a smaller set of representable values.
There is no universal accuracy penalty. For a well-calibrated vision or speech model, INT8 post-training quantization may lose less than one percentage point. Some models show no measurable degradation, while sensitive models can lose several points. LLMs are more variable: 8-bit weight quantization is often close to the FP16 baseline, whereas 4-bit quantization may preserve task quality or cause substantial failures depending on the method, architecture, prompts, and evaluation set.
Practical accuracy ranges
Treat these figures as planning ranges, not guarantees:
- FP16 or BF16: Usually negligible quality loss compared with FP32, provided the hardware and kernels are stable.
- INT8 weights and activations: Often 0–1 percentage point loss for supported CNNs, transformers, and speech models after representative calibration; sensitive layers may perform worse.
- INT4 weight-only quantization: Frequently a small loss on general language tasks, but degradation can become visible on mathematical reasoning, code, long-context work, multilingual prompts, or small models.
- Aggressive low-bit formats: Can deliver major memory savings, but require layer-wise testing and may need quantization-aware training or selective higher precision.
Measure the change in percentage points, not relative percentages. A classifier moving from 90% to 89% has lost one percentage point, or 1.1% relative accuracy. For generative systems, exact-match accuracy alone is insufficient: track task success, factuality, refusal behaviour, latency, throughput, and output length.
What determines the loss?
The first variable is the model and task. Large, redundant models may tolerate low-bit weights better than compact models. Outlier-heavy layers, embedding tables, attention projections, normalisation components, and the first and final layers are common sources of sensitivity.
The second is what gets quantized. Weight-only quantization usually protects activations and is a practical starting point for LLM inference. Quantizing both weights and activations can produce greater speed and memory gains, but activation ranges must be represented accurately. KV-cache quantization can extend context within a fixed memory budget, yet it may affect long-context quality.
Calibration data matters. Static and post-training methods estimate ranges from examples, so a calibration set that does not represent Indian languages, accents, code-switching, domain terminology, or production input length can create misleadingly good benchmarks. A deployment serving Hindi-English queries should calibrate and test on that traffic profile—not only on English academic prompts. For an overview of the choices, see what is model quantization.
Finally, the quantization format and runtime matter. A format that is efficient in one accelerator or CPU kernel may be slower elsewhere. For transformer serving, compare formats using the target runtime rather than assuming that the smallest checkpoint is fastest. Which quantization format is best for Transformers? provides a useful decision framework.
How to measure accuracy loss correctly
Build a reproducible FP32 or BF16 baseline first, then evaluate quantized variants under identical conditions:
1. Freeze the model version, tokenizer, prompts, preprocessing, decoding settings, and random seeds.
2. Create a calibration set separate from the final test set. Include production-like difficulty, languages, lengths, and edge cases.
3. Evaluate the baseline and each quantized model on the same examples.
4. Report absolute score changes, confidence intervals where possible, latency, peak RAM or VRAM, model size, and energy or cost per request.
5. Inspect failures by category instead of relying on one aggregate score.
For classification, use accuracy alongside macro-F1, recall for minority classes, calibration error, and a confusion matrix. For speech systems, measure word error rate and character error rate by language, accent, noise condition, and code-switching; the guidance on benchmarking speech-to-text accuracy in India is particularly relevant. For LLMs, combine exact-match or pass-rate tests with structured-output validity, tool-call success, retrieval-grounded faithfulness, and human review of a stratified sample.
A useful acceptance rule is not “less than 1% loss” in the abstract. Define a business threshold—for example, no more than 0.5 percentage points on a safety-critical classifier, or no more than 2% lower code-test pass rate while meeting a latency target. If quality falls only for a small but important slice, reject the candidate or protect that slice with a higher-precision route.
Ways to reduce degradation
Start conservatively. Test FP16 or BF16, then INT8, before moving to 4-bit. For simpler deployments, post-training quantization can be sufficient. Dynamic quantization is useful when activation ranges vary significantly, while static quantization can offer better performance when calibration data is representative.
Use mixed precision rather than forcing every layer to the same bit width. Keep sensitive layers, embeddings, output heads, normalisation, or KV cache in FP16/INT8 while quantizing robust weight matrices more aggressively. Per-channel or group-wise scales often outperform one scale for an entire tensor because they handle different channel distributions more precisely.
If post-training methods fail, use quantization-aware training. QAT exposes the model to simulated rounding during training and lets weights adapt. Even a short, carefully controlled fine-tuning run can recover quality, but it needs representative data and strict checks against overfitting. For LLMs, compare methods such as GPTQ, AWQ, bitsandbytes-based inference, and runtime-native formats on the actual workload—not just perplexity. What is GPTQ quantization? and What is BitsAndBytes quantization? explain two common routes.
A deployment checklist for Indian AI teams
Before shipping a quantized model:
- Test English plus the Indian languages and code-switching patterns your product supports.
- Include noisy mobile audio, low-bandwidth conditions, regional names, addresses, and domain-specific vocabulary where relevant.
- Compare tail latency and peak memory under concurrent load, not only single-request speed.
- Log quantization configuration, calibration data version, runtime, kernel, hardware, and model commit.
- Add canary monitoring for quality regressions and a rollback path to the higher-precision model.
- Re-run the benchmark after changing the accelerator, inference engine, tokenizer, or context length.
The right answer to “how much accuracy is lost after quantization?” is therefore empirical: often little with INT8, potentially more with 4-bit, and highly dependent on the model and workload. Quantize in stages, measure on representative Indian production data, protect sensitive components with mixed precision, and choose the lowest precision that meets your quality, latency, and cost requirements.