Apply for AI Grants India

Quantization is one of the most effective ways to make an AI model fit and run reliably on a constrained device. It replaces high-precision values—usually FP32 weights and activations—with smaller representations such as FP16, INT8, or, for selected workloads, INT4. The result can be a smaller model, lower peak memory use, faster inference, and reduced battery consumption.

For Indian product teams, this matters across offline-first apps, affordable Android phones, point-of-sale devices, agricultural sensors, health tools, and edge deployments with intermittent connectivity. Quantization is not simply a file-compression step, however. The target runtime, processor, operators, model architecture, and accuracy tolerance determine whether the change delivers a real benefit.

Start with a deployment budget

Before converting the model, define measurable limits for the device:

• Peak RAM: Include model weights, runtime overhead, activation tensors, input buffers, and temporary workspaces—not only the model file size.
• Latency: Measure p50 and p95 inference time, preferably after warm-up and under realistic thermal conditions.
• Power and battery impact: A faster model may still be inefficient if it forces frequent CPU wake-ups.
• Accuracy and safety thresholds: Set task-specific limits for recall, precision, word error rate, calibration, or clinically relevant sensitivity.
• Runtime and hardware: Identify whether deployment uses TensorFlow Lite, ONNX Runtime, ExecuTorch, MediaPipe, NNAPI, Android GPU, or a vendor accelerator.

A model that is 70% smaller on disk may not reduce peak RAM by the same amount if activations remain FP32 or the runtime inserts conversion operations. Use the target hardware early, alongside a broader AI model optimization for mobile devices plan covering pruning, architecture selection, and profiling.

Choose the right quantization method

FP16 quantization

FP16 halves the storage required for weights compared with FP32 and is often a low-risk first step. It is useful on GPUs and mobile accelerators with native half-precision support. CPU speedups vary, and FP16 may not reduce activation memory as much as full INT8 quantization.

Dynamic-range or dynamic quantization

Dynamic quantization stores weights in lower precision but converts activations during inference. It is relatively easy to apply and is often effective for transformer or recurrent layers dominated by Linear and embedding operations. The runtime pays some conversion cost, so benchmark it rather than assuming it will be faster.

Static post-training INT8 quantization

Static quantization converts both weights and activations using scale and zero-point values estimated from a representative calibration dataset. It usually offers the best balance of memory and speed for CPU and edge inference, provided the device has optimized INT8 kernels.

The calibration set should resemble production traffic. For an Indian-language speech or text application, include the expected languages, scripts, accents, code-switching, noise levels, and device microphones—not just a convenient sample from training data.

Quantization-aware training

Quantization-aware training, or QAT, simulates low-precision operations during fine-tuning. Use it when post-training INT8 causes unacceptable accuracy loss, especially in sensitive vision layers, small models, detection heads, or language models with narrow error margins. QAT costs additional training time but often recovers accuracy more effectively than repeatedly changing calibration settings.

For LLMs, weight-only INT8 or INT4 methods can reduce memory substantially, but generation speed depends on matrix-kernel support and the runtime. If the goal is local inference, first review practical guidance on how to deploy large language models locally.

A practical workflow

1. Establish a floating-point baseline

Record the original model’s task quality, file size, peak resident memory, cold-start time, warm latency, throughput, and energy use. Save representative inputs and a fixed evaluation script so every quantized version is compared consistently.

2. Inspect unsupported operations

Quantization works best when the runtime has optimized kernels for the model’s operators. Unsupported layers can remain in FP32, creating mixed-precision execution and extra copies. Check operator coverage, tensor layouts, delegates, and fallback warnings before interpreting results.

3. Try the least disruptive conversion

Start with FP16 or dynamic quantization if you need a quick baseline. Move to full INT8 when RAM and latency targets require it. A simplified PyTorch example for supported modules is:

import torch
from torch.ao.quantization import quantize_dynamic

model = MyModel().eval()
quantized = quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8,
)
torch.save(quantized.state_dict(), "model-int8.pt")

This example is not a universal export path. For production, follow the quantization and export flow required by your chosen runtime, then verify that the generated artifact actually uses integer kernels on the device.

4. Calibrate with representative data

For static INT8, collect a small but diverse calibration set. It should cover short and long inputs, bright and dark images, common accents, background noise, and difficult edge cases. Poor calibration can produce saturation, weak activations, and large accuracy drops even when the dataset is technically valid.

5. Evaluate layer-level failures

Compare errors by class, language, demographic group, image condition, or sequence length. If only certain layers are sensitive, use selective mixed precision rather than reverting the entire model to FP32. Keeping a few layers at FP16 or FP32 can preserve quality while retaining most memory savings.

6. Benchmark on real hardware

Test cold start, repeated inference, concurrent workloads, background memory pressure, and thermal throttling. On Android, measure with the same OS version, delegate, and thread configuration expected in production. Also test installation size and model-load time, which matter for users with limited storage or slow networks.

Troubleshooting accuracy and memory problems

• Accuracy falls sharply: Improve calibration coverage, inspect activation ranges, exclude sensitive layers, or fine-tune with QAT.
• Latency does not improve: Check whether the runtime is falling back to FP32 or inserting frequent dequantization operations.
• Peak RAM remains high: Profile activations and temporary buffers; weight quantization alone may not solve the largest allocation.
• Results differ across devices: Confirm operator versions, delegate selection, thread counts, and numerical tolerances.
• The model becomes unstable: Watch for outlier channels and attention or normalization layers that are poorly suited to aggressive low-bit conversion.

For computer vision applications, quantization should be evaluated alongside preprocessing, input resolution, and post-processing. A smaller backbone may deliver more practical savings than forcing every layer into INT8; teams building such systems can also review approaches to building computer vision models on GitHub.

Production checklist

Before release, confirm that:

• The quantized artifact loads successfully on every supported device tier.
• Peak RAM stays below the budget under realistic app conditions.
• Accuracy meets agreed thresholds on a held-out, representative test set.
• The runtime uses intended INT8, FP16, or INT4 kernels rather than silent fallbacks.
• Crash rates, thermal behavior, battery use, and latency are monitored after rollout.
• A floating-point fallback or remotely configurable model version exists for high-risk failures.

Quantization is most valuable when treated as an engineering experiment with clear budgets, reproducible evaluation, and hardware-specific validation. Start with the simplest method that meets the target, then introduce calibration, selective precision, or QAT only where measurements justify the added complexity.