Quantization is no longer a niche optimisation reserved for embedded systems. In 2026, it is one of the main ways Indian developers run language, vision, and speech models on laptops, workstations, edge devices, and private servers without paying for every inference call.
But there is no single best quantized model for local inference. The right choice depends on the model family, task, quantization format, accelerator, context length, and acceptable quality loss. A 4-bit language model may be ideal for a developer laptop, while an 8-bit vision model may be safer for a production inspection system.
The short answer
For most local text-generation projects, start with a capable 7B–9B instruction model in a 4-bit format, usually GGUF for CPU-friendly runtimes or AWQ/GPTQ for supported NVIDIA GPUs. Move to 8-bit when quality is more important than memory, and choose a smaller model before using extreme quantization.
For mobile and edge computer vision, prefer an INT8 TensorFlow Lite, ONNX Runtime, or vendor-specific model validated on the target device. For Indian-language applications, benchmark the exact languages and scripts you need rather than relying on English-language scores. Teams building Hindi or regional-language tools can also review this guide to open-source small language models for Hindi.
What quantization changes
Quantization stores model weights and, sometimes, activations at lower numerical precision. Common choices include FP16, INT8, and 4-bit or 3-bit weight formats. Lower precision generally reduces memory use and can improve throughput, but the result depends heavily on the runtime and hardware.
- FP16 or BF16: High quality and broad compatibility, but large memory requirements.
- INT8: A strong choice for stable production inference, especially on mobile, CPU, and edge accelerators.
- 4-bit weight-only quantization: Often the best balance for local LLM use; it substantially reduces memory while retaining useful quality.
- 3-bit and lower: Useful only when memory is severely constrained and the quality trade-off has been measured.
Weight-only quantization is usually easier to deploy than full weight-and-activation quantization. Activation quantization can deliver additional speed, but it requires calibration and kernel support. A model advertised as “4-bit” is therefore not automatically faster on every device.
Best choices by use case
Local LLMs on laptops and CPUs
Choose a GGUF model and run it with llama.cpp-compatible software, Ollama, or another runtime that supports the relevant architecture. A 4-bit quantisation such as Q4_K_M is a practical starting point for many 7B–9B models. It keeps RAM requirements manageable while preserving reasonable instruction-following quality.
Use Q5 or Q6 when your machine has additional memory and the task involves structured output, coding, translation, or long documents. If you are deploying a larger model, estimate memory for weights, KV cache, runtime overhead, and the operating system—not just the published file size.
Developers who need a complete local serving workflow should first understand how to deploy large language models locally, including model storage, API compatibility, concurrency, and observability.
NVIDIA GPU inference
For NVIDIA GPUs, AWQ and GPTQ are common options, while FP8 is increasingly relevant on newer datacenter and workstation hardware. AWQ often performs well for serving workloads with modern GPU kernels; GPTQ remains widely available across model repositories and inference libraries.
Select the format supported by your serving stack rather than converting blindly. Check compatibility with vLLM, TensorRT-LLM, Transformers, ExLlama, or the specific application framework. A theoretically smaller model can be slower if the runtime falls back to inefficient kernels.
Mobile and edge vision
For image classification, detection, and segmentation, INT8 TensorFlow Lite or ONNX models are usually more dependable than LLM-style weight-only formats. MobileNet remains a sensible baseline for constrained devices; EfficientNet or newer compact backbones may deliver better accuracy at a higher compute cost.
Calibration data matters. Use representative images from Indian deployment conditions—local lighting, camera quality, document formats, skin tones, road scenes, or agricultural environments. A generic calibration set can produce a model that looks strong in benchmarks but fails in the field. For a broader deployment workflow, see this AI model optimisation guide for mobile devices.
Vision-language and multimodal systems
Multimodal models are harder to quantize because the vision encoder, projector, and language model may use different precision requirements. Quantizing only the language component can reduce memory while preserving visual quality, but it may not deliver the expected end-to-end speedup.
Test image resolution, OCR accuracy, grounding, and multilingual prompts separately. Teams working on Indian-language multimodal applications may find open-source vision-language models for Indian languages useful when comparing model families.
How to choose the right quantized model
Start with the deployment constraint, not the model leaderboard. Record:
- Available memory: Include system RAM or VRAM, KV cache, batch size, and context length.
- Latency target: Measure time to first token and tokens per second for LLMs; measure end-to-end milliseconds for vision.
- Concurrency: A model that is fast for one user may be inefficient for ten simultaneous requests.
- Quality threshold: Define acceptable error rates for extraction, translation, classification, or generation.
- Runtime support: Confirm that the chosen format has optimised kernels on your CPU, GPU, NPU, or mobile chipset.
- Data sensitivity: Local inference can help keep health, financial, government, and enterprise data on-device, but logs and caches still require protection.
For Indian deployments, add language coverage, script handling, code-mixed input, and accent or dialect robustness to the test plan. An English benchmark cannot predict performance on Hindi-English queries, Kannada speech, or low-resource scripts.
A practical benchmarking process
1. Choose two or three model sizes and two quantization levels for each.
2. Build a representative evaluation set from real, anonymised requests.
3. Measure cold-start time, first-token latency, sustained throughput, peak memory, and energy use.
4. Score task quality using exact-match, F1, groundedness, translation quality, or human review as appropriate.
5. Test long context, malformed inputs, concurrent requests, and device thermals.
6. Compare the quantized model with the original FP16 or BF16 baseline.
Do not evaluate only the model file. Measure the full application, including tokenisation, retrieval, image preprocessing, post-processing, and network calls. For video or multimodal projects, a targeted comparison such as evaluating vision models for video understanding can help expose bottlenecks that a static accuracy score misses.
Common mistakes
- Assuming the lowest-bit model is always the fastest.
- Comparing models with different prompts, context lengths, or sampling settings.
- Ignoring KV-cache memory during long conversations.
- Using a calibration dataset unrelated to production traffic.
- Treating a small benchmark improvement as meaningful without confidence intervals.
- Deploying a quantized model without testing safety refusals, hallucination rates, and structured-output reliability.
Recommendation
For a first local LLM deployment, benchmark a reputable 7B–9B instruction model in Q4_K_M or an equivalent 4-bit format, then compare it with Q5 or INT8 if quality is marginal. Use GGUF for flexible CPU and laptop deployment, AWQ or GPTQ for compatible NVIDIA GPU serving, and INT8 TFLite or ONNX for mobile and edge vision.
The best model is the smallest quantized model that meets your quality, latency, memory, and privacy requirements on the actual target device. If you are building an India-focused AI product and need help funding evaluation or deployment work, explore AI Grants India.