How to Quantize a Model for Android

Quantizing a model for Android is a deployment decision, not simply a file-conversion step. The right quantization scheme can reduce download size, memory use, inference latency, and energy consumption—important constraints for Android phones used across India’s varied network, hardware, and battery conditions. The wrong scheme can introduce unacceptable accuracy loss or fail to use the device’s available accelerator.

This guide explains how to choose a quantization method, convert a model, test it on representative Android hardware, and troubleshoot the problems that commonly appear after conversion.

What model quantization changes

Quantization represents model values with fewer bits. A standard neural network often stores weights and activations as 32-bit floating-point values (FP32). Quantized models may use FP16 or 8-bit integers (INT8), reducing memory traffic and arithmetic cost.

The practical benefits are:

Smaller downloads and app packages: Useful where users rely on mobile data or intermittent connectivity.
Lower peak memory use: Particularly valuable for camera, speech, and language models running alongside the Android UI.
Faster inference: INT8 kernels can outperform FP32 on supported CPUs and accelerators.
Lower energy use: Less memory movement often matters as much as raw arithmetic.
More predictable offline operation: Smaller models are easier to bundle for privacy-sensitive, low-connectivity use cases.

Quantization does not guarantee a speedup. Performance depends on the operators in the graph, the runtime, delegate support, thread count, and the target chipset. Treat the result as a new deployment artifact that must be measured.

Choose the right quantization method

FP16 weight quantization

FP16 stores weights with 16-bit floating-point precision while commonly retaining floating-point execution. It can nearly halve model size and usually causes little accuracy change. It is a sensible first option when the target device has GPU support or when INT8 calibration is difficult.

Dynamic-range quantization

Dynamic-range quantization converts weights to INT8, while some activation values are quantized during inference. It requires no representative dataset and is quick to try. It often works well for CPU-bound models, although its latency and memory benefits vary by operator.

Full integer quantization

Full INT8 quantization converts weights and activations to integer representations. It generally offers the best CPU efficiency and is often needed for dedicated mobile NPUs or DSPs. You must provide a representative dataset so the converter can estimate activation ranges.

Quantization-aware training

Quantization-aware training (QAT) inserts simulated quantization during training. The model learns to tolerate reduced precision and can preserve accuracy better than post-training quantization, especially for sensitive vision, audio, and language workloads. Use QAT when calibration alone produces unacceptable degradation and you can retrain or fine-tune the model.

For broader deployment planning, compare these choices with the recommendations in AI model optimization for mobile devices.

Before conversion: define the Android target

Write down the constraints before changing the model:

Minimum Android version and supported ABIs, such as arm64-v8a.
CPU-only, GPU, or vendor accelerator execution.
Maximum model size and peak RAM budget.
Target latency, throughput, and battery limits.
Offline, cloud-assisted, or hybrid inference requirements.
Accuracy thresholds for each important class, language, or user segment.

Benchmark on representative devices rather than relying only on a desktop. A mid-range Android phone in India may have very different memory bandwidth and accelerator support from a flagship development handset. Also decide whether the model should be downloaded after installation; this can reduce initial APK size but adds model-versioning and connectivity logic.

TensorFlow Lite conversion workflow

The example below uses TensorFlow Lite. Start with a validated FP32 baseline and keep its predictions, metrics, and preprocessing pipeline unchanged.

import tensorflow as tf

model = tf.keras.models.load_model("model.keras")
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]

tflite_model = converter.convert()
with open("model_dynamic_int8.tflite", "wb") as f:
    f.write(tflite_model)

For FP16 weights:

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

with open("model_fp16.tflite", "wb") as f:
    f.write(converter.convert())

For full INT8 quantization, the representative dataset must resemble real Android inputs. Include variation in lighting, accents, background noise, camera quality, text length, or image orientation as appropriate. Do not use only clean training examples.

def representative_data():
    for sample in calibration_samples.take(100):
        yield [sample]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

with open("model_int8.tflite", "wb") as f:
    f.write(converter.convert())

The exact input shape, scaling, and data type must match the Android preprocessing code. A common deployment bug is to quantize the model correctly but feed normalized floating-point values into an INT8 input without applying its scale and zero-point.

Inspect inputs, outputs, and operators

After conversion, inspect the model’s input and output details. For an INT8 tensor, the runtime exposes quantization parameters:

real_value = scale * (quantized_value - zero_point)

Use the model’s actual scale and zero-point rather than hard-coding assumptions. Check whether every operator is supported by the intended runtime or delegate. Unsupported operations may force parts of the graph back to FP32, causing extra copies and disappointing latency.

If your source model is PyTorch or another framework, export to a supported mobile format first and verify numerical parity. ONNX Runtime and other deployment stacks have their own quantization APIs and operator constraints; conversion is not automatically equivalent across runtimes.

Validate accuracy and Android performance

Run the FP32 and quantized models on the same held-out test set. Report more than one aggregate score:

Classification accuracy, F1, precision, recall, and confusion matrix.
Detection mAP and performance on small or partially occluded objects.
Speech word-error rate across accents and noisy environments.
Translation or language-model quality across supported Indian languages.
Calibration, false-positive rates, and failure cases that affect users most.

Then benchmark on physical devices. Measure cold start, warm inference, p50 and p95 latency, peak memory, sustained performance after repeated runs, and battery or thermal behaviour. Test realistic camera resolutions, audio buffers, and concurrent app activity. A model that is fast for ten inferences may throttle during a five-minute scanning session.

For computer-vision projects, the model’s preprocessing and camera pipeline can dominate total latency; resources on building computer vision models on GitHub can help structure reproducible experiments. For language applications, evaluate representative Hindi, Marathi, Telugu, Sanskrit, or other target-language inputs rather than assuming English benchmarks transfer.

Android integration checklist

Package the .tflite file in assets or download it through a versioned model service.
Use the current TensorFlow Lite or LiteRT Android runtime supported by your stack.
Select CPU, GPU, or NNAPI delegates only after measuring them on target devices.
Reuse interpreter and tensor buffers; avoid allocating on every request.
Configure threads conservatively to prevent thermal throttling and UI contention.
Keep preprocessing identical between evaluation and production.
Record model version, runtime version, device model, delegate, and quantization type.
Add a fallback model or server path if unsupported operators or devices are unavoidable.

A model registry and staged rollout are especially useful when supporting a wide Android fleet. Ship quantized variants behind a feature flag, monitor crash rates and quality signals, and retain the previous model for rollback.

Common failure modes

Accuracy drops sharply: Improve representative data, inspect outlier ranges, try per-channel weight quantization, or use QAT. Check preprocessing first; many apparent quantization failures are input-scaling mismatches.

INT8 is not faster: Confirm that the delegate supports the operators and that the model is not silently falling back to floating point. Compare one-thread and multi-thread CPU runs and measure end-to-end latency, not just interpreter time.

The app crashes or runs out of memory: Reduce concurrent buffers, avoid loading multiple model copies, verify tensor shapes, and test the largest supported input size.

Results differ between Python and Android: Compare preprocessing byte-for-byte, inspect input/output quantization parameters, and test a fixed set of serialized inputs through both pipelines.

Final recommendation

Start with dynamic-range or FP16 quantization to establish a safe baseline. Move to full INT8 when CPU, accelerator, size, or battery targets justify the calibration work. Use QAT when post-training methods fail your accuracy threshold. The production choice should be the smallest model that meets quality and latency requirements on real Android devices—not the model with the most aggressive bit reduction.

Teams building local language applications can also review open-source small language models for Hindi and how to deploy large language models locally when deciding whether quantization should happen on-device, at the edge, or on a server.

Apply for AI Grants India