Apply for AI Grants India

Why run a quantized Tamil model offline?

Offline inference is useful when a Tamil application must work with unreliable connectivity, sensitive data, or strict latency requirements. A field-service app, school device, call-centre tool, or public-service kiosk can process text or speech locally instead of sending user data to a remote API. This also makes operating costs more predictable and allows deployment in areas where bandwidth is expensive or intermittent.

Quantization reduces the precision used to store and compute model values. A model converted from FP32 to INT8, for example, can require substantially less memory and often run faster on CPUs and mobile hardware. The trade-off is that accuracy, especially on spelling variants, code-mixed Tamil-English text, transliteration, and low-frequency vocabulary, must be measured rather than assumed.

For wider deployment guidance, see this practical overview of AI model optimisation for mobile devices. If your system is a generative assistant rather than a classifier or encoder, compare the runtime decisions with how to deploy large language models locally.

Choose the model and runtime together

Start with the task, not the file format. Common offline Tamil use cases include:

• Text classification: intent detection, moderation, sentiment, and document routing.
• Token classification: named-entity recognition, keyword extraction, and address parsing.
• Translation: Tamil-English or Tamil-language support, provided the model and tokenizer are bundled locally.
• Speech: automatic speech recognition requires a compatible audio model, feature extractor, and language decoder.
• Generation: compact causal or encoder-decoder models, subject to device memory and latency limits.

Then select a runtime that supports the model architecture and target hardware. TensorFlow Lite is practical for Android and embedded deployments. ONNX Runtime is flexible across Python, desktop, Android, and server environments. For transformer models, a GGUF-compatible runtime may be more convenient, but only when the model architecture and tokenizer are supported. Do not assume that any .onnx, .tflite, or quantized weight file can load in every runtime.

Tamil model selection also requires checking the training and evaluation data. Look for coverage of modern Tamil, regional variation, formal text, code-mixing, Unicode normalisation, and the exact task you need. Work on the broader Indian-language ecosystem can benefit from the lessons in open-source vision-language models for Indian languages, particularly around licensing, multilingual evaluation, and data coverage.

Prepare an offline package

Create a clean environment on a connected development machine, but design the final package to work without network access. Pin runtime and library versions, download every dependency in advance, and record checksums for model files. Your deployment bundle may include:

• Quantized model weights.
• Tokenizer files, vocabulary, merges, and special-token configuration.
• Pre-processing and post-processing code.
• Runtime libraries for the target operating system and CPU architecture.
• A small test dataset and expected outputs.
• Licence, attribution, and model-card information.

Tamil text must be normalised consistently at both input and training time. Preserve Unicode Tamil characters, handle zero-width characters deliberately, and decide whether punctuation, whitespace, numerals, and Latin-script transliteration are retained. A tokenizer mismatch can look like a model-quality problem even when the quantized weights are correct.

Quantize and export the model

There are three common approaches:

• Dynamic post-training quantization: weights are quantized after training while some activations are handled dynamically. It is usually the quickest option for CPU inference.
• Static INT8 quantization: weights and activations are calibrated using representative inputs. This can improve speed and memory use, but the calibration set must reflect real Tamil traffic.
• Quantization-aware training: the model is trained while simulating reduced precision. Use it when post-training conversion causes unacceptable quality loss or when hardware requires a specific INT8 path.

Build a representative calibration set containing short and long Tamil sentences, mixed-script inputs, names, numerals, punctuation, spelling variation, and domain-specific terms. Never calibrate only on clean textbook Tamil if the production app will process chat messages or speech transcripts.

Export the model only after validating that input names, tensor shapes, dynamic axes, attention masks, and output semantics are preserved. Keep the original FP32 or FP16 model as a reference. The quantized model should be compared against it on the same examples before it is packaged for a device.

Run inference locally

A minimal ONNX Runtime pattern looks like this:

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession(
    "tamil_model.int8.onnx",
    providers=["CPUExecutionProvider"],
)

inputs = {
    "input_ids": input_ids.astype(np.int64),
    "attention_mask": attention_mask.astype(np.int64),
}
outputs = session.run(None, inputs)

The exact tensor names and data types depend on the exported model. Inspect session.get_inputs() and session.get_outputs() rather than hard-coding assumptions. For TensorFlow Lite, load the interpreter, call allocate_tensors(), inspect input and output quantisation parameters, then set tensors and invoke the model:

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="tamil_model.int8.tflite")
interpreter.allocate_tensors()
input_info = interpreter.get_input_details()[0]
output_info = interpreter.get_output_details()[0]
interpreter.set_tensor(input_info["index"], quantized_input)
interpreter.invoke()
result = interpreter.get_tensor(output_info["index"])

For INT8 or UINT8 models, convert values using the tensor's scale and zero point. Feeding floating-point data directly into an integer tensor will either fail or produce incorrect results. Apply dequantisation to outputs when the application expects probabilities or logits.

Benchmark quality, speed, and memory

A successful local run is not the same as a production-ready deployment. Evaluate the FP32 and quantized versions on a held-out Tamil test set and report task-specific metrics. For classification, inspect macro-F1 and confusion matrices. For generation or translation, combine automatic scores with human review of fluency, names, negation, numbers, and meaning preservation.

Measure at least:

• Cold-start and warm inference latency.
• Peak resident memory and model size on disk.
• Throughput at realistic batch sizes, often one for mobile apps.
• Battery or thermal impact during sustained use.
• Failure rates for empty, very long, malformed, and mixed-script input.

Test on the oldest supported device, not only a developer laptop. If latency is poor, reduce sequence length, use a smaller model, enable supported CPU delegates, or move expensive pre-processing out of the critical path. Hardware acceleration is useful only when the converted operators are actually supported; unsupported operations may fall back to the CPU and erase the expected gain.

Common failure modes

Accuracy drops after quantization: inspect errors by category, then try better calibration data, per-channel weight quantization, selective higher precision, or QAT. Tamil named entities and rare tokens often expose weaknesses first.

The model loads but produces nonsense: verify tokenizer files, Unicode normalisation, vocabulary version, tensor order, attention masks, and label mapping. Compare intermediate outputs with the reference model.

The app crashes on device: check ABI compatibility, peak memory, thread count, and maximum sequence length. Add input limits and graceful fallback messages.

Offline operation is incomplete: remove runtime downloads, remote telemetry dependencies, and lazy model fetching. Exercise the application in airplane mode from a clean installation.

If your project needs a custom Tamil model, benchmark it against related Indian-language systems; the methodology in benchmarking NLP models for Telugu and Sanskrit is a useful template for documenting datasets, metrics, and limitations.

Production checklist

Before release, confirm that:

• The model, tokenizer, runtime, and licences are bundled and version-pinned.
• Inference works with networking disabled.
• Quantization parameters are handled correctly.
• Tamil, transliterated, code-mixed, and malformed inputs are tested.
• Quality and latency targets are defined on real target hardware.
• Logs avoid storing sensitive user text by default.
• Updates are signed, reproducible, and reversible.

Offline Tamil AI is most reliable when treated as an engineering system rather than a model-conversion task. Choose the runtime early, preserve the tokenizer, calibrate with real Tamil data, and publish measured quality and device performance alongside the model.