Indian-language translation is an engineering problem shaped by data quality, script diversity, compute constraints, and uneven language coverage. A model that works well for Hindi-to-English may perform poorly on Marathi, Assamese, Kannada, or code-mixed speech. Quantisation can make a translation system cheaper and faster to serve, but it does not repair weak data or remove linguistic bias.
This guide explains how to build a quantised neural machine translation system for Indian languages, with practical decisions for teams targeting mobile apps, government services, education products, customer support, or offline deployments. For broader context on data scarcity and evaluation, pair this workflow with the guide to low-resource Indic natural language processing.
Define the deployment target first
Start with the device, latency budget, and quality threshold—not the quantisation library. Write down:
- Source and target languages, including scripts and dialects.
- Whether translation is text-only or part of a speech pipeline.
- Maximum model size and RAM available on the target device.
- Expected latency for a sentence, document, or streaming request.
- Offline, private-cloud, or public API deployment requirements.
- Minimum acceptable quality for names, numbers, addresses, and domain terminology.
A 4-bit model on a modern server may be useful for cost reduction, while an Android application may need an int8 model supported by the device’s neural accelerator. Establish a float16 baseline before optimising so that every compression decision can be measured against a known reference.
Curate Indic training and calibration data
Quantisation is sensitive to the quality and distribution of the data used to train and calibrate the model. Build language-pair-specific datasets rather than assuming that a large multilingual corpus is representative.
Useful sources include licensed parallel corpora, public-sector documents, translated product content, subtitles, support conversations, and carefully reviewed synthetic data. Track provenance and consent, and remove personal information before training. For low-resource pairs, use back-translation and multilingual transfer cautiously; synthetic sentences should not overwhelm naturally written text.
Preprocessing should preserve information that matters in Indian languages:
- Keep Unicode normalisation consistent, especially for Devanagari and combining marks.
- Detect and label code-mixed text instead of silently discarding it.
- Protect named entities, URLs, phone numbers, dates, currency, and addresses.
- Deduplicate near-identical sentence pairs and remove misaligned translations.
- Balance formal, conversational, regional, and domain-specific examples.
- Create separate development and test sets by domain and language variety.
Inspect tokenisation manually. A tokenizer that fragments common words, suffixes, or transliterated terms excessively will increase sequence length and inference cost. SentencePiece or unigram tokenisation is often a practical starting point, but test it on each target language rather than relying on English-centric assumptions.
Choose and train a translation model
Use an encoder-decoder architecture designed for sequence-to-sequence translation, such as a multilingual Transformer or an Indic-focused open model. BERT is primarily an encoder model and GPT-style models are not automatically the best translation choice; select an architecture with an appropriate generation head and tokenizer.
For a new system, compare three routes:
- Fine-tune an existing multilingual model: Usually the fastest path when labelled data and compute are limited.
- Train a compact model from scratch: Appropriate when licensing, privacy, or a narrow domain demands full control.
- Distil a larger teacher model: Useful when you need a small student model while retaining translation behaviour.
Train with language tags or explicit source-target identifiers for multilingual models. Monitor validation loss separately for every language direction. Aggregate scores can hide a severe regression in a low-resource language. Use curriculum or sampling strategies that prevent high-resource pairs from dominating the optimiser.
Before quantisation, evaluate the full-precision checkpoint with sacreBLEU or chrF, plus human review. COMET-style metrics can add semantic signal, but automated metrics should not be the sole measure for morphologically rich languages. Build a challenge set containing negation, honorifics, spelling variation, code-mixing, numerals, named entities, and culturally specific expressions.
Select a quantisation strategy
Quantisation reduces the precision of weights and, depending on the method, activations. The main options are:
- Dynamic post-training quantisation: Weights are quantised after training while activations are handled dynamically. It is simple and often useful for CPU inference.
- Static post-training quantisation: Weights and activations are calibrated using representative samples. It can deliver better speed on supported hardware but requires careful calibration.
- Quantisation-aware training (QAT): The model simulates low-precision arithmetic during fine-tuning. This generally offers the best recovery when post-training quantisation causes a quality drop.
- Weight-only 4-bit quantisation: Common for larger Transformer models where memory bandwidth is the main bottleneck. Validate kernel and hardware support before promising latency gains.
Start with int8 weights and activations where possible. Use a calibration set that reflects production traffic across languages, sentence lengths, scripts, and domains. Calibration data should not contain private user content unless it is properly governed and approved.
With PyTorch, inspect torchao or supported backend workflows; with ONNX Runtime, TensorFlow Lite, or mobile runtimes, confirm that the exported graph uses real low-precision kernels rather than silently falling back to float operations. Export one model, run it on the actual target device, and profile end-to-end latency, memory, and energy consumption.
Evaluate quality, speed, and failure modes
Compare full-precision, float16, int8, and lower-bit variants using the same test set and decoding configuration. Track:
- chrF and sacreBLEU by language direction.
- Terminology accuracy for priority domains.
- Named-entity, number, date, and URL preservation.
- Hallucination, omission, and untranslated-span rates.
- P50, P95, and P99 latency under realistic concurrency.
- Peak RAM, model download size, battery impact, and cost per million characters.
A small average BLEU loss may be acceptable if latency and cost improve, but a large drop on safety-critical content is not. Review samples side by side with native speakers, including speakers from different regions. For customer support or public services, add a confidence or escalation path rather than presenting uncertain output as authoritative.
Deploy with safeguards
Package the model with a versioned tokenizer, preprocessing rules, language detection, and decoding configuration. Keep these components together; changing normalisation or token IDs independently can invalidate results. Consider on-device inference for sensitive text and server inference for longer documents or unsupported language pairs.
For APIs, expose language direction, model version, and request limits. Log aggregate quality and latency signals without retaining raw text by default. Monitor language-specific drift, especially after adding new domains or retraining on synthetic data. A rollout canary and automatic rollback are safer than replacing the production model in one step.
If the product includes speech input or output, the translation model is only one part of the stack. Review the architecture alongside guidance on building AI apps for the next billion users in India and, for voice interfaces, the practical voice agent architecture and deployment guide.
Common mistakes to avoid
- Quantising before establishing a reproducible full-precision baseline.
- Measuring only English-facing aggregate scores.
- Using an unrepresentative calibration set.
- Assuming smaller storage always means faster inference.
- Ignoring tokenizer growth and sequence-length inflation.
- Testing on a desktop instead of the actual phone, server, or accelerator.
- Treating synthetic data as equivalent to native human translation.
- Omitting licence, consent, and privacy checks for training data.
A practical build sequence
1. Select one language pair and define production constraints.
2. Clean, licence, and audit parallel data; reserve a native-reviewed test set.
3. Fine-tune or train a seq2seq baseline and record quality and resource metrics.
4. Export to the intended runtime and verify numerical equivalence before quantising.
5. Try int8 post-training quantisation with representative calibration data.
6. Use QAT or selective higher precision for layers and language pairs that regress.
7. Benchmark on target hardware, conduct native-speaker review, and canary deploy.
8. Monitor quality, latency, cost, and language coverage continuously.
Quantisation is most valuable when treated as part of a complete deployment discipline. Strong Indic data, language-aware evaluation, hardware-specific profiling, and transparent fallback behaviour will usually matter more than choosing the most aggressive bit width. For teams building production systems in India, the goal is not merely a smaller checkpoint—it is reliable translation that remains accessible under real bandwidth, device, and compute constraints.
FAQ
Does quantisation reduce translation quality?
It can. The impact depends on the architecture, bit width, calibration data, and language pair. Test each variant and use QAT or mixed precision when important directions regress.
Should I use int8 or 4-bit quantisation?
Use int8 as a dependable starting point for CPU and mobile deployment. Consider 4-bit weight-only quantisation when memory is the main constraint and your runtime provides tested kernels.
How much data is needed?
There is no universal threshold. A smaller, clean, domain-matched corpus can outperform a larger noisy one. Low-resource pairs benefit from transfer learning, back-translation, and native-speaker validation.
Can quantised models run offline?
Yes, if the model and runtime fit the device’s storage, RAM, and accelerator support. Benchmark the complete application, including tokenisation and post-processing, not only the neural network.
Apply for AI Grants India
If you are building an Indic translation product with measurable public, commercial, or accessibility impact, learn about AI Grants India and prepare a proposal covering the problem, data governance, evaluation plan, deployment constraints, and expected users.