Code-mixed text is normal digital communication in India: “kal meeting hai,” “invoice send pannunga,” and Bengali-English or Marathi-Hinglish messages can appear in the same product workflow. Models trained only on clean, monolingual text often struggle with transliteration, spelling variation, borrowed words, and rapid switches between scripts.
Quantization makes these models cheaper to serve and easier to run on constrained hardware. It reduces weight and activation precision—commonly from FP32 to INT8 or lower—while aiming to preserve task quality. This guide explains how to build a production-ready quantized model for code-mixed Indian languages, with practical choices for data, training, evaluation, and deployment.
Define the task and deployment target first
Do not begin with quantization. Begin by specifying what the model must do and where it will run. Sentiment classification, language identification, moderation, named-entity recognition, translation, speech-text post-processing, and text generation have different accuracy and latency requirements.
Write down:
- Target language pairs or mixtures, such as Hinglish, Tanglish, or Bengali-English.
- Expected scripts: Latin transliteration, native Indic scripts, or both.
- Maximum input length and throughput requirement.
- Hardware target: CPU server, GPU, Android device, browser, or embedded system.
- Acceptable quality loss after quantization.
- Privacy, retention, and latency constraints for user data.
For products serving first-time internet users, model size is only one part of the problem. Input normalization, offline behaviour, and graceful handling of unknown words matter just as much. The broader design principles in building AI apps for India’s next billion users are useful when selecting a deployment architecture.
Build a representative code-mixed dataset
A quantized model cannot compensate for weak or biased training data. Collect examples from the actual channels your product will process: chat, search queries, support tickets, comments, voice transcripts, or educational content. Obtain consent, remove personal information, and document collection sources and licences.
Your dataset should capture:
- Language switches within a sentence, not just separate Hindi and English samples.
- Native-script and Romanized Indic text.
- Phonetic spellings such as “accha,” “acha,” and “achha.”
- Regional vocabulary, abbreviations, emojis, numerals, and punctuation patterns.
- Different levels of mixing, from mostly Indic text to mostly English text.
- Difficult cases, including named entities, addresses, product names, and abusive content.
Create train, validation, and test splits by user, conversation, or time period where possible. Randomly splitting near-duplicate messages can produce inflated results. Maintain a challenge set with rare spellings, script changes, short messages, and language pairs underrepresented in the main corpus.
For low-resource languages, active learning is often more effective than indiscriminate scraping. Start with a small labelled set, run the model, and send uncertain or high-impact examples for review. The workflow complements the techniques covered in low-resource Indic natural language processing.
Choose the tokenizer and base model carefully
Use a multilingual or Indic-focused encoder or decoder that already represents the languages and scripts in your target distribution. Inspect its tokenizer before fine-tuning. A code-mixed sentence that fragments into many single-character or unknown tokens will increase latency and reduce useful context.
Compare candidate tokenizers using:
- Average tokens per character and per message.
- Unknown-token frequency.
- Fragmentation of Romanized Indic words.
- Coverage of numerals, emojis, URLs, and product vocabulary.
- Memory and latency on the intended hardware.
For classification and tagging, a compact encoder is usually easier to quantize and validate than a large generative model. For generation, consider parameter-efficient fine-tuning, distillation, or a smaller instruction model before applying low-bit quantization. Keep the original checkpoint, tokenizer, label map, and preprocessing code versioned together.
Fine-tune a strong full-precision baseline
Train and evaluate an FP32 or BF16 baseline before quantization. This establishes whether errors come from the data and model or from reduced precision. Use class-balanced sampling or weighted loss when labels are uneven, and report results separately by language pair, script, and mixing level.
Useful metrics include:
- Macro-F1 and per-class F1 for classification.
- Entity-level precision, recall, and F1 for NER.
- Exact match and calibrated confidence for intent detection.
- BLEU or chrF alongside human review for translation.
- Task quality, toxicity, hallucination rate, and rejection behaviour for generation.
- P50, P95 latency, peak memory, model size, and energy use.
Do not rely on aggregate accuracy. A model can appear strong while failing consistently on Romanized Kannada or short Hinglish queries. Include native speakers and reviewers familiar with regional usage in error analysis.
Select the right quantization method
The best method depends on the architecture and runtime.
- Dynamic post-training quantization quantizes weights ahead of time and computes activation scales at runtime. It is a fast starting point for CPU inference, especially for smaller encoder models.
- Static post-training quantization calibrates activation ranges on representative samples. It can improve speed and memory use, but calibration data must include real code-mixed patterns.
- Quantization-aware training (QAT) simulates low-precision operations during fine-tuning. Use it when post-training quantization causes a material quality drop.
- Weight-only low-bit quantization can reduce memory substantially for generative models, with the exact trade-off depending on the runtime and hardware.
Quantize one component at a time where possible. Embeddings, layer normalisation, softmax, and output heads can be more sensitive than linear layers. Mixed precision is often the practical answer: keep sensitive operations at higher precision while quantizing the bulk of the model.
Implement a reproducible quantization pipeline
Export the baseline model to the runtime you will actually benchmark. Common options include PyTorch quantization workflows, ONNX Runtime, and TensorFlow Lite. Validate that tokenization and post-processing produce identical outputs before comparing speed.
A robust pipeline should:
- Freeze preprocessing and tokenizer versions.
- Generate calibration samples stratified by language, script, length, and mixing ratio.
- Record weight size, activation memory, throughput, and latency.
- Test CPU instruction-set compatibility and mobile delegates where relevant.
- Save quantization configuration, calibration data version, and evaluation logs.
- Package a full-precision fallback for high-risk or low-confidence cases.
Export failures and unsupported operators are common. Test the converted model early rather than after completing the entire training cycle. For voice products, quantized text models may sit behind speech recognition and synthesis; latency budgets should be measured end to end. See the guidance on building a voice agent architecture for a broader production view.
Evaluate quality, speed, and safety after quantization
Run the exact same fixed test suite against the full-precision and quantized models. Compare absolute and relative changes, not just whether the model still meets a single threshold.
Break down results by:
- Language and language pair.
- Script and transliteration style.
- Input length and token count.
- Rare words, names, URLs, and numerals.
- High-confidence versus uncertain predictions.
- Demographic or regional slices relevant to the application.
Inspect examples where predictions changed. A small overall F1 decline may be acceptable for offline analytics but unacceptable for safety moderation or financial support. For generation, check repetition, code-switch preservation, factuality, and harmful output. Add monitoring for language drift after launch, because slang and transliteration conventions change quickly.
Deploy with an operating plan
Package the quantized model with a versioned tokenizer and explicit metadata. Use canary deployment, shadow traffic, or an A/B test before replacing the baseline. Log anonymised input characteristics, latency, confidence, and fallback rates rather than storing raw personal messages by default.
A practical release checklist includes:
- Model-card documentation covering languages, scripts, data sources, and limitations.
- Reproducible benchmark results on target hardware.
- Rollback to the previous model and full-precision fallback.
- Human escalation for uncertain or high-risk cases.
- A retraining schedule driven by new language patterns and observed errors.
- Clear user communication where automated decisions affect access, money, or safety.
Common mistakes to avoid
- Quantizing before establishing a trustworthy baseline.
- Calibrating only on English or clean native-script text.
- Removing punctuation, emojis, or transliteration clues during cleaning.
- Reporting one accuracy number for several language communities.
- Assuming INT8 always improves latency without benchmarking the actual runtime.
- Treating smaller size as a substitute for privacy, governance, or human review.
FAQ
Does quantization always reduce accuracy?
No. Some models retain nearly all baseline quality after INT8 conversion. Sensitive tasks and poorly calibrated activation ranges may require QAT or mixed precision.
Should I use INT8 or 4-bit weights?
Start with INT8 for predictable production inference, especially on CPUs and mobile runtimes. Evaluate 4-bit weight-only options when memory is the primary constraint and the runtime supports them reliably.
How much calibration data is needed?
There is no universal number. Use a few hundred to several thousand representative samples, then confirm that quality stabilises as you add language, script, length, and domain diversity.
Can one model handle all Indian code-mixed languages?
It can, but a single model may underperform on low-resource pairs. Compare a shared multilingual model with language adapters or specialised models using per-language benchmarks and total serving cost.
How should a grant-backed prototype proceed?
Define a narrow use case, release a measurable full-precision baseline, collect consented and representative data, and publish quantized benchmark results. Projects with clear deployment constraints and reproducible evaluation are easier to validate and scale through Indian open-source AI developer projects.