Small language models are becoming the practical route to adding generative AI to smartphones, tablets, point-of-sale devices, and other edge hardware. Instead of sending every prompt to a cloud API, a compact model can handle focused tasks locally—with lower latency, better privacy, and useful offline capability.
For Indian builders, the opportunity is especially relevant. Mobile-first users may have inconsistent connectivity, limited data budgets, and diverse language needs. The right model is not simply the one with the fewest parameters. It is the model that meets a defined quality target within the device’s memory, thermal, battery, licensing, and latency limits.
What makes a language model mobile-efficient?
A mobile language model is efficient when it delivers acceptable task quality under strict device constraints. Evaluate it across several dimensions:
- Peak memory: Include model weights, runtime buffers, tokeniser data, and the key-value cache used during generation.
- Prefill and decode latency: Prompt processing and token-by-token generation behave differently, so benchmark both.
- Energy and thermals: Sustained inference can drain batteries or trigger thermal throttling even when short tests look good.
- Model quality: Measure the tasks your product actually performs rather than relying only on general benchmark scores.
- Offline reliability: Test airplane mode, weak networks, app restarts, and low-memory conditions.
- Hardware coverage: Android devices in India span a wide range of CPUs, GPUs, and NPUs; performance on a flagship phone says little about a budget handset.
A model with one billion parameters may be appropriate for classification, rewriting, extraction, or short structured responses. A larger model may be justified for multi-step reasoning, but only if the use case can tolerate its memory and power cost. For many products, a small model paired with deterministic code, retrieval, or a cloud fallback is more dependable than an oversized model running continuously.
Core techniques for reducing size and cost
Quantisation reduces the precision used to store and compute model weights. Four-bit formats can substantially reduce memory, while eight-bit formats often preserve more quality and offer broader hardware support. Quantisation must be tested on representative prompts: multilingual output, names, numbers, code-mixed text, and long context can expose degradation that a general benchmark misses.
Pruning removes weights or structures that contribute little to the target task. Structured pruning is generally easier to accelerate on mobile hardware than arbitrary sparsity, because runtimes and processors can exploit regular shapes more effectively.
Knowledge distillation trains a smaller student model to reproduce useful behaviour from a stronger teacher. For production, distil on the exact intents, formats, and languages your application needs—not just generic web text.
Architecture and runtime choices also matter. Grouped-query attention can reduce key-value cache memory; shorter context limits lower working memory; and efficient tokenisers reduce startup overhead. Export paths such as LiteRT, ExecuTorch, Core ML, or ONNX Runtime should be selected based on the target operating systems and available accelerators. Read the practical considerations in this AI model optimisation for mobile devices guide before committing to an inference stack.
A practical deployment workflow
Start with a narrow product contract. Define supported languages, maximum prompt length, expected response length, acceptable error rate, and whether the model must work fully offline. A customer-support classifier, for example, needs a different model from an on-device writing assistant.
Then follow this sequence:
1. Create a representative evaluation set. Include real, anonymised queries; spelling errors; code-mixed Hindi-English; regional names; numerals; and adversarial inputs.
2. Choose a baseline model. Compare a few compact open models with compatible licences and language coverage. Check whether commercial use, redistribution, and fine-tuning are permitted.
3. Fine-tune or adapt selectively. Use parameter-efficient fine-tuning where possible, and preserve a held-out test set to detect overfitting.
4. Quantise and export early. A model that performs well in desktop Python may fail after conversion because of unsupported operators, tokeniser differences, or accelerator constraints.
5. Benchmark on real devices. Record time to first token, tokens per second, peak RAM, battery impact, temperature, and crash rate across low-, mid-, and high-tier hardware.
6. Add guardrails and fallbacks. Validate structured outputs, cap generation length, filter sensitive content, and route difficult or high-risk requests to a trusted server when consent and connectivity allow.
7. Monitor after release. Track failed intents, user corrections, latency, and battery complaints without collecting unnecessary personal data.
Indian-language and privacy considerations
Indic support cannot be inferred from a model’s language list. Test Devanagari and other scripts, transliteration, spelling variation, numerals, honorifics, and code-mixing. A model may produce fluent Hindi but fail on Marathi, Bengali, Tamil, Telugu, or conversational Hinglish. For background, compare your pipeline with this guide to low-resource Indic natural language processing and review available open-source small language models for Hindi.
On-device inference can reduce exposure of messages, voice transcripts, and personal documents, but it is not automatically private. Logs, crash reports, copied prompts, model downloads, and analytics can still leak sensitive information. Encrypt model files where practical, minimise telemetry, obtain clear consent, and define retention rules. For regulated applications such as health or finance, keep high-impact decisions reviewable and avoid presenting generated text as authoritative advice.
Common mobile use cases
- Keyboard assistance: next-word prediction, rewriting, tone changes, and spelling correction.
- Offline translation and summarisation: useful for field staff, travellers, and users with unreliable connectivity.
- Voice interfaces: combine a compact language model with on-device speech recognition and strict intent schemas. Product teams assessing voice workflows can also consult this guide to voice agent software for small businesses.
- Document extraction: turn invoices, forms, or receipts into structured fields, with deterministic validation before storage.
- Personal productivity: summarise notes, draft messages, or search local documents without uploading them.
- Retail and service applications: support staff with product lookup, multilingual FAQs, and guided workflows on shared devices.
How to choose between local, cloud, and hybrid inference
Use local inference when offline operation, privacy, predictable latency, or low per-request cost matters most. Use cloud inference when the task requires a larger context, complex reasoning, rapid model upgrades, or centralised governance. A hybrid design is often strongest: handle routine, low-risk requests locally and escalate uncertain cases with an explicit user signal.
Do not hide the transition. Tell users when content leaves the device, protect requests in transit, and avoid sending more context than the server needs. For Indian startups, this architecture can control cloud bills while preserving a quality path for difficult queries.
What to measure before shipping
Set release thresholds rather than optimising for a single headline metric:
- p50 and p95 time to first token and completion latency;
- peak RAM and app startup time;
- battery consumption over a realistic session;
- quality by language, device tier, and task type;
- invalid-output, refusal, and hallucination rates;
- crash, thermal-throttling, and model-load failure rates.
A compact model is production-ready only when it remains useful under the worst supported conditions. Treat model updates like software releases: version the weights and prompt templates, test migrations, and keep a rollback path.
FAQ
What is the best model size for a phone?
There is no universal size. Begin with the smallest model that meets your measured quality target, then test memory, latency, and battery use on the least powerful supported device.
Is four-bit quantisation safe for quality?
It can be, particularly for narrow tasks, but quality varies by model and language. Compare quantised and full-precision outputs on your own evaluation set before deployment.
Can small models support Indian languages offline?
Yes, but coverage varies sharply. Test each target language, script, transliteration pattern, and code-mixed input separately; do not rely on an English benchmark.
Should every mobile AI feature run entirely on-device?
No. Local, cloud, and hybrid inference each have valid uses. Choose based on privacy, connectivity, risk, quality, cost, and the device’s actual capabilities.
Apply for AI Grants India
If your Indian startup, research group, or public-interest project is building privacy-preserving mobile AI, language technology, or offline tools, explore support through AI Grants India. A strong application should state the target users, supported devices and languages, evaluation plan, privacy safeguards, and measurable deployment outcomes.