Small language models (SLMs) are increasingly useful on phones and edge devices because they can handle short, focused tasks without sending every prompt to a cloud API. For Indian builders, that can mean offline voice commands, keyboard suggestions, customer-support assistants, translation, document search, and regional-language interfaces that work despite unreliable connectivity.
The engineering goal is not simply to make a model smaller. It is to deliver acceptable quality within a strict budget for latency, RAM, storage, battery, heat, and privacy. The right optimisation strategy depends on the task, target devices, language mix, and whether inference must work fully offline.
Start with the mobile workload
Define the product constraint before choosing a model. A model that is excellent for summarisation may be wasteful for intent classification or slot filling.
Document:
- Task: generation, classification, extraction, autocomplete, translation, or speech-related text processing.
- Context length: average and worst-case input size; long context increases memory use and latency.
- Response target: maximum acceptable time to first token and tokens per second.
- Offline requirement: fully offline, offline-first with cloud fallback, or cloud-only.
- Device range: entry-level Android phones, mid-range devices, premium phones, or iPhones.
- Data sensitivity: whether prompts, transcripts, or personal data may leave the device.
For Indic applications, evaluate Hindi, Tamil, Bengali, Marathi, Telugu, Kannada, Malayalam, and code-mixed text separately. Tokenisers can be much less efficient for regional languages than for English, increasing sequence length and inference cost. Work on low-resource Indic natural language processing and open-source small language models for Hindi can help inform model and dataset choices.
Choose the smallest model that meets the quality bar
Begin with a capable teacher or reference system, then compare smaller candidates on representative mobile inputs. Parameter count alone is a poor selection metric: architecture, tokenizer efficiency, context window, and runtime support all affect real performance.
Use a held-out evaluation set that includes:
- Short and long user queries.
- Spelling variation, transliteration, and code-mixing.
- Noisy speech transcripts and informal messages.
- Names, addresses, product terms, and local place names.
- Safety-sensitive or ambiguous requests.
- Adversarial prompts and malformed input.
Track task accuracy alongside hallucination rate, refusal behaviour, formatting correctness, and language fidelity. A smaller classifier may outperform a generative model for routing or intent detection. For generation, knowledge distillation can train a compact student to reproduce the teacher’s useful behaviour, while task-specific fine-tuning may produce greater gains than adding parameters.
Compress for the target hardware
Quantisation
Quantisation reduces weight and sometimes activation precision. 4-bit weights can substantially reduce storage and memory bandwidth, while 8-bit or mixed-precision approaches often preserve more quality. Do not assume that the lowest bit width is automatically best: measure quality degradation in every supported language and test long prompts, where errors can compound.
Use representative calibration data rather than generic samples. Include the actual prompt lengths and language distribution expected in production. Quantisation-aware training may be worthwhile when post-training quantisation causes unacceptable accuracy loss.
Pruning and sparsity
Pruning removes weights, heads, or neurons that contribute less to the target task. Unstructured sparsity may reduce theoretical computation without improving speed unless the runtime and hardware support sparse kernels. Structured pruning is often more practical because it produces smaller, regular tensor shapes, but it requires careful retraining and benchmarking.
Distillation and architecture changes
Distil only the capability the application needs. A compact model trained for classification, extraction, or constrained response generation can be faster and more reliable than a general-purpose SLM. If the product needs Indic text generation, preserve tokenizer coverage and test script-specific output rather than relying on English-centric benchmarks.
Optimise the inference path
The model file is only one part of mobile performance. Export it to a runtime supported by your application and hardware, such as a mobile-friendly ONNX, Core ML, TensorFlow Lite, or vendor-specific format. Validate operator compatibility early; unsupported operations can silently force slower CPU execution.
Use:
- KV caching for autoregressive generation to avoid recomputing prior tokens.
- Streaming output so users see useful partial results quickly.
- Bounded context windows and prompt templates that remove unnecessary tokens.
- Batch size one for interactive workloads unless the app has a clear batching opportunity.
- Warm-up and model preloading only when the latency benefit justifies memory use.
- Asynchronous execution to keep the user interface responsive.
- Cancellation when users submit a new request or leave the screen.
Benchmark CPU, GPU, and NPU paths independently. Hardware acceleration can improve throughput but may increase startup time, thermal load, or memory pressure. Test sustained use, not only a single fast inference. A model that performs well for 30 seconds but throttles after five minutes is not production-ready.
For a broader deployment checklist, compare these practices with the AI model optimisation for mobile devices deployment guide.
Design an offline-first product architecture
On-device inference is valuable when connectivity is expensive, unavailable, or inappropriate for sensitive data. Keep the local model responsible for narrow, predictable tasks and use a cloud model only when the user explicitly permits it or the task exceeds the local capability.
A practical fallback policy can consider:
- Network availability and estimated cost.
- Battery level and thermal state.
- Prompt sensitivity and consent.
- Confidence score or validation failure.
- Model version and device capability.
Avoid training directly on private user data by default. Federated learning and on-device personalisation require strong controls for consent, update security, poisoning resistance, and data deletion. For most first releases, secure model updates and local inference are simpler than continuous on-device training.
Measure the metrics users actually feel
Build a device matrix covering low-cost Android phones common in India, mid-range devices, and premium hardware. Record cold start and warm start, time to first token, completion speed, peak RAM, package size, energy draw, thermal state, crash rate, and battery impact over repeated sessions.
Quality tests should include English, Indic scripts, transliterated text, and mixed-language prompts. Compare the compressed model with the uncompressed baseline and define release thresholds in advance. Monitor production telemetry only with appropriate consent and minimise retained text; aggregate latency and error signals wherever possible.
Common mistakes to avoid
- Optimising parameter count while ignoring tokenizer inefficiency.
- Benchmarking only on a flagship phone.
- Reporting average latency without p95 or cold-start results.
- Assuming GPU or NPU execution is faster for every model shape.
- Quantising without testing regional languages and safety behaviour.
- Shipping a large model download without resumable updates or version rollback.
- Treating a cloud fallback as invisible when it changes privacy expectations.
A practical deployment sequence
1. Define the task, supported languages, device floor, and latency budget.
2. Establish a quality baseline with a larger model or cloud reference.
3. Select a compact architecture and remove unnecessary context or output freedom.
4. Distil, fine-tune, quantise, and export to the target runtime.
5. Benchmark on representative devices under cold, warm, and sustained conditions.
6. Red-team prompts, privacy flows, and fallback behaviour.
7. Roll out gradually with signed model updates, telemetry controls, and rollback support.
Optimizing small language models for mobile devices is a product-and-systems problem, not just a compression exercise. Builders who measure language quality, hardware behaviour, and user privacy together can ship faster offline experiences without turning the phone into a miniature cloud server.