Production optimisation is not the same as achieving the best validation score. A deployable model must meet a measurable service-level objective (SLO), remain reliable under Indian traffic patterns and infrastructure constraints, and provide evidence when its quality changes. Treat the model, preprocessing code, runtime, hardware, and monitoring as one production system.
Start with a deployment contract
Before changing the architecture, define what “ready” means for the application. Record a baseline on representative data and under realistic load—not only on a developer laptop or a clean benchmark dataset.
Set targets for:
- Quality: accuracy, F1, recall, calibration, word-error rate, or a business metric such as fraud caught per 1,000 transactions.
- Latency: p50, p95, and p99 response times, including tokenisation, image decoding, network calls, and post-processing.
- Throughput: requests per second or items per minute at the expected concurrency.
- Cost: rupees per request, GPU-hour, or monthly infrastructure spend.
- Availability: error rate, timeout budget, recovery time, and behaviour when dependencies fail.
- Resource limits: CPU, GPU memory, RAM, disk, power, and bandwidth available in the target environment.
Use production-like inputs: multilingual text, low-resolution images, long documents, noisy audio, and the actual class imbalance. If your team is still validating the modelling workflow, a strong machine learning portfolio project for beginners in India can provide useful practice in reproducible evaluation and deployment discipline.
Reduce the model before optimising the server
Profile the model first. Identify expensive operators, memory transfers, repeated preprocessing, and layers that do not map efficiently to the target accelerator. Optimisation without profiling often shifts the bottleneck rather than removing it.
Quantisation
Move from FP32 to FP16 or BF16 when the hardware and model support it. INT8 post-training quantisation can reduce memory use and improve CPU or accelerator throughput, but validate accuracy on a calibration set that reflects production traffic. For sensitive models, quantisation-aware training usually preserves quality better than applying compression after training.
Pruning and distillation
Structured pruning removes channels, heads, or blocks that hardware can skip efficiently. Unstructured sparsity may reduce parameter count without improving real latency unless the runtime supports sparse kernels. Knowledge distillation is often more dependable: train a smaller student model against the teacher’s logits, labels, and—where appropriate—task-specific hard examples.
Architecture choices
A smaller architecture trained for the job can outperform a compressed general-purpose model. Limit input resolution, sequence length, and generation tokens where the product allows it. For language or vision-language systems, route simple requests to a smaller model and reserve the larger model for cases that need it. Teams building agents should also review how to deploy open-source AI agents in production, since tool calls and orchestration can dominate end-to-end latency.
Build a predictable inference path
Export the model to a stable interchange or serving format such as ONNX where supported, then benchmark the exported graph—not just the training framework. Runtimes such as ONNX Runtime, TensorRT, OpenVINO, and vendor-specific accelerators can fuse operators and select better kernels, but compatibility and numerical differences must be tested.
Keep preprocessing and post-processing close to the model contract. Version tokenisers, image transforms, label maps, and normalisation rules alongside the model. A model server that receives one format while the application silently sends another can produce plausible but incorrect predictions.
Choose a serving pattern based on the workload:
- Online synchronous inference: use strict timeouts, request validation, bounded queues, and a fallback response.
- Dynamic batching: combine requests for better accelerator utilisation, but cap the batching window to protect p95 latency.
- Asynchronous jobs: use a durable queue for document processing, transcription, or bulk scoring.
- Streaming: send partial results only when the user experience benefits and the system can enforce backpressure.
- Edge inference: place the model near cameras, devices, or intermittent networks when data residency, bandwidth, or latency makes central serving unsuitable.
For GPU deployments, measure utilisation, memory pressure, cold-start time, and data-transfer overhead. A smaller model on a well-utilised CPU may be cheaper and more reliable than an under-utilised GPU. In India, also account for regional availability, egress charges, power constraints, and the sensitivity of data handled by the system.
Make releases reversible
Package the model, runtime dependencies, configuration, and preprocessing code together. Use immutable artefacts with a model version, data snapshot, evaluation report, and dependency lockfile. A container image alone is not a reproducibility strategy if the weights or prompts can change outside version control.
Release progressively:
- Run offline regression tests and adversarial cases before deployment.
- Shadow a candidate model against live requests without returning its output to users.
- Use canary traffic, then compare quality, latency, cost, and failure rates.
- Keep blue-green rollback or a previous model warm enough to restore service quickly.
- Record which model version produced every important prediction where auditability matters.
If the system includes an agent rather than a single predictor, test tool permissions, prompt-injection resistance, retries, and maximum execution time separately. The operational lessons in how to deploy Llama 3 agents in production are relevant even when your final architecture uses another open model.
Monitor quality, not just uptime
Infrastructure dashboards cannot tell you whether a model is becoming less useful. Track request volume, queue depth, p50/p95/p99 latency, errors, timeouts, token or image counts, hardware utilisation, and cost. Then add model-specific signals:
- Input schema violations and missing fields.
- Distribution shifts in language, geography, device, or user segment.
- Confidence, calibration, abstention, and class proportions.
- Ground-truth quality when labels arrive later.
- Slice-level performance for accents, Indian languages, low-bandwidth users, and rare but consequential cases.
Set alerts on sustained changes rather than single noisy points. Data drift is a trigger for investigation, not automatic proof that retraining is needed. Establish a retraining policy with approval thresholds, data quality checks, bias review, rollback criteria, and a holdout set that is never used for tuning.
A practical optimisation loop
1. Establish a reproducible baseline on production-like data.
2. Profile end-to-end latency and resource use.
3. Change one variable—precision, architecture, batching, runtime, or hardware.
4. Re-run quality, load, failure, and cost tests.
5. Document the trade-off and promote only if the deployment contract still holds.
6. Canary the release and monitor it long enough to capture real traffic variation.
Do not optimise solely for benchmark throughput. The best production model is the one that meets quality and reliability requirements at an acceptable total cost. For founders moving from a prototype to a company, transitioning from research to a deep tech startup in India offers useful context on turning technical performance into a sustainable product decision.
FAQ
Is quantisation always the first step?
No. First measure the bottleneck and set an accuracy tolerance. Quantisation is powerful, but batching, input limits, operator fusion, or a smaller architecture may deliver a better trade-off.
Should every model run on a GPU?
No. Benchmark the complete workload. CPUs can be more economical for low-volume or small models, while GPUs are valuable for high concurrency and large matrix operations.
How often should a production model be retrained?
Use drift, labelled performance, business impact, and data freshness—not a fixed calendar alone. Retrain only when new data passes quality and governance checks.
What should an India-based team test first?
Test regional languages, code-switching, variable network quality, mobile hardware, privacy requirements, and traffic spikes around the product’s actual usage patterns.
Apply for AI Grants India
Building a production AI system in India? AI Grants India helps founders and teams discover support, funding, and opportunities for turning technically sound prototypes into deployable products.