Apply for AI Grants India

Open weight models make it possible to run, adapt, and deploy capable AI systems without depending entirely on a hosted API. The trade-off is operational: your team must manage model files, GPU memory, kernels, serving software, latency, and cost. A well-chosen open weight models GPU runtime can determine whether an application feels responsive and affordable—or becomes an expensive system that spends most of its time waiting on data and memory transfers.

This guide focuses on practical inference and fine-tuning decisions for builders in India, including teams working with local workstations, campus clusters, cloud GPUs, and shared production infrastructure.

What an open weight model runtime does

A GPU runtime is the software layer that turns model operations into work executed by the GPU. It typically includes:

• A framework such as PyTorch or JAX.
• GPU drivers and CUDA or ROCm libraries.
• Kernel libraries for matrix multiplication, attention, and communication.
• A model execution engine such as vLLM, TensorRT-LLM, ONNX Runtime, or llama.cpp.
• A serving layer that handles requests, batching, authentication, and monitoring.

The model’s parameter count is only the starting point. Runtime performance also depends on context length, concurrency, prompt and output distribution, quantization format, GPU memory bandwidth, and whether the workload is training, batch inference, or interactive serving.

Open weights do not automatically mean open licensing. Before deployment, check the model licence, acceptable-use terms, commercial restrictions, attribution requirements, and the provenance of training data. For Indian public-sector, health, education, or finance use cases, also document where prompts and generated outputs are stored and who can access them.

Start with a measurable performance target

Do not optimise for “GPU utilisation” alone. Define targets that reflect user experience and operating cost:

• Time to first token (TTFT): how quickly streaming generation begins.
• Inter-token latency: how rapidly tokens arrive after generation starts.
• Requests per second or tokens per second: useful for capacity planning.
• P95 and P99 latency: the experience of slower requests under load.
• Peak VRAM and host RAM: the resources needed to avoid out-of-memory failures.
• Cost per million input and output tokens: essential when comparing cloud GPUs.
• Quality at the selected precision: speed gains are not useful if accuracy collapses.

Create a fixed benchmark set before changing the runtime. Include short and long prompts, different output lengths, concurrent users, and the languages your application actually serves. An Indian-language assistant, for example, should be tested with Devanagari, Romanised text, and code-switching rather than only English prompts.

Choose the runtime around the workload

For large language model serving, vLLM is often a strong starting point because continuous batching and paged attention help improve throughput when requests arrive continuously. TensorRT-LLM can deliver excellent performance on supported NVIDIA hardware, but requires more careful engine building and version management. ONNX Runtime is useful when models are exported to ONNX and need portability across supported execution providers. For smaller models and edge devices, llama.cpp and related CPU/GPU backends can offer a simpler deployment path.

The best choice depends on the workload:

• Interactive chat: prioritise TTFT, streaming, and predictable tail latency.
• Batch document processing: prioritise throughput and aggressive batching.
• Fine-tuning: prioritise memory capacity, checkpointing, and distributed training support.
• Edge or offline deployment: prioritise binary size, quantisation, power use, and hardware compatibility.
• Multimodal systems: verify support for vision encoders, image preprocessing, and custom kernels.

Teams building complete systems should also review this practical guide to highly performant AI runtimes, especially when model serving is only one component of a larger application.

Fit the model into GPU memory

VRAM is usually the first constraint. A rough estimate for model weights is:

parameters × bytes per parameter

A 7-billion-parameter model requires roughly 14 GB for FP16 weights before accounting for the KV cache, runtime buffers, activations, and allocator overhead. Quantisation can reduce this substantially, but the real footprint depends on the format and implementation.

Use these levers deliberately:

• FP16 or BF16: a practical baseline on modern data-centre GPUs.
• INT8: useful when supported kernels preserve acceptable quality.
• 4-bit quantisation: often effective for local inference, with quality varying by model and task.
• Weight offloading: allows larger models to run, but PCIe transfers can damage latency.
• Tensor parallelism: splits computation across GPUs, at the cost of communication overhead.
• Shorter context limits: reduces KV-cache growth and prevents one request from consuming all memory.

Quantisation should be evaluated on your task, not only on a generic benchmark. Test retrieval accuracy, tool-call reliability, refusal behaviour, and Indian-language output quality. Keep an unquantised or higher-precision reference for comparison.

Improve the data path and batching

A powerful GPU can remain idle if tokenisation, image decoding, retrieval, or database calls are slow. Profile the entire request path rather than focusing only on the model kernel.

For batch and fine-tuning jobs:

• Use pinned host memory where supported.
• Pre-tokenise stable datasets and store efficient shards locally or on fast attached storage.
• Increase data-loader workers carefully; too many can exhaust CPU and RAM.
• Prefetch batches so the next transfer overlaps with current computation.
• Avoid repeated conversion between Python objects and tensors.
• Keep frequently used model files and datasets close to the GPU node.

For serving, continuous batching is generally more useful than a large fixed batch size. Set limits for maximum concurrent sequences, context length, and queued requests. Without admission control, a few long prompts can create a memory spike that affects every user.

Profile before changing settings

Use framework profilers, Nsight Systems or Nsight Compute on NVIDIA hardware, and runtime-specific metrics to identify the actual bottleneck. Track GPU duty cycle, memory bandwidth, kernel launch gaps, CPU wait time, data-transfer time, KV-cache usage, and queue depth.

Common diagnoses include:

• Low GPU utilisation and high CPU activity: tokenisation, preprocessing, or Python overhead is limiting the request.
• High VRAM use with falling throughput: context length or KV-cache pressure is too high.
• High compute utilisation but poor latency: batching or model parallelism may be unsuitable for interactive traffic.
• Large gaps between kernels: synchronisation, data copies, or inefficient operators are interrupting execution.
• Good benchmark results but poor production latency: concurrency, network calls, or tail behaviour were not measured.

Keep a reproducible environment file or container image. CUDA, driver, framework, and kernel-library mismatches can create silent performance regressions or prevent the model from loading at all.

Design for Indian infrastructure and budgets

GPU availability and pricing vary widely across Indian cloud providers, regional zones, and local vendors. Compare the full cost of ownership rather than hourly GPU price alone:

• GPU rental and minimum billing duration.
• Persistent storage for model checkpoints and datasets.
• Egress and inter-zone data-transfer charges.
• CPU, RAM, and high-speed local disk requirements.
• Managed Kubernetes or serving-platform costs.
• Engineering time spent maintaining custom builds.

For development, a smaller local GPU or rented single-GPU instance may be more economical than a multi-GPU setup. For production, reserve capacity only after measuring demand and request patterns. Cache model files, use autoscaling with warm replicas where latency matters, and scale by tokens or queue depth rather than CPU percentage alone.

Open-source ecosystems can reduce experimentation costs. Developers exploring deployment patterns may find this overview of building high-performance AI applications with open-source tools useful, while teams starting from a smaller scope can review open-source AI projects for student developers.

A practical deployment checklist

Before exposing an open weight model to users, verify:

• The licence permits your intended use.
• The runtime, driver, and GPU combination is supported and pinned.
• A benchmark covers realistic languages, context lengths, and concurrency.
• Quantised output has been compared with a quality baseline.
• VRAM limits, queue limits, timeouts, and cancellation are enforced.
• Prompts and outputs are protected through access controls and retention rules.
• Metrics cover TTFT, throughput, P95 latency, errors, OOM events, and cost.
• A fallback model or degraded mode exists for GPU exhaustion.
• Model versions, prompts, adapters, and runtime builds are reproducible.

For agentic applications, production deployment adds tool permissions, sandboxing, retries, and observability. See the guide on deploying open-source AI agents in production before allowing a model to call external systems.

Conclusion

The fastest open weight model is not necessarily the largest GPU or the newest kernel. Performance comes from matching model size, precision, context length, batching strategy, runtime, and hardware to a measured workload. Start with a representative benchmark, reduce memory pressure, profile the full pipeline, and make cost and quality visible alongside speed. That process produces a runtime your team can operate—not just a benchmark result that works on one developer machine.