Apply for AI Grants India

Sanjaya RLM is best deployed as an infrastructure project, not a one-line model-server install. Teams in India may choose local GPUs for lower latency, predictable costs, tighter control over sensitive prompts, or requirements around procurement and data handling. The right design depends on the model size, Hindi and English workload, concurrency, context length, and whether the cluster serves experimentation or production traffic.

This guide focuses on a practical 2026 deployment path for research labs, public-sector teams, Indian enterprises, and builders operating private GPU infrastructure. Before implementation, verify Sanjaya’s current model card, licence, supported architectures, tokenizer files, context window, and quantisation compatibility. Do not assume that settings for Llama or Mistral will work unchanged.

Start with a deployment profile

Define the workload before buying or allocating GPUs. Record:

• Model variant: parameter count, precision, and whether a chat or base checkpoint is required.
• Traffic: expected requests per minute, concurrent users, prompt length, and generated output length.
• Latency target: time to first token for interactive applications and total completion time for batch jobs.
• Availability: single-node development, scheduled HPC jobs, or a continuously running API.
• Data boundary: ordinary internal data, personal data, regulated records, or a fully air-gapped environment.

For teams new to private inference, the broader guide to deploying large language models locally provides useful background on model packaging, local APIs, and operational trade-offs.

Size GPUs around memory, not just parameters

A rough weight-memory estimate is:

• FP16 or BF16: approximately 2 bytes per parameter.
• INT8: approximately 1 byte per parameter, plus scales and runtime overhead.
• 4-bit: approximately 0.5 bytes per parameter, plus quantisation metadata.

Inference also needs memory for the KV cache, activations, CUDA workspace, tokenizer processes, and batching. Long Hindi prompts can consume more tokens than an equivalent English prompt, so leave headroom rather than allocating the entire card to weights.

For an 8B-class model, one 24GB GPU may work for quantised inference, while FP16 serving may require a larger card or tensor parallelism. RTX 4090 systems can be attractive for development and low-cost internal services, but they lack enterprise features such as ECC memory and may be difficult to operate at scale. A100 40GB or 80GB cards remain practical for production, and newer data-centre GPUs can improve throughput if the software stack supports them.

A 70B-class model generally needs multiple high-memory GPUs, especially in BF16. Plan for four to eight enterprise GPUs depending on precision, context length, concurrency, and acceptable headroom. NVLink within a node can reduce communication overhead; across nodes, use high-bandwidth, low-latency networking rather than ordinary 1GbE.

Build a reproducible software environment

Use a supported Linux distribution, a pinned NVIDIA driver, a compatible CUDA runtime, and a tested inference-server version. Ubuntu LTS is common, but the exact driver and CUDA combination should follow the chosen vLLM, PyTorch, and GPU support matrix.

A reliable baseline includes:

• Ubuntu LTS or the operating system standardised by your HPC team.
• NVIDIA Container Toolkit for containerised deployments.
• Docker or an approved alternative for repeatable images.
• vLLM for OpenAI-compatible serving and continuous batching.
• Prometheus-compatible metrics and centralised logs.
• A private model registry or controlled object store for weights.

Download the model and tokenizer through a controlled build machine, record checksums, scan the files, and promote them into the cluster registry. For defence, BFSI, and other restricted environments, test the complete image and model bundle before moving it into the air-gapped zone. Treat --trust-remote-code as a supply-chain decision: use it only when required, and review the repository code first.

Serve Sanjaya with vLLM

A starting command for a single node might look like this:

python -m vllm.entrypoints.openai.api_server \
  --model /models/sanjaya-rlm-8b \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 8192 \
  --served-model-name sanjaya-rlm \
  --host 0.0.0.0 \
  --port 8000

Treat these values as a baseline, not a guaranteed configuration. Set --tensor-parallel-size to the number of GPUs used by the replica, and lower --max-model-len if the KV cache causes out-of-memory failures. Increase it only after measuring real prompt lengths. If the checkpoint requires custom code or a different dtype, follow its documentation rather than forcing the flags above.

For quantised models, use the format explicitly supported by the checkpoint and vLLM version. AWQ, GPTQ, or other formats can deliver useful memory savings, but quality and kernel support vary by GPU generation. Evaluate Hindi quality, code-mixed prompts, numerals, names, and long documents—not only English benchmark scores. The local hardware fine-tuning guide is also relevant when you plan to adapt Sanjaya rather than only serve it.

Choose Slurm or Kubernetes deliberately

Use Slurm when GPUs are shared by researchers and jobs are scheduled. A production job should request the correct GPU type, reserve adequate CPU and RAM, write logs to durable storage, and expose the service only through approved network routes.

Use Kubernetes when you need replicas, rolling deployments, admission controls, autoscaling, and integration with enterprise observability. KServe or a comparable serving layer can standardise deployments, but do not autoscale blindly: model loading is expensive, and GPU fragmentation can make a nominally available cluster unusable.

For multi-node tensor parallelism, validate NCCL, RDMA, firewall rules, hostnames, and time synchronisation. Benchmark node-local and cross-node configurations separately. If interconnect performance is poor, a smaller model with more replicas may outperform a larger model spread across slow links.

Secure the private API and data path

Local hosting does not automatically mean secure hosting. Put the inference server behind an internal gateway with authentication, authorisation, rate limits, request-size limits, and TLS. Disable public exposure of the vLLM port. Separate operators, application identities, and end users; do not share a cluster-wide API key.

Log operational metadata without retaining sensitive prompt content by default. If prompts must be recorded for debugging, define retention, redaction, access approval, and deletion procedures. Encrypt model files, backups, and logs at rest. Apply OS patches, pin container images, restrict shell access, and monitor unexpected outbound connections.

The DPDP Act, contractual terms, sectoral rules, and an organisation’s own data-governance policy may all apply. Keeping traffic in India can support residency and control objectives, but it is not by itself a compliance guarantee. Map the data flows, identify the data fiduciary and processors where relevant, document access, and involve legal and security teams. A secure local-first operating system approach can complement, but cannot replace, application-level controls.

Benchmark the workload Indian users actually generate

Create a representative test set containing Hindi, English, Hinglish, Devanagari punctuation, names, dates, numbers, code, and long documents. Measure:

• TTFT: time to first token.
• Inter-token latency: responsiveness during generation.
• Throughput: output tokens per second per request and cluster-wide.
• Concurrency: quality and latency as simultaneous requests rise.
• GPU memory: weights, KV cache, and peak allocation.
• Failure rate: timeouts, malformed outputs, and rejected requests.

Compare BF16 and quantised versions using the same prompts and sampling settings. Test cold starts separately from warm serving. Do not publish a single tokens-per-second number without specifying GPU, model variant, context length, batch size, and quantisation.

Troubleshoot common failures

• Out of memory: reduce context length or concurrency, lower GPU utilisation, use a supported quantisation format, or add GPUs.
• Hindi quality degradation: confirm the Sanjaya tokenizer and chat template; do not substitute a generic tokenizer.
• Slow multi-GPU serving: inspect NCCL logs, PCIe or NVLink topology, and inter-node bandwidth.
• Uneven GPU use: check tensor-parallel configuration and whether another process occupies a card.
• Container starts but API fails: verify CUDA compatibility, model permissions, tokenizer files, and the server logs.
• Air-gapped installation breaks: mirror every Python wheel, container layer, model file, and dependency before isolation.

Production checklist

Before accepting real traffic, confirm that you have:

• A pinned model, tokenizer, image, driver, and inference-server version.
• A rollback copy of the previous working deployment.
• Authentication, network restrictions, audit logging, and secret rotation.
• Load tests using Indic and code-mixed prompts.
• Alerts for GPU memory, temperature, ECC errors, latency, queue depth, and failed requests.
• A documented incident and data-deletion process.
• Capacity plans for peak demand and GPU maintenance.

Once the serving layer is stable, Sanjaya can become a component in internal search, translation, support, and public-service systems. For larger deployments, the local language model deployment guide for Indian enterprises offers a useful architecture lens, while teams building departmental workflows can review guidance on integrating generative AI into local information systems.