How to Run a Gujarati Small Language Model Offline

Gujarati applications often handle information that should not leave the device: customer conversations, public-service records, health queries, legal documents, and internal business data. A small language model (SLM) can process this text locally, reducing latency and cloud costs while keeping the deployment under your control.

This guide explains how to run a Gujarati small language model offline on a laptop, workstation, or edge computer. It focuses on inference rather than training a model from scratch, which is usually the sensible starting point for Indian-language applications.

Choose the right Gujarati model

Start by defining the task. A model used for Gujarati text completion has different requirements from one used for classification, summarisation, translation, or retrieval-augmented question answering.

When comparing candidate models, check:

Gujarati coverage: Review the model card, tokenizer vocabulary, training languages, and examples. “Multilingual” does not guarantee strong Gujarati output.
Model size: A 1B–4B parameter model is more practical for CPU or modest GPU deployment than a 7B or larger model.
License: Confirm that the licence permits commercial use, redistribution, and any fine-tuning you plan to do.
Context length: Longer context increases memory use. Select the smallest window that fits your documents.
Instruction tuning: For chat, extraction, and structured responses, use an instruction-tuned checkpoint rather than a base model.
Tokenizer behaviour: Test Gujarati spelling, punctuation, numerals, and mixed Gujarati-English input. Inefficient tokenisation can make a small model expensive to run.

For background on tokenisation, datasets, and evaluation in Indian languages, see this builder’s guide to low-resource Indic NLP. A Hindi SLM is not automatically a Gujarati solution, but the model-selection and benchmarking principles in this guide to open-source Hindi small language models are useful comparisons.

Hardware and offline preparation

A practical baseline is 16 GB RAM, a modern 4–8 core CPU, and at least 10–20 GB of free disk space for model files, caches, and test data. An 8 GB machine can run a heavily quantised model, but response speed and context length may be limited. A GPU with 6–12 GB of VRAM improves latency, although it is not mandatory.

Prepare the machine while it still has internet access:

Install Python 3.10 or 3.11 in a virtual environment.
Download the model repository, tokenizer files, configuration, and licence.
Cache all Python wheels and runtime dependencies.
Copy a small Gujarati test set to the target device.
Record model hashes and package versions for reproducible deployment.
Disable telemetry and remove any code that calls external APIs.

Do not assume that downloading a model once makes an application offline. Some libraries attempt to fetch missing configuration files or tokenizer assets at runtime. Test with networking disabled before deployment.

Install a local inference runtime

For a Transformers-compatible checkpoint, create an isolated environment and install the required packages:

python -m venv .venv
source .venv/bin/activate          # Linux/macOS
# .venv\\Scripts\\activate       # Windows

pip install torch transformers accelerate safetensors

If the device is CPU-only, install the appropriate CPU build of PyTorch. For lower-memory deployment, use a quantised GGUF model with a runtime such as llama.cpp or an equivalent local server. Quantisation commonly reduces a model from several gigabytes to a fraction of that size, with a quality trade-off that must be measured on Gujarati examples.

Keep model files in a fixed local directory and set offline flags when using Hugging Face libraries:

export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1

On Windows, set the same variables through the system environment settings or in the shell session. These flags help expose missing files instead of silently attempting a network request.

Run Gujarati inference locally

Use a causal language model for generation. Replace the path below with a local model directory, not an online repository identifier:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_path = "./models/gujarati-slm"

tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    local_files_only=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    local_files_only=True,
    torch_dtype=torch.float32,
    device_map="auto"
)

prompt = "ગુજરાતમાં નાના વ્યવસાય માટે ડિજિટલ ચુકવણીના ત્રણ ફાયદા સમજાવો."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For a chat or instruction model, apply its documented chat template rather than concatenating arbitrary role labels. Keep prompts short, specify the desired Gujarati script, and request a defined output format for extraction tasks. If the model generates Hindi, English, or malformed Gujarati, test a stronger Gujarati prompt before changing the model.

Reduce memory and improve speed

Quantisation is usually the first optimisation to try. Common 4-bit formats can make local inference possible on laptops, while 8-bit formats generally preserve more quality. Benchmark at least two settings using the same prompts and record:

Time to first token
Tokens per second
Peak RAM and VRAM
Output length and truncation rate
Gujarati factual accuracy
Rate of code-switching or script errors

For CPU deployments, use a runtime built for the target processor and limit the number of concurrent requests. For mobile or edge use, consider the techniques covered in this AI model optimisation guide for mobile devices. Avoid increasing context length merely because the model supports it; unused context consumes memory and slows every request.

Evaluate Gujarati quality before deployment

A few impressive demonstrations are not enough. Build a small, representative test set containing customer questions, formal Gujarati, colloquial Gujarati, Gujarati-English code-switching, names, dates, currency values, and domain-specific terms.

Evaluate both quality and safety:

Does the answer preserve Gujarati meaning and grammar?
Are numbers, names, and negations copied accurately?
Does summarisation omit important qualifiers?
Does the model invent citations, policies, or facts?
Does it expose text from the prompt or memorised training data?
Does it refuse unsafe or sensitive requests consistently?

For business use, compare the SLM with a deterministic baseline such as keyword rules or a traditional classifier. A smaller model is valuable when it is reliable, not simply when it is inexpensive.

Build an offline application safely

Place the model behind a local service rather than embedding inference logic throughout the application. A simple HTTP API on 127.0.0.1 can support a desktop interface, internal tool, or kiosk while keeping traffic on the device. Add request limits, timeouts, structured logging, and a clear model version.

Encrypt sensitive files at rest, restrict access to model and prompt logs, and avoid storing raw user text unless it is required. If the application handles government, health, financial, or employee information, define retention and deletion rules before collecting data. Offline operation improves privacy, but it does not remove the need for access controls.

For multilingual products that combine text with images or scanned documents, assess open-source vision-language models for Indian languages separately; a Gujarati text SLM alone will not reliably read documents or photographs.

Common problems and fixes

Out-of-memory errors: Use a smaller checkpoint, 4-bit quantisation, shorter context, or CPU offloading.
Missing files in offline mode: Re-download the complete repository, including tokenizer and generation configuration, then test with the network disconnected.
Poor Gujarati output: Compare tokenisers, use Gujarati instruction data, and test a multilingual model against a Gujarati-focused checkpoint.
Slow CPU inference: Reduce max_new_tokens, use a quantised runtime, and avoid simultaneous requests.
Repetitive answers: Lower the sampling temperature, add an explicit answer structure, or use an instruction-tuned model.
Incorrect facts: Add retrieval from a local, versioned knowledge base rather than expecting the model to memorise changing information.

A practical deployment checklist

Before handing the system to users, confirm that:

The model, tokenizer, licence, and dependencies are stored locally.
The application works with Wi-Fi and Ethernet disabled.
Gujarati test cases meet your accuracy threshold.
Memory use remains safe under realistic prompts.
Logs do not capture unnecessary personal data.
Model updates are versioned, tested, and reversible.
Users can report incorrect Gujarati output for review.

Running a Gujarati SLM offline is achievable on modest hardware, but the strongest results come from disciplined model selection and evaluation. Start with a narrow task, quantify Gujarati performance, then optimise the runtime and expand only after the local system is dependable.

Apply for AI Grants India