Apply for AI Grants India

Ollama does not provide a general-purpose ollama.quantize() Python API. In practice, you quantize a supported model with llama.cpp tooling, export it as a GGUF file, and then create an Ollama model from that file. This distinction matters: it prevents broken installation instructions and gives you control over quality, memory use, and inference speed.

For Indian teams running models on developer laptops, on-premise servers, or cost-sensitive GPU instances, quantization can make local deployment practical. It is particularly useful when serving Hindi or other Indian-language models whose full-precision checkpoints may exceed available VRAM or RAM. If your broader goal is local serving, first review this guide to deploying large language models locally.

What quantization changes

Quantization stores model weights at lower numerical precision. A 16-bit model may be converted to 8-bit, 6-bit, 5-bit, or 4-bit representations. The result is usually a smaller file and lower memory pressure, with a trade-off in output quality and sometimes speed.

Common GGUF choices include:

• Q8_0: Closest to FP16 quality, but with relatively high memory use.
• Q6_K: A strong quality-first option when hardware allows it.
• Q5_K_M: A practical balance for many production workloads.
• Q4_K_M: A popular default for local use because it substantially reduces memory use while preserving useful quality.
• Q3 and below: Appropriate only when memory is severely constrained; test carefully for reasoning, multilingual output, and instruction following.

Quantization is not the same as distillation, pruning, or fine-tuning. It changes how existing weights are represented; it does not add new training data or teach the model new capabilities.

Check compatibility before converting

Ollama works with models supported by its underlying runtime, most commonly GGUF models for llama.cpp-compatible architectures. A standard Transformers checkpoint is not automatically ready for Ollama.

Before downloading or converting anything, confirm:

• The architecture is supported by your installed llama.cpp and Ollama versions.
• You have the original model configuration, tokenizer files, and any required vocabulary files.
• The model licence permits conversion and redistribution.
• You have enough temporary disk space for the source checkpoint, converted FP16 GGUF, and final quantized file.
• Your calibration and evaluation prompts represent actual use, including Indian languages if relevant.

For example, a Hindi-focused model should be tested with Devanagari, code-mixed Hindi-English, names, numbers, and regional terminology. You can compare this work with practical guidance on open-source small language models for Hindi.

Set up llama.cpp

Clone and build llama.cpp on the machine where conversion will run. Build instructions vary by operating system and accelerator, so use the project’s current documentation rather than copying an old binary from an unrelated release.

A typical Linux workflow looks like this:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

The resulting tools are usually placed under build/bin. Depending on the release, names may include convert_hf_to_gguf.py, llama-quantize, and llama-cli. Python conversion scripts may require packages such as PyTorch, Transformers, SentencePiece, or protobuf:

python -m venv .venv
source .venv/bin/activate
pip install -U pip torch transformers sentencepiece protobuf safetensors

Use a GPU-enabled build only when your conversion or validation workflow benefits from it. Quantization itself can be CPU-intensive, and a capable Indian cloud VM may be cheaper for a one-time conversion than maintaining a GPU server.

Convert a Transformers checkpoint to GGUF

Place the downloaded model in a directory such as ./model-source. Convert it to an FP16 GGUF file first:

python convert_hf_to_gguf.py ./model-source \
  --outfile ./model-f16.gguf \
  --outtype f16

The exact arguments can differ by model architecture and llama.cpp version. Read the converter’s help output and resolve tokenizer or architecture errors before proceeding. Do not rename an incompatible file and assume Ollama will accept it.

If the model uses custom code, unsupported layers, or a non-standard tokenizer, conversion may fail or produce incorrect results. In that case, check the model author’s recommended conversion path and llama.cpp support status. Vision-language models often require additional projector files and a different packaging process; a text-only GGUF recipe is not sufficient. For multimodal work, see related guidance on open-source vision-language models for Indian languages.

Quantize the GGUF file

Use the quantization binary generated during the llama.cpp build:

./build/bin/llama-quantize \
  ./model-f16.gguf \
  ./model-q4_k_m.gguf \
  Q4_K_M

For a quality comparison, create two variants rather than guessing:

./build/bin/llama-quantize ./model-f16.gguf ./model-q6_k.gguf Q6_K
./build/bin/llama-quantize ./model-f16.gguf ./model-q4_k_m.gguf Q4_K_M

The best choice depends on context length, concurrent users, KV-cache settings, and available RAM or VRAM. A model that fits in memory at startup may still fail under a long prompt because the KV cache adds a substantial runtime cost. If you are targeting phones or edge hardware, use the same measurement discipline described in this AI model optimisation guide for mobile devices.

Validate quality and runtime behaviour

Do not judge a quantized model only by file size or tokens per second. Test the tasks that matter to your application:

• Instruction following and refusal behaviour
• Hindi, English, and code-mixed prompts where applicable
• Long-context retrieval and summarisation
• Structured JSON or tool-call output
• Arithmetic, reasoning, and domain-specific terminology
• First-token latency, generation speed, and peak RAM or VRAM

Run the same prompt set against FP16, Q6, Q5, and Q4 variants. Save outputs and compare them with a fixed temperature and seed where supported. For production systems, measure p50 and p95 latency under realistic concurrency rather than a single interactive request. If your application produces repetitive answers after compression, review methods for reducing repetitive responses in LLM applications.

A simple runtime check with llama.cpp may look like:

./build/bin/llama-cli \
  -m ./model-q4_k_m.gguf \
  -p "Explain this in Hindi and English: ..." \
  -n 256

Watch for truncated output, garbled Unicode, hallucinated formatting, and failures on longer prompts. These issues may indicate an unsuitable quantization level, incorrect chat template, insufficient context configuration, or a conversion problem—not necessarily a weakness in the original model.

Import the quantized model into Ollama

Create a Modelfile in the same directory:

FROM ./model-q4_k_m.gguf

PARAMETER temperature 0.2
PARAMETER num_ctx 4096

SYSTEM You are a precise assistant. Answer in the language used by the user.

Then build the Ollama model:

ollama create my-model-q4 -f Modelfile
ollama run my-model-q4

Use a model name that records the variant, such as my-model-q4 or my-model-q6, so evaluations remain reproducible. Keep the original GGUF, conversion command, llama.cpp commit, Modelfile, licence, and benchmark prompts in version control or an artefact registry.

Common mistakes to avoid

• Installing ollama with pip and expecting it to convert or quantize checkpoints.
• Quantizing a Transformers directory without first converting it to a compatible GGUF.
• Choosing Q4 solely because it is smallest.
• Ignoring tokenizer, chat-template, or special-token configuration.
• Benchmarking only English prompts when the product serves Indian-language users.
• Deleting the FP16 intermediate before validating the quantized variants.
• Assuming GPU offload removes all system-RAM or KV-cache requirements.

A practical decision rule

Start with Q5_K_M or Q6_K when quality and multilingual reliability matter. Use Q4_K_M when memory is the binding constraint and evaluation shows acceptable degradation. Move lower only after measuring the effect on your real workload. For a Hindi assistant, medical workflow, or retrieval system, a slightly larger model that preserves terminology and instruction following is usually more valuable than the smallest possible file.

Quantization is therefore an engineering optimisation, not a one-click export. Convert with the correct toolchain, test representative prompts, document the result, and package the validated GGUF through Ollama.