Apply for AI Grants India

What you will build

This guide presents a reproducible workflow for evaluating an Odia-capable language model on IndicGenBench with Hugging Face. It focuses on the parts that commonly produce misleading results: using the official split, matching the task’s prompt format, generating with fixed settings, selecting suitable metrics, and reporting failures rather than only a headline score.

IndicGenBench is useful because it places Indian-language generation and understanding in a common evaluation setting. For Odia, careful benchmarking matters especially because script handling, limited training data, transliteration, spelling variation, and uneven tokenizer coverage can affect results. If you are comparing several Indian-language models, first review this practical framework for benchmarking multilingual LLMs in India so your experiment has a clear evaluation protocol.

1. Confirm the benchmark task and licence

Before installing packages, inspect the IndicGenBench repository and documentation associated with the version you intend to use. Do not assume that every task is a classification problem or that every dataset uses the same fields. Depending on the task, examples may contain an instruction, context, source sentence, reference answer, label, or multiple accepted outputs.

Record the following in your experiment notes:

• IndicGenBench commit, release, or dataset version
• Odia task name and official split names
• Dataset licence and model licence
• Required prompt or instruction template
• Reference-output format and evaluation script
• Whether the task is zero-shot, few-shot, fine-tuned, or retrieval-augmented

Keep the test set untouched. If you tune prompts, decoding parameters, or checkpoints against the test set, the resulting score is no longer a clean test estimate. For broader dataset selection, the Indian language LLM benchmark datasets guide is a useful companion.

2. Create a clean Hugging Face environment

Use a virtual environment and pin the main dependencies. Versions can affect tokenisation, chat templates, generation defaults, and metric implementations.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "torch>=2.2" "transformers>=4.45" "datasets>=2.20" \
  accelerate evaluate sentencepiece sacrebleu

For a CUDA machine, install the PyTorch build that matches your driver. On limited hardware, use device_map="auto", reduced precision where supported, and a smaller model. Quantisation can make evaluation possible on a single GPU, but compare quantised and full-precision runs when score differences are small.

Clone the official benchmark repository only if its evaluation scripts are required:

git clone <official-IndicGenBench-repository-url>
cd <repository-directory>
git rev-parse HEAD

Replace the placeholder with the repository URL from the current IndicGenBench documentation. Avoid copying an unverified evaluate.py from a blog post: the exact script and preprocessing rules determine the reported score.

3. Load and inspect the Odia data

If the benchmark is published as a Hugging Face dataset, use its documented identifier. If it is stored locally, load the file format supplied by the benchmark rather than converting it casually.

from datasets import load_dataset

# Use the official identifier or local file described by IndicGenBench.
data = load_dataset("<official-dataset-id>", name="odia")
print(data)
print(data["test"][0])

Check Unicode before running inference. Odia text should remain intact after loading, serialisation, and prompt construction.

sample = data["test"][0]
for key, value in sample.items():
    if isinstance(value, str):
        print(key, repr(value[:200]))

Look for empty fields, duplicated examples, unexpected English text, inconsistent whitespace, and accidental transliteration. Do not normalise away meaningful punctuation or alter the reference answers unless the official evaluator explicitly requires it. Save a small audit file containing example IDs and the final prompts used for evaluation.

4. Load the model and tokenizer correctly

Choose a model that actually supports Odia, not merely one marketed as multilingual. Check its model card for language coverage, training data, context length, instruction tuning, and known limitations. For generation tasks, use a causal language model and its chat template where available.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "<hugging-face-model-id>"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
model.eval()

Use the benchmark’s exact prompt format. A generic prompt can create a different task from the one IndicGenBench intends to measure. For chat models, construct messages and apply the tokenizer’s template; for base models, follow the benchmark’s plain-text format.

Also measure tokenisation on Odia examples. Excessive token fragmentation can increase cost and reduce the effective context available to the model. Compare token counts across models before interpreting quality differences.

5. Run deterministic, auditable inference

Start with greedy decoding or the benchmark’s prescribed decoding settings. Fix the random seed, disable sampling unless the protocol requires it, and save predictions alongside example IDs.

from transformers import set_seed

set_seed(2026)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
prediction = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

The correct max_new_tokens depends on the task. Too small a value truncates answers; too large a value can encourage irrelevant continuation. Remove prompt echoes only according to the model’s output format, and never silently delete Odia characters during post-processing.

For every run, store model ID and revision, tokenizer revision, hardware, precision, prompt template, decoding parameters, seed, and software versions. This is essential when comparing results with benchmarking multilingual LLMs in India or publishing a model report.

6. Use the official evaluator and add targeted checks

Run the IndicGenBench evaluator exactly as documented. A typical command may resemble the following, but the actual flags depend on the repository version:

python <official-evaluation-script>.py \
  --predictions outputs/odia_predictions.jsonl \
  --references <official-odia-test-file> \
  --task <odia-task-name>

Do not substitute accuracy for a generation metric without justification. Depending on the task, useful measures may include exact match, token-level F1, BLEU, chrF, ROUGE, or a task-specific score. For Odia generation, character-aware metrics can be informative when word segmentation is inconsistent, but automated metrics should be paired with human inspection.

Create an error-analysis table with:

• Example ID and task category
• Input, reference, and model output
• Metric score
• Error type: omission, hallucination, translation drift, spelling, script, or formatting
• Whether the error changes meaning

Sample high-scoring and low-scoring outputs manually. Check named entities, numbers, negation, honorifics, code-mixed text, and culturally specific references. A model can achieve a respectable aggregate score while failing on precisely the cases users care about.

7. Report results responsibly

Publish a table that includes the model, prompt regime, decoding settings, precision, test-set version, metric values, and confidence or variation across repeated runs where applicable. If you compare against other Indian-language systems, place the findings alongside relevant work such as benchmarking NLP models for Telugu and Sanskrit, while keeping language-specific results separate.

State limitations clearly: test-set size, domain mismatch, possible data contamination, evaluator sensitivity, and whether outputs were post-processed. Do not claim that a benchmark score proves production readiness. For an Odia application, follow up with in-domain tests covering government services, education, agriculture, customer support, or the domain you intend to serve.

Common mistakes to avoid

• Loading a dataset split different from the official Odia test split
• Evaluating a classification head on a generation task
• Using English prompts when the benchmark specifies Odia or bilingual prompts
• Sampling outputs and reporting one lucky result
• Comparing scores from different preprocessing or evaluator versions
• Stripping Unicode punctuation or normalising references inconsistently
• Fine-tuning on test examples or publicly leaked evaluation data
• Reporting only the aggregate score without qualitative Odia error analysis

A reliable Hugging Face evaluation is less about a single command than about preserving the benchmark contract from input loading through scoring. With pinned versions, official data, deterministic generation, and transparent analysis, IndicGenBench can provide a useful baseline for improving Odia NLP rather than merely producing another leaderboard number.