Apply for AI Grants India

IndicGLUE can help you compare Hindi language models on standardised language-understanding tasks, but only if the evaluation setup is correct. Dataset configuration, label mapping, tokenisation, split selection, and metric choice can all change the result. This guide presents a reproducible workflow for benchmarking a Hindi model with Hugging Face tools in 2026, while also showing how to avoid common evaluation mistakes.

What IndicGLUE measures

IndicGLUE is a benchmark suite for evaluating language understanding across Indian languages. Depending on the task and the current dataset implementation, Hindi evaluation may cover sentence classification, natural language inference, sentiment, question answering, or related supervised tasks. Do not assume every IndicGLUE configuration has the same schema or metric. Inspect the dataset card and feature definitions before writing your preprocessing function.

For teams building Hindi products, a benchmark score is only one signal. It is useful for comparing checkpoints, but it does not replace testing on your target domain—such as government forms, education content, customer support, or code-mixed Hindi. If you are comparing compact Hindi models, pair this workflow with a review of open-source small language models for Hindi.

Prepare a reproducible environment

Use a fresh virtual environment and pin the main packages. APIs in transformers and datasets change over time, so recording versions is essential when publishing results.

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install "torch" "transformers" "datasets" "evaluate" "accelerate" "scikit-learn"

Record the environment and hardware:

python - <<'PY'
import platform, torch, transformers, datasets
print(platform.platform())
print("torch", torch.__version__)
print("transformers", transformers.__version__)
print("datasets", datasets.__version__)
print("cuda", torch.cuda.is_available())
PY

Set random seeds, note the GPU model, and save the exact model revision. A benchmark should be rerunnable by another researcher rather than tied to an unrecorded local installation.

Choose and inspect the Hindi model

For classification, load a model with AutoModelForSequenceClassification. A base encoder such as IndicBERT or a multilingual encoder may be suitable, but check its pretraining languages, tokenizer behaviour, licence, and expected input format. A base checkpoint without a task-specific classification head generally needs fine-tuning before its score is meaningful.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "ai4bharat/indic-bert"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    num_labels=2,
    problem_type="single_label_classification",
)

If the checkpoint already has a task-specific head, preserve its label mapping. Otherwise, define the number of labels from the dataset and save an explicit id2label/label2id mapping. Do not silently compare a Hindi-only model with a multilingual model under different fine-tuning conditions.

Load the correct IndicGLUE configuration

Dataset names and configuration identifiers can change. First inspect the available configurations rather than copying an outdated task name:

from datasets import get_dataset_config_names, load_dataset

configs = get_dataset_config_names("ai4bharat/IndicGLUE")
print(configs)

Use the dataset identifier and task configuration shown in the current Hugging Face dataset card. Then inspect columns, labels, and splits:

dataset = load_dataset("ai4bharat/IndicGLUE", "YOUR_TASK_CONFIG")
print(dataset)
print(dataset["train"].features)
print(dataset["train"][0])

Some tasks use one text field; others use sentence pairs, question-context fields, or answer spans. Build preprocessing from the actual schema. If a dataset is unavailable under that identifier, use the official IndicGLUE repository or dataset card and document the source, commit, and access date instead of substituting an unrelated corpus.

Tokenise without leaking information

For a single-sentence task:

def tokenize_batch(batch):
    return tokenizer(
        batch["text"],
        truncation=True,
        max_length=256,
    )

tokenized = dataset.map(tokenize_batch, batched=True)

For sentence-pair tasks, pass both fields:

def tokenize_pairs(batch):
    return tokenizer(
        batch["premise"],
        batch["hypothesis"],
        truncation=True,
        max_length=256,
    )

Avoid padding="max_length" during mapping unless fixed-length inputs are required. Dynamic padding usually reduces memory use. Use DataCollatorWithPadding at evaluation time, and verify that Devanagari text has not been normalised or altered differently across splits. Keep the official validation or test split intact; never tune on the test set.

Fine-tune and evaluate with Trainer

After aligning the dataset label column with the model’s expected labels field, define task-appropriate metrics. Accuracy alone can hide class imbalance, so include macro-F1 where relevant.

import evaluate
import numpy as np
from transformers import (
    DataCollatorWithPadding, Trainer, TrainingArguments
)

accuracy = evaluate.load("accuracy")
f1 = evaluate.load("f1")
collator = DataCollatorWithPadding(tokenizer=tokenizer)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {
        "accuracy": accuracy.compute(
            predictions=predictions, references=labels
        )["accuracy"],
        "macro_f1": f1.compute(
            predictions=predictions,
            references=labels,
            average="macro",
        )["f1"],
    }

args = TrainingArguments(
    output_dir="./indicglue-hindi-results",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="macro_f1",
    greater_is_better=True,
    per_device_eval_batch_size=32,
    report_to="none",
    seed=42,
)

trainer = Trainer(
    model=model,
    args=args,
    eval_dataset=tokenized["validation"],
    processing_class=tokenizer,
    data_collator=collator,
    compute_metrics=compute_metrics,
)

results = trainer.evaluate()
print(results)

For a true benchmark, fine-tuning is normally performed on the official training split, with model selection on validation and one final evaluation on test if labels are available. Run at least three seeds for close comparisons and report mean and standard deviation. If you are studying several Indic languages, the methodology in benchmarking NLP models for Telugu and Sanskrit offers a useful comparison framework.

Report results responsibly

A useful report should include:

• Model name, revision, tokenizer, and vocabulary details.
• IndicGLUE task and configuration, dataset version, and split used.
• Maximum sequence length, truncation policy, batch size, learning rate, epochs, and seed.
• Accuracy, macro-F1, weighted-F1, or task-specific metrics.
• Number of runs, mean and standard deviation, hardware, and runtime.
• Whether text was native Hindi, transliterated, code-mixed, or filtered.

Inspect the confusion matrix and review errors by label, text length, spelling variation, dialect, and code-mixing. Hindi benchmarks may contain distributional gaps that are invisible in one aggregate score. Check for duplicate examples, train–validation overlap, annotation ambiguity, and examples where truncation removes the decisive evidence.

Common failure modes

• Wrong configuration: the script loads a similarly named dataset with different labels or language coverage.
• Incorrect label IDs: label names are mapped in a different order between training and evaluation.
• Invalid metric: accuracy is reported for a task requiring F1, exact match, or span-level scoring.
• Test-set tuning: hyperparameters are repeatedly adjusted against the final test split.
• Unfair comparison: models use different data, epochs, sequence lengths, or seeds.
• Tokenizer mismatch: the model receives text processed with a tokenizer from another checkpoint.
• Unsupported claims: a high benchmark score is presented as proof of broad Hindi capability.

When the model must run on affordable Indian hardware or at the edge, benchmark latency and memory separately from task quality. Techniques covered in AI model optimisation for mobile devices can reduce deployment cost, but always re-evaluate after quantisation or distillation because compression can affect Hindi and code-mixed inputs unevenly. For local experiments, deploying large language models locally provides complementary guidance on resource planning.

Final checklist

Before publishing a Hindi IndicGLUE result, confirm that the dataset card, configuration, model revision, environment, preprocessing code, seed, and metric implementation are recorded. Release evaluation scripts and prediction files where licensing permits. Most importantly, supplement the benchmark with a small, representative Hindi test set from your intended application. That combination gives builders a more honest view of model quality than a single leaderboard number.