How to Benchmark Code-Mixed Indian Language Models on Hugging Face

Code-mixed Indian text is not simply multilingual text placed in the same dataset. A single message may combine Hindi, English, transliterated Hindi, emojis, local slang, and platform-specific spelling. That combination creates evaluation problems that a single accuracy score can hide.

This guide explains how to benchmark code mixed Indian language models on Hugging Face in a way that is reproducible, fair, and useful for production decisions. It covers task design, dataset preparation, Hugging Face tooling, metrics, controlled comparisons, and error analysis for builders working with Indian users.

Start by defining the evaluation question

Before selecting a model, write down what you need to measure. “Best model” is not a meaningful benchmark result without a task, user segment, and deployment constraint.

Common evaluation tasks include:

Sentiment and intent classification: Useful for support tickets, commerce, fintech, and social listening.
Named entity recognition: Tests whether a model can identify people, places, organisations, products, and amounts across scripts.
Toxicity and safety detection: Requires careful handling of reclaimed language, slang, and regional insults.
Retrieval and semantic similarity: Measures whether a system groups equivalent Hindi-English and English queries.
Generation: Covers translation, summarisation, response drafting, and question answering.

Specify the language pair or mix, script, domain, and user context. Hindi written in Devanagari is a different evaluation case from Hindi transliterated into Latin script, and both differ from Hinglish customer support messages. Teams working with low-resource Indic data should also review this builder’s guide to low-resource Indic NLP before finalising a benchmark.

Build a trustworthy dataset

Hugging Face Datasets makes it easy to load and process data, but it does not make a weak dataset reliable. Dataset quality is the main determinant of benchmark quality.

Create explicit fields such as:

text: the original user input, preserving spelling, punctuation, emojis, and script.
label: the task target, with a documented label mapping.
language_tags: token- or sentence-level language annotations where available.
script: for example, Latin, Devanagari, Bengali, or mixed script.
domain: such as payments, education, retail, or public services.
source_split: provenance and collection period, without exposing personal information.

Use speaker-, user-, and conversation-level separation wherever possible. Randomly splitting messages can place near-duplicates from the same conversation in both training and test sets, producing inflated scores. Deduplicate normalised text, transliteration variants, and copied customer-service templates before splitting.

Hold out a test set that reflects actual use. Include spelling variation, abbreviated words, code-switch points, regional vocabulary, Romanised Indian languages, and messages containing English product names. If labels are subjective, use multiple annotators and report agreement. Do not silently discard disagreement: it may indicate an unclear taxonomy rather than bad annotation.

For open datasets, verify the licence, consent, personally identifiable information policy, and annotation documentation before uploading or redistributing them through the Hub.

Prepare a reproducible Hugging Face environment

A practical baseline uses Transformers, Datasets, Evaluate, and PyTorch:

pip install -U transformers datasets evaluate accelerate torch sentencepiece

Pin versions in requirements.txt or a lock file. Record the model revision, tokenizer revision, dataset version, random seeds, hardware, maximum sequence length, and batch size. Uploading a model or dataset card to the Hugging Face Hub is useful, but the card should document limitations and test conditions rather than only reporting the highest score.

Load data and tokenise it with the model’s own tokenizer:

from datasets import load_dataset
from transformers import AutoTokenizer

model_id = "your-org/your-code-mixed-model"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
dataset = load_dataset("your-org/your-benchmark")

def tokenize(batch):
    return tokenizer(
        batch["text"],
        truncation=True,
        max_length=256,
    )

tokenized = dataset.map(tokenize, batched=True)

Inspect tokenisation before training or evaluation. Compare token counts for Devanagari, Latin transliteration, English-heavy text, and emoji-rich messages. A model that performs well on short, clean examples may be expensive or ineffective on the long, noisy inputs found in Indian apps.

Select metrics that match the task

For classification, report macro-F1, per-class precision and recall, and a confusion matrix. Macro-F1 prevents a dominant “neutral” or “other” class from hiding poor performance on minority intents. Accuracy can be included, but it should not be the headline metric for imbalanced data.

For named entity recognition, use entity-level precision, recall, and F1 rather than token accuracy. State whether partial span matches count. For retrieval, report recall at K, mean reciprocal rank, or nDCG. For generation, combine automatic and human evaluation: BLEU or ROUGE may provide useful signals, but they do not reliably capture meaning, politeness, code-switch naturalness, or factuality.

Break every headline metric down by:

Language mixture and script.
Input length and proportion of English tokens.
Transliteration quality and spelling noise.
Domain, geography, and user segment.
Frequent versus rare intents or entities.

This exposes whether a model is learning language competence or merely memorising common English phrases and templates.

Compare models under controlled conditions

A fair benchmark changes one major variable at a time. Keep the dataset version, splits, preprocessing, label mapping, sequence length, and evaluation code constant. Compare a multilingual baseline, an Indic-focused model, and your fine-tuned model where possible. Include a simple majority-class or keyword baseline; a complex model should beat it by a meaningful margin.

For supervised evaluation, the Hugging Face Trainer API can standardise training and prediction:

import evaluate
from transformers import TrainingArguments, Trainer

f1 = evaluate.load("f1")

def compute_metrics(pred):
    predictions = pred.predictions.argmax(axis=-1)
    return f1.compute(
        predictions=predictions,
        references=pred.label_ids,
        average="macro",
    )

args = TrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="f1",
    seed=42,
)

Use several seeds for small datasets and report the mean and spread. If the difference between two models is smaller than run-to-run variation, do not present it as a decisive improvement. For generation and chat systems, freeze prompts, decoding settings, context windows, and safety filters. Measure latency, memory use, and cost alongside quality; a slightly weaker model may be the better choice for an Indian-language product with strict inference budgets.

Analyse failures, not just scores

After evaluation, export misclassified and low-confidence examples. Group failures into categories such as transliteration ambiguity, named-entity confusion, negation, sarcasm, mixed-script input, regional vocabulary, and unsafe interpretation. Review examples with native or highly proficient speakers of the relevant languages. Machine translation back into English can assist triage, but it should not replace native-language review.

Test robustness with controlled perturbations: spelling changes, punctuation removal, extra English words, emojis, and script conversion. Also evaluate out-of-domain data. A model trained on social media Hinglish may fail on concise banking or healthcare messages even when its aggregate test score is strong.

Document examples that should not be used. Public leaderboards can encourage optimisation for narrow datasets, while code-mixed text may contain personal data or culturally sensitive content. Publish a model card with intended use, excluded use cases, known language and script gaps, annotation limitations, and responsible deployment guidance.

A practical reporting checklist

A useful benchmark report should include:

Dataset source, licence, collection period, size, and label definitions.
Exact train, validation, and test split strategy.
Model and tokenizer identifiers with revisions.
Preprocessing, maximum length, seeds, and hardware.
Overall and subgroup metrics with confidence intervals where feasible.
Baselines, ablations, and statistical or seed-based uncertainty.
Error categories, representative examples, and privacy safeguards.
Inference latency, memory requirements, and deployment constraints.

For teams building Indian-language products, benchmark results should feed directly into release gates and monitoring. Track drift in scripts, slang, intents, and domains after launch rather than treating evaluation as a one-time academic exercise. Related work on Indian open-source AI developer projects can also help teams find reusable evaluation infrastructure and community practices.

FAQ

What is the most important metric for code-mixed classification?

Usually macro-F1, supported by per-class results and subgroup breakdowns. The correct choice depends on the cost of false positives and false negatives in your application.

Should I translate code-mixed text before evaluation?

Not by default. Translation can remove the very phenomena you need to measure and introduce new errors. Evaluate original text first, then optionally compare with a translation-based pipeline as a separate experiment.

Can I use a random train-test split?

Only when examples are independent. For conversational, user-generated, or templated data, group by user, conversation, source, or time to reduce leakage.

Is a high benchmark score enough for deployment?

No. Validate privacy, robustness, latency, cost, safety, and performance across scripts, dialects, domains, and user groups before production release.

Apply for AI Grants India