Apply for AI Grants India

Indian-language AI needs evaluation that reflects how people actually write and speak across scripts, dialects, domains, and code-mixed settings. A benchmark built only around English-style assumptions can hide serious weaknesses in tokenisation, translation, named-entity recognition, speech transcripts, and safety. This guide explains how to create a custom Hugging Face benchmark for Indian languages that other researchers and builders can reproduce, compare, and improve.

Start with a precise benchmark brief

Write a one-page brief before collecting data. Define:

• Languages and varieties: Specify language, script, region, dialect where relevant, and whether Romanised text or code-mixing is included.
• Tasks: Choose a small set with a clear user need, such as classification, NER, question answering, summarisation, translation, transliteration, or toxicity detection.
• Users and domains: News, government services, education, healthcare, commerce, and customer support have different vocabulary and risk profiles.
• Evaluation unit: Decide whether the unit is a sentence, document, conversation turn, utterance, or translation pair.
• Success criteria: Define the minimum acceptable performance and the failure modes that matter most—not only a single leaderboard score.

Avoid combining unrelated tasks into one headline score. Publish per-language and per-task results first, then provide an aggregate only with a transparent weighting scheme. This is particularly important when one language has far more examples than another.

A benchmark intended for practical systems should also reflect deployment conditions. For example, a customer-support model may face spelling variation, short messages, English words embedded in an Indian-language sentence, and noisy mobile input. If the benchmark excludes these cases, it will overstate real-world performance. Teams planning production systems can use the same design discipline described in best practices for fine-tuning LLMs on custom data.

Build a governed, representative dataset

Use licensed, traceable sources. Potential inputs include public government documents, permissively licensed corpora, carefully reviewed web content, synthetic examples used only where appropriate, and newly collected annotations. Record the source, licence, collection date, domain, language label, script, and processing history for every example.

For Indian languages, representation requires more than a language column. Capture metadata such as:

• Script and transliteration format
• Region or dialect, where consent and privacy allow
• Code-mixing and borrowed vocabulary
• Formal versus conversational register
• Domain and document type
• Spelling, OCR, and speech-transcription noise

Remove personal information and sensitive content unless it is essential to the task and covered by a documented governance process. Do not scrape private material or assume that publicly accessible text is automatically suitable for redistribution. Publish a dataset card with provenance, consent or licence details, known gaps, intended uses, and prohibited uses.

Create train, validation, and test splits at the speaker, author, document, or source level where possible. Random sentence-level splitting can leak near-duplicates and make results look better than they are. Keep a hidden test set for public evaluation if leaderboard integrity matters. Deduplicate across splits using normalised text, hashes, and near-duplicate detection; account for Unicode variants, punctuation, and transliteration differences.

Design annotation that survives linguistic variation

Write annotation guidelines with examples for every language and task. Translators or annotators should not be expected to infer English-centric categories. For NER, clarify treatment of honorifics, compound names, organisations, locations, and ambiguous boundaries. For sentiment or intent, include culturally specific expressions and allow an uncertainty label when a sentence is genuinely ambiguous.

Use at least two annotators for a meaningful sample and report agreement by language and label. Resolve disagreements with an adjudication protocol rather than silently choosing one answer. Track annotator training, compensation, review time, and escalation paths. Community participation is valuable, but it should be structured and fairly compensated—not treated as free quality control.

Choose metrics that expose failure

Use task-appropriate metrics and report disaggregated results:

• Classification: Macro-F1, per-class precision and recall, and calibration where predictions drive decisions.
• NER: Entity-level precision, recall, and F1 with clearly documented matching rules.
• Question answering: Exact match and token-level F1, supplemented by human review for valid alternative answers.
• Translation: BLEU or chrF as diagnostics, plus COMET or human evaluation where feasible. Check adequacy, fluency, terminology, and gender or politeness errors.
• Summarisation: Automatic scores alongside factuality, coverage, and harmful-content review.
• Generation: Human preference protocols, rubric-based scoring, and targeted tests for hallucination, refusal, and code-mixing.

Never report only an average across languages. Include confidence intervals or bootstrap estimates, sample counts, and variance across random seeds. A small improvement may not be meaningful if it disappears across seeds or affects only the highest-resource language.

Implement the benchmark with Hugging Face

Keep the repository modular so users can run data preparation, evaluation, and reporting independently. A practical structure is:

benchmark/
├── data/                 # download instructions or gated files
├── configs/              # language and task definitions
├── src/                  # preprocessing and evaluation code
├── scripts/              # run and aggregation commands
├── tests/                # metric and leakage tests
├── dataset_card.md
├── benchmark_card.md
└── README.md

Use the datasets library for loading and versioning, and store explicit feature schemas rather than relying on automatic type inference. Normalise Unicode carefully, but preserve the original text for auditability. Do not lower-case or strip diacritics unless that transformation is part of the stated task. Make preprocessing configurable so researchers can compare raw, normalised, and transliterated settings.

For evaluation, use a pinned Python environment, deterministic seeds, versioned metric implementations, and automated tests for edge cases. Hugging Face Evaluate can help package reusable metrics, while a custom evaluation script may be preferable for language-specific rules. Provide a single command that runs a baseline end to end, and log model revision, tokenizer revision, hardware, batch size, decoding settings, and runtime.

Start with strong, reproducible baselines: a majority or lexical baseline, a multilingual encoder, an Indian-language model, and—where relevant—a general instruction-tuned model. Do not compare models with incompatible prompting, context lengths, or access to external data. Include parameter counts and compute details so results are interpretable, not merely ranked.

Audit results before publishing a leaderboard

Run targeted slices for script, region, domain, code-mixing, length, spelling noise, and low-resource languages. Inspect false positives and false negatives manually. Look for memorisation, train-test contamination, label imbalance, and performance gaps that aggregate metrics conceal.

Add safety and quality checks for high-impact use cases. A model that performs well on intent classification may still mishandle caste, religion, gender, disability, or regional references. Document whether the benchmark measures social bias, harmful stereotypes, privacy leakage, or refusal behaviour; do not imply that a general NLP score certifies safety.

If the benchmark will support an application such as multilingual voice support, pair text evaluation with realistic transcripts and interaction tests. Guidance on voice agent versus IVR for customer support is useful when translating model scores into a service-level evaluation plan.

Release it so others can reproduce it

Publish the code on GitHub and datasets or gated access rules on the Hugging Face Hub. Tag immutable releases, include a changelog, and provide a citation file. The README should show installation, authentication requirements, data preparation, baseline execution, metric definitions, and expected outputs.

A strong benchmark card should state:

• Scope, languages, tasks, and intended users
• Data sources, licences, consent, and privacy controls
• Annotation process and known disagreement
• Splits, leakage checks, and contamination policy
• Metrics, limitations, and unsupported interpretations
• Maintenance owner, issue process, and release version

Invite language experts and affected communities to review future versions. For ideas on sustaining a public technical project, see Indian open-source AI developer projects. Treat benchmark updates as versioned changes: adding data, revising labels, or changing tokenisation can make scores incomparable.

A practical launch checklist

Before announcing the benchmark, verify that:

• Every example has a documented provenance and licence status.
• Splits are deduplicated and leakage-tested.
• Metrics are validated with unit tests and language-specific examples.
• Results are reported per language, task, and relevant slice.
• At least one baseline can be reproduced on documented hardware.
• Dataset and benchmark cards disclose limitations and risks.
• The release includes version tags, code, environment files, and citation guidance.

A useful Indian-language benchmark is not the one with the largest dataset or the most impressive average score. It is the one that makes failures visible, respects contributors and data subjects, and gives builders a dependable basis for choosing and improving models.