Apply for AI Grants India

Why publish evaluation results on Hugging Face

Publishing a benchmark is more than uploading a spreadsheet. A useful Hugging Face repository lets another researcher or product team understand what was tested, on which languages, under what conditions, and how to reproduce the numbers. For Indian-language AI, this matters because results can change sharply across scripts, dialects, domains, code-mixed inputs, and speech or text normalization choices.

A public evaluation record also prevents strong claims from being detached from their evidence. If your work involves low-resource languages, first review the practical considerations in low-resource Indic natural language processing. If the project is part of a broader open-source effort, publishing alongside Indian open-source AI developer projects can make it easier for contributors to discover and extend.

Decide what you are publishing

Choose the repository type and scope before creating files:

• Dataset repository: Use this for benchmark inputs, annotations, prompts, evaluation manifests, human ratings, or aggregate result tables. Do not upload restricted or personally identifiable data.
• Model repository: Use this when the evaluated model or a fine-tuned checkpoint is also being released.
• Code repository: Keep evaluation scripts, configuration files, and environment instructions in a code repository when that makes maintenance easier. Link it clearly from the dataset or model card.
• Results-only repository: Suitable when test data cannot be redistributed. Publish aggregate outputs, hashes, scripts, access instructions, and a precise description of the unavailable data.

Define whether the repository covers one language, a language family, or several tasks. A single multilingual repository is appropriate only when its directory structure and result schema remain easy to query. Otherwise, separate repositories can reduce ambiguity.

Prepare a reproducible result package

Before uploading, create a local release directory with predictable names. A practical structure is:

indic-evaluation/
├── README.md
├── dataset-card.md
├── results/
│   ├── summary.csv
│   ├── per_language.csv
│   └── per_example.jsonl
├── configs/
│   ├── evaluation.yaml
│   └── prompts.txt
├── scripts/
│   └── evaluate.py
├── CITATION.cff
├── LICENSE
└── CHANGELOG.md

Your main table should include the model identifier and revision, language, script, task, dataset split, number of examples, metric, score, and—where relevant—confidence interval or standard deviation. Record whether text was transliterated, normalized, translated, or code-mixed. For speech systems, include audio sampling rate, transcription convention, and whether punctuation and numerals were scored.

Avoid reporting only a single average across Hindi, Tamil, Marathi, Bengali, Telugu, Kannada, Malayalam, Gujarati, Punjabi, Odia, Assamese, Urdu, or other languages. Macro-averages can hide serious failures. Publish per-language and per-task scores, along with the aggregation method and sample counts.

Create the Hugging Face repository

Sign in at Hugging Face, select New model or New dataset, and choose a stable, descriptive repository name. Names such as indic-qa-evaluation or multilingual-asr-results are clearer than a project codename. Set visibility deliberately: public repositories improve access, while private or gated repositories may be necessary for sensitive material.

You can upload through the web interface, Git, or the Hugging Face Hub Python library. For repeatable releases, Git or the API is preferable because every update can be reviewed and tied to a commit. Keep large files in supported formats such as CSV, JSONL, Parquet, or compressed archives, and include checksums for downloadable artifacts.

A minimal upload workflow using the Hub library looks like this:

from huggingface_hub import HfApi

api = HfApi()
api.create_repo("your-org/indic-evaluation", repo_type="dataset", exist_ok=True)
api.upload_folder(
    folder_path="indic-evaluation",
    repo_id="your-org/indic-evaluation",
    repo_type="dataset",
)

Test the repository in a clean environment before publishing. A reader should be able to clone it, install the stated dependencies, run the documented command, and obtain the published values or understand any unavoidable differences.

Write a useful README and dataset card

The README is the entry point for users. Put the most important facts near the top:

• What languages, scripts, tasks, and domains are covered.
• Which models and revisions were evaluated.
• A compact results table with links to detailed files.
• How to reproduce the evaluation.
• Known limitations, exclusions, and failure cases.
• Dataset, model, and annotation licences.
• Citation information and a contact or issue tracker.

Use a dataset card to explain provenance, collection dates, consent or access conditions, annotation procedures, demographic and geographic coverage, and intended use. Hugging Face metadata can improve discovery: add language tags using standard language codes where possible, task tags, dataset or model tags, licences, libraries, and any relevant modalities. Do not use broad labels such as “Indian language” as a substitute for naming each language and script.

If your evaluation uses a model card, state whether the model was trained or fine-tuned on data from the test set or a near-duplicate. This is especially important for generative models, where benchmark contamination and prompt formatting can materially affect scores.

Report results responsibly

Metrics need context. Explain tokenization, exact-match rules, BLEU or chrF settings, word error rate normalization, toxicity or safety labels, and judge prompts for generative evaluations. For LLM-as-a-judge studies, publish the judge model, rubric, sampling procedure, temperature, number of runs, and inter-rater or agreement information where available.

Separate automatic scores from human judgements and distinguish statistically meaningful differences from small numerical changes. Include error examples, not just leaderboard rankings. For Indian languages, inspect script mixing, spelling variants, honorifics, named entities, dialect variation, and translation artifacts. A system that scores well on standardized text may still fail on real user inputs.

Do not release raw text, audio, translations, or annotations when permissions do not allow redistribution. Publish aggregates, redacted examples, data statements, access procedures, and a secure contact route instead. This is particularly important for health, education, legal, and voice data.

Version, cite, and maintain the release

Tag a release such as v1.0.0 after validating the files. Record the evaluation date, code commit, model revision, dataset revision, hardware, software versions, and key configuration changes. Add a CHANGELOG.md so users can distinguish corrected scores from newly added experiments. If a score changes, explain why rather than silently replacing the old file.

Include CITATION.cff and a BibTeX entry. Cite the benchmark creators, data providers, model developers, annotators, and infrastructure used. Invite issues or pull requests, but review contributions before merging and retain a clear maintainer policy.

For teams building practical multilingual products, transparent evaluation is also a product asset. It can guide deployment decisions for use cases such as open-source vision-language models for Indian languages, rather than treating one headline score as evidence of broad reliability.

Pre-publication checklist

Before making the repository public, verify that:

• Every score has a language, task, split, sample count, and metric definition.
• Model and dataset revisions are pinned.
• Scripts and configuration reproduce the reported results.
• Licences permit the files and intended use.
• Sensitive information and restricted examples are removed.
• Per-language results are available, not only an aggregate.
• README, dataset card, citation, and changelog are complete.
• Repository tags and file names are consistent.
• A release tag or commit identifies the published version.

Publishing this way turns an evaluation result into a durable research artifact. It gives Indian-language builders a clearer basis for comparison, replication, and deployment—and gives users enough evidence to judge whether a model is suitable for their language and context.