Apply for AI Grants India

Marathi generation systems need more than a single automatic score. A model may produce fluent Devanagari text while changing names, dropping negation, mixing Hindi or English, or failing on regional vocabulary. A useful benchmark therefore combines reproducible evaluation, Marathi-aware preprocessing, targeted test cases, and human quality checks.

This guide shows how to benchmark Marathi generation quality with Hugging Face Evaluate for translation, summarization, instruction following, and other text-generation tasks. The workflow is suitable for model comparison, fine-tuning decisions, and release gates in 2026.

Define the task before choosing a metric

Start by writing down what the model is expected to generate. A benchmark for Marathi-to-English translation is not interchangeable with one for Marathi summarization or chatbot responses.

Record the following for every evaluation run:

• Task: translation, summarization, question answering, rewriting, dialogue, or free-form generation.
• Input and output language: Marathi-only, Marathi-to-English, or code-mixed output.
• Target audience: urban users, rural users, students, government-service users, or a specialist group.
• Generation settings: model checkpoint, prompt template, temperature, top-p, maximum tokens, and decoding strategy.
• Quality risks: factual errors, omissions, dialect mismatch, unsafe content, transliteration, or excessive code mixing.

For broader coverage, pair this project with an Indian language LLM benchmark dataset or compare the methodology with benchmarking multilingual LLMs in India.

Build a Marathi evaluation set

Use a held-out test set that reflects actual usage rather than only clean literary sentences. A practical first benchmark contains 500–2,000 examples, with a fixed test split that is never used for training or prompt development.

Include:

• Standard Marathi from news, public information, education, and customer support.
• Long and short inputs, including sentences with subordinate clauses and numbered lists.
• Proper nouns, dates, currency amounts, addresses, abbreviations, and named entities.
• Colloquial phrasing, regional vocabulary, and carefully documented Marathi dialect variation.
• Code-mixed Marathi where users naturally switch between Marathi, Hindi, and English.
• Negation, honorifics, gender and number agreement, tense, and postposition usage.
• Adversarial cases involving spelling variants, noisy punctuation, and transliterated Marathi.

Keep a metadata file with fields such as domain, dialect, length_bucket, difficulty, and risk_type. This lets you report where a model succeeds instead of hiding weaknesses behind one aggregate number. If dialect coverage is important, use the methodology in fine-tuning AI models for Marathi dialects when constructing slices.

Install and load Hugging Face Evaluate

Install the evaluation library in a pinned environment so that future runs use the same dependencies.

python -m venv .venv
source .venv/bin/activate
pip install -U evaluate datasets sacrebleu rouge_score bert_score pandas

Load only the metrics required for your task:

import evaluate

bleu = evaluate.load("sacrebleu")
rouge = evaluate.load("rouge")
bertscore = evaluate.load("bertscore")

For production benchmarking, save the Python version, package lockfile, model revision, tokenizer revision, and evaluation-set hash. A score without this provenance is difficult to reproduce or audit.

Select metrics that match Marathi generation

SacreBLEU is useful for translation when you have high-quality references. It measures n-gram overlap, so it should be reported alongside qualitative checks. Use the expected input format: predictions are strings, while each reference example is typically a list of reference strings.

result = bleu.compute(
    predictions=predictions,
    references=[[reference] for reference in references]
)
print(result)

ROUGE-1, ROUGE-2, and ROUGE-L are common for summarization. They show lexical overlap and sequence similarity, but they can reward copied wording and penalize valid paraphrases. Report each component rather than only one ROUGE value.

result = rouge.compute(
    predictions=predictions,
    references=references,
    use_stemmer=False
)

For semantic similarity, BERTScore can be useful, but validate the selected multilingual model on Marathi before treating it as authoritative. Language coverage and tokenizer behaviour affect the result. For open-ended answers, add task-specific checks for factuality, completeness, instruction adherence, and harmful or inappropriate output.

Do not use BLEU or ROUGE as a standalone quality gate. Marathi morphology, word-order flexibility, spelling variants, and legitimate paraphrases can make surface overlap misleading.

Normalize carefully, not aggressively

Apply the same transparent preprocessing to references and predictions. Document Unicode normalization, whitespace handling, punctuation policy, and treatment of zero-width characters. Devanagari text may contain visually similar characters or combining marks that affect exact matching.

Avoid normalizing away information that matters. For example, stripping all punctuation can conceal errors in decimals, dates, phone numbers, or legal clauses. Keep both versions when possible:

• A canonical evaluation view for stable metric computation.
• The original output for human review and error analysis.

Check for Devanagari versus Latin transliteration, accidental Hindi substitutions, English leakage, repeated phrases, empty outputs, and copied prompts. These checks often expose regressions faster than a metric dashboard.

Run a reproducible benchmark

Generate outputs once, store them with input IDs, and compute metrics from the stored file. Do not regenerate text separately for every metric because sampling can change the outputs.

import json
from pathlib import Path

records = []
for item, prediction, reference in zip(test_set, predictions, references):
    records.append({
        "id": item["id"],
        "input": item["input"],
        "prediction": prediction,
        "reference": reference,
        "metadata": {"domain": item["domain"], "slice": item["slice"]}
    })

Path("marathi_predictions.jsonl").write_text(
    "\n".join(json.dumps(r, ensure_ascii=False) for r in records),
    encoding="utf-8"
)

Report the overall score and results by slice. Include bootstrap confidence intervals or at least variation across multiple fixed seeds when comparing close model versions. A two-point improvement may not be meaningful if the test set is small or domain coverage is narrow.

Add Marathi-focused human evaluation

Ask fluent Marathi reviewers to score a stratified sample using a short rubric. For generation tasks, rate meaning preservation, fluency, grammar, terminology, factual accuracy, and instruction compliance on a consistent scale. Have at least two reviewers assess overlapping examples and resolve disagreements with an adjudication rule.

Track error categories, not just average ratings:

• Wrong case markers, tense, gender, or number agreement.
• Omitted or invented facts.
• Incorrect names, numbers, dates, and units.
• Unnatural literal translation or excessive Hindi/English mixing.
• Dialect or register mismatch.
• Repetition, refusal errors, and unsafe responses.

A model intended for voice interfaces should also be tested on speech-recognition noise and spoken-style Marathi; the evaluation plan can complement work on voice agents for India SMB lead generation where regional-language reliability affects user conversion and support quality.

Set release gates and improve the model

Define thresholds before reviewing results. For example, require no regression in named entities or numbers, a minimum human factuality score, and acceptable performance on every high-risk domain slice. Use automatic metrics to locate regressions, then inspect examples to identify the cause.

Improvement actions may include:

• Expanding underrepresented domains and dialect slices.
• Removing duplicated or low-quality training examples.
• Adding terminology constraints or retrieval for specialist content.
• Fine-tuning with preference data from native Marathi reviewers.
• Adjusting decoding settings separately for translation and open-ended generation.
• Adding targeted regression tests for every severe error found.

What a strong report should contain

Publish the model revision, dataset provenance, split policy, preprocessing rules, metric versions, decoding parameters, overall scores, slice scores, confidence intervals, human-evaluation protocol, and representative failures. State clearly whether references were single or multiple and whether code-mixed text was included.

The goal is not to produce an impressive number. It is to establish whether the model is accurate, natural, consistent, and safe for the Marathi users it serves. Hugging Face Evaluate provides the measurement layer; a carefully designed Marathi benchmark provides the credibility.