Why Indian language safety needs its own benchmark
A model that appears safe in English can fail badly in Hindi, Tamil, Bengali, Marathi, Telugu, Kannada, Malayalam, Gujarati, Punjabi, Odia, or Urdu. Safety meaning changes with script, dialect, code-mixing, transliteration, formality, and local references. Direct translation of an English test set is therefore a starting point—not a complete evaluation.
For builders, the goal is not to produce one headline score. It is to identify which harms occur, in which languages, under which prompts, and with what severity. This is especially important for low-resource languages, where training data and safety labels are often thinner; the practical constraints are covered in this builder’s guide to low-resource Indic NLP.
A useful benchmark should measure both harmful compliance and harmful refusal. A model that answers a benign health, caste, religious, or political question with an unnecessary refusal is not serving users well. A model that responds fluently while reinforcing a stereotype is also unsafe.
Define the safety scope before collecting data
Write a short risk taxonomy before opening a dataset repository. At minimum, include:
- Abuse and harassment: threats, slurs, demeaning descriptions, and targeted abuse.
- Identity harms: caste, tribe, religion, gender, sexuality, disability, region, and migrant-status stereotypes.
- Violence and criminal assistance: instructions that facilitate physical harm, weapons, fraud, or exploitation.
- Sexual and child safety: sexual content, grooming, exploitation, and age ambiguity.
- Self-harm: encouragement, romanticisation, or failure to provide an appropriate supportive response.
- Privacy and personal data: requests to expose, infer, or misuse sensitive information.
- Misinformation and high-stakes advice: medical, legal, financial, electoral, and communal claims.
- Language-specific failure modes: offensive transliterations, slur variants, abusive code-mixing, and culturally loaded idioms.
For each category, define the expected behaviour: answer, answer with caution, ask a clarifying question, or refuse and redirect. Avoid treating every sensitive topic as disallowed. The expected response must reflect the user’s intent and the risk of the requested action.
Build a representative Indic test set
Use a layered dataset rather than a single translated prompt file. A practical evaluation set contains:
1. Native-authored prompts: commission fluent speakers to write realistic prompts in each target language.
2. Professional translations: translate selected prompts, then have a second native reviewer check meaning, register, and implied intent.
3. Adversarial variants: include misspellings, slang, transliteration, mixed scripts, emojis, euphemisms, and multi-turn escalation.
4. Benign controls: pair harmful prompts with legitimate educational, journalistic, support, and historical questions.
5. Conversation tests: evaluate whether a model becomes unsafe after repeated persuasion, role-play, or instruction-hierarchy attacks.
Record language, script, dialect or region where relevant, prompt source, risk category, intended behaviour, severity, and annotation notes. Keep personally identifying information out of examples. If real user data is used, obtain appropriate consent, minimise fields, and document redaction.
Do not assume that a Hindi prompt written in Devanagari represents Hinglish typed in Latin script. Test both. For each language, report coverage separately for native script, transliteration, code-mixing, and dialectal forms. This is a stronger approach than collapsing all Indic performance into one average.
Prepare the Hugging Face evaluation pipeline
Store the benchmark as a versioned dataset on Hugging Face Hub, with a dataset card describing provenance, licences, annotation instructions, known gaps, and intended use. Use datasets to load fixed splits and transformers to run the target model with deterministic generation settings. Pin model revisions, tokenizer versions, prompts, decoding parameters, and evaluation code so results can be reproduced.
A minimal record might contain:
{
"id": "hi_identity_0042",
"language": "Hindi",
"script": "Devanagari",
"prompt": "...",
"risk_category": "identity_harm",
"severity": 2,
"expected_action": "refuse_redirect",
"reference_notes": "Avoid reinforcing caste stereotypes"
}Evaluate the base model and the complete production stack separately. Safety middleware, system prompts, retrieval, classifiers, and post-processing can materially change outcomes. Save raw outputs, not only scores, so reviewers can inspect whether a refusal is respectful, whether a safe answer is actually useful, and whether a harmful completion slipped through in mixed language.
Score safety with metrics that match the product
No single automatic metric is sufficient. Report a dashboard with at least these measures:
- Unsafe compliance rate: proportion of disallowed prompts receiving actionable harmful assistance.
- Safe completion rate: proportion of benign prompts answered accurately without needless refusal.
- Refusal precision: share of refusals that were actually warranted.
- Refusal quality: whether the response declines clearly, avoids repeating harmful details, and offers a safe alternative.
- Severity-weighted harm: assign greater weight to severe failures such as child exploitation, credible violence, or self-harm encouragement.
- Cross-language parity: compare performance by language, script, prompt type, and risk category—not only the overall mean.
- Robustness rate: measure consistency across transliteration, spelling variation, code-mixing, and paraphrase.
- Calibration and uncertainty: check whether confidence or safety labels remain reliable across languages.
Automatic toxicity classifiers can help triage results, but they often underperform on code-mixed and low-resource text. Treat them as signals, not ground truth. Use native-speaker review for final labels, especially for sarcasm, reclaimed language, caste references, religious context, and ambiguous intent.
Design reliable human evaluation
Recruit at least two independent reviewers per item where possible, with language fluency and training in the relevant harm categories. For high-severity cases, add an adjudicator. Measure agreement, document disagreements, and revise unclear guidelines before scaling annotation.
Give reviewers a structured rubric. They should label whether the output is harmful, whether it follows the intended action, whether it introduces stereotypes, whether it is culturally or linguistically inappropriate, and whether a safer useful response was possible. Provide skip options and wellbeing support for reviewers exposed to disturbing content.
A reviewer panel should not be treated as a single “Indian perspective”. India’s linguistic and social contexts vary significantly. Sample reviewers across regions and backgrounds, and record demographic context only when necessary for analysis and with appropriate safeguards.
Analyse failures and publish useful results
Break down results by language, script, risk, severity, prompt form, and model version. Include confidence intervals or bootstrap uncertainty for key rates, particularly when a language has a small sample. A model with a strong average score but poor performance on one language or severe category should not be described as safe overall.
Maintain a failure log with the prompt, output, expected behaviour, observed failure, likely cause, and proposed fix. Common fixes include better system instructions, language-specific safety data, improved tokenisation, retrieval safeguards, refusal templates, or a separate moderation model. Re-run the full regression suite after every change; do not replace old failures with newly sampled examples.
Share aggregate results and representative redacted examples through a model or dataset card. Withhold prompts that would meaningfully enable abuse, and respect dataset licences and annotator privacy. Open-source projects from Indian developers can provide useful infrastructure and comparison points; this overview of Indian open-source AI developer projects is a relevant starting point.
A practical release gate for 2026
Before deploying an Indic-language model, require:
- No unresolved critical failures in child safety, credible violence, or self-harm categories.
- Separate minimum thresholds for every supported language and script.
- A documented human-review process for severe or ambiguous cases.
- Regression tests covering transliteration, code-mixing, dialect variation, and multi-turn attacks.
- Monitoring that can identify language-specific drift without storing unnecessary user content.
- A public limitations statement and a clear route for reporting harmful outputs.
Benchmarking is valuable only when it changes engineering decisions. Use Hugging Face to make datasets, model revisions, evaluation code, and reports reproducible—but keep native speakers and affected communities central to deciding what “safe” means.