Kanglish—Kannada and English used together in the same sentence, message, or conversation—does not behave like standard Kannada or standard English. Users switch scripts, transliterate Kannada into Latin characters, shorten words, add English technical terms, and vary spelling across regions and keyboards. A useful benchmark must measure those realities rather than reward a model for memorising one narrow dataset.
This guide shows how to benchmark Kanglish models on Hugging Face in a way that is reproducible, task-specific, and useful for builders working on Indian-language products.
Define the task before choosing a metric
Start by writing down what the model is expected to do. “Kanglish performance” is not one number. A classifier, translator, speech-text system, and chat model require different test designs.
Common benchmark tasks include:
- Intent or sentiment classification: Predict a label for customer messages, social posts, or support queries.
- Named-entity recognition: Identify people, places, organisations, products, and other entities across Kannada and English spans.
- Language identification: Classify Kannada, English, Kanglish, code-mixed text, and possibly transliterated Kannada.
- Translation or transliteration: Convert Kanglish into Kannada script, English, or a chosen target representation.
- Generation and assistance: Produce answers, summaries, rewrites, or replies while preserving meaning and the user’s language preference.
Keep separate test sets for each use case. A model that scores well on sentiment may still produce poor transliterations or unsafe customer-support answers.
For broader Indian-language comparisons, the methodology in benchmarking NLP models for Telugu and Sanskrit is a useful reference, but do not transfer its datasets or metrics blindly to Kanglish.
Build a representative Kanglish evaluation set
The evaluation set is usually more important than the evaluation script. Collect examples from the actual product environment, subject to consent, privacy safeguards, and platform terms. Useful sources may include anonymised support conversations, opt-in user prompts, public posts, subtitles, and synthetic variations reviewed by native speakers.
Stratify the data so that results reveal where the model works and fails. Track at least:
- Kannada script mixed with English words.
- Kannada written in Latin script, including inconsistent spellings.
- Romanised Kannada mixed with English abbreviations and emojis.
- Formal, casual, dialectal, and code-switched sentences.
- Short queries, long messages, misspellings, and noisy punctuation.
- Names, addresses, prices, dates, phone numbers, and product terminology.
- Urban and non-urban usage patterns, where your consented data supports that comparison.
Create train, validation, and test splits by conversation, user, or source, not by randomly splitting individual sentences. Otherwise, nearly identical messages can appear in both training and test data and inflate the score. Keep a locked, private test set for final reporting if the benchmark will guide a public model release.
Document annotation rules. Annotators should decide how to label ambiguous cases, mixed-script entities, sarcasm, spelling variants, and messages whose intent depends on context. Measure agreement on a sample and record disagreements rather than silently forcing uncertain labels.
Inspect tokenisation and data leakage
Before running a full benchmark, inspect how each candidate model tokenises representative Kanglish examples. Excessive fragmentation can increase sequence length, memory use, and error rates—especially for Romanised Kannada, where spelling is inconsistent.
A quick Hugging Face inspection looks like this:
from transformers import AutoTokenizer
model_id = "your-org/your-kanglish-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
samples = [
"Nale meeting ide, notes share maadi",
"ನಾಳೆ meeting ಇದೆ, notes share ಮಾಡಿ",
" ಈ product tumba useful ide"
]
for text in samples:
tokens = tokenizer.tokenize(text)
print(text)
print(tokens, "token_count=", len(tokens))Report average and high-percentile token counts, unknown-token rates where applicable, and truncation rates. Also check for leakage: duplicated posts, template-generated examples, benchmark text included in pretraining, or labels embedded in filenames and metadata.
If you are fine-tuning, freeze the test set before training. Do not repeatedly tune against it. Use a validation set for model selection and reserve the test set for the final comparison.
Select metrics that match the task
For classification, report macro F1, per-class precision and recall, accuracy, and a confusion matrix. Macro F1 prevents a large English-heavy class from hiding poor performance on smaller Kannada or Kanglish categories. For imbalanced production tasks, include weighted F1 and class support as secondary figures.
For sequence labelling, use entity-level precision, recall, and F1 rather than token accuracy alone. Report results separately for Kannada-script, Latin-script, and mixed-script entities if your annotations support that breakdown.
For translation or transliteration, BLEU can provide a reference point, but it is weak for legitimate spelling variation and mixed-language outputs. Add chrF or TER where appropriate, and perform human evaluation for meaning preservation, script correctness, fluency, and whether English terms were unnecessarily translated. For generated responses, score factuality, instruction following, harmful content, and language consistency with a structured rubric.
Always include operational measures:
- Latency at a stated batch size and sequence length.
- Throughput and peak GPU or CPU memory.
- Model size and precision mode.
- Cost per 1,000 or 1 million tokens, where relevant.
- Failure rate, truncation rate, and maximum supported context.
A smaller model that is slightly less accurate but substantially cheaper and faster may be the better choice for an Indian-language application. Teams planning on-device or private inference can also compare the results with approaches covered in how to deploy large language models locally.
Run a reproducible Hugging Face evaluation
Use fixed dataset versions, model revisions, random seeds, and environment files. Store the exact commit hash for each model and dataset, the tokenizer configuration, prompt templates, decoding settings, and hardware details.
For classification models, the Hugging Face Trainer API can produce predictions consistently:
from transformers import AutoModelForSequenceClassification, Trainer
import numpy as np
from sklearn.metrics import accuracy_score, f1_score
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return {
"accuracy": accuracy_score(labels, predictions),
"macro_f1": f1_score(labels, predictions, average="macro"),
"weighted_f1": f1_score(labels, predictions, average="weighted"),
}
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=NUM_LABELS)
trainer = Trainer(model=model, tokenizer=tokenizer, compute_metrics=compute_metrics)
results = trainer.evaluate(test_dataset)
print(results)For generative models, use identical prompts, maximum lengths, temperature, top-p, stop conditions, and post-processing rules. Evaluate deterministic decoding first so model comparisons are fair; then run a separate robustness experiment across multiple seeds or decoding settings.
Publish a dataset card and model card describing language coverage, consent and licensing, known gaps, intended use, and limitations. Hugging Face’s Evaluate library can help standardise metric calculation, but you still need task-specific validation and human review.
Add slices and human error analysis
A single aggregate score cannot explain a Kanglish model’s behaviour. Create slices for script, code-switch ratio, message length, spelling noise, topic, and dialect or regional variation where labels are reliable. Compare each slice with the overall result and with a strong English-only and Kannada-capable baseline.
Review false positives and false negatives manually. Categorise errors such as:
- Misreading Romanised Kannada as English.
- Translating an English product or personal name incorrectly.
- Losing negation, tense, politeness, or intent during code-switching.
- Hallucinating when the input is short or misspelled.
- Producing an inappropriate script or register for the user.
Use two or more trained reviewers for a meaningful sample, blind them to model identity, and report agreement. If the benchmark supports a public service, include safety tests for personal data, scams, harassment, medical advice, and financial requests rather than treating language accuracy as the only risk.
Report results builders can act on
A useful report includes the dataset version, sample counts, language and script distribution, baseline models, metric definitions, confidence intervals where feasible, hardware, latency, cost, and every preprocessing step. Show both aggregate and slice-level results. Include representative successes and failures, with private data redacted.
Avoid ranking models from tiny differences. Bootstrap confidence intervals or repeated runs can show whether a one-point improvement is meaningful. Re-test after tokenizer changes, fine-tuning, quantisation, prompt changes, or dataset updates.
For teams building Kannada-language systems, fine-tuning large language models for Sanskrit translation offers useful lessons on parallel data and evaluation design, while fine-tuning AI models for Marathi dialects highlights why regional variation deserves explicit benchmark slices.
A practical benchmark checklist
Before publishing or using the result, confirm that you have:
- Defined the task, user population, and acceptable failure modes.
- Used consented, licensed, representative Kanglish data.
- Prevented user, template, and benchmark leakage.
- Inspected tokenisation across scripts and spelling variants.
- Reported task, robustness, fairness, safety, and efficiency metrics.
- Compared against relevant Kannada, English, and multilingual baselines.
- Performed slice-level and human error analysis.
- Locked versions, seeds, prompts, hardware, and preprocessing.
- Documented limitations and a plan for periodic re-evaluation.
A strong Kanglish benchmark is not merely a leaderboard number. It is an evidence system that tells a product team which users are served well, which inputs remain risky, and what to improve next. As models and datasets change through 2026, versioned evaluations will make those improvements measurable instead of anecdotal.
Apply for AI Grants India
If you are building an Indian-language model, evaluation dataset, or responsible AI product, apply for AI Grants India to explore support for research, infrastructure, and deployment.