Apply for AI Grants India

Tanglish—Tamil expressed in Latin script, often mixed with English—is common in messaging, social media, customer support, and search. It is also difficult to evaluate: spelling is informal, users switch scripts and languages mid-sentence, and the same word may carry different meanings across regions and contexts.

A useful benchmark must therefore measure more than a single accuracy number. This guide shows how to benchmark Tanglish models on Hugging Face using controlled datasets, task-appropriate metrics, reproducible inference, and error analysis that reflects Indian user behaviour.

Define the task before choosing a metric

Start by writing down what the model is expected to do. Tanglish is a data format and linguistic setting, not one standalone NLP task. Common benchmark tasks include:

• Sentiment or intent classification: Detect sentiment, support intent, abuse, spam, or topic.
• Named-entity recognition: Identify people, places, organisations, products, and culturally specific entities.
• Transliteration: Convert Latin-script Tamil into Tamil script, or assess whether a model can understand both forms.
• Translation: Translate Tanglish into Tamil, English, or another Indian language.
• Retrieval and semantic similarity: Match a user query with FAQs, documents, or support responses.
• Generation: Produce a safe, relevant answer in Tanglish or Tamil.

Task definition determines the correct metric. Accuracy can be misleading when labels are imbalanced. Use macro-F1 for classification, span-level precision, recall, and F1 for NER, and chrF or character error rate for transliteration. For translation, report chrF alongside BLEU; character-level metrics are often more informative for spelling variation. Generative systems also need human ratings for factuality, relevance, code-switching quality, and safety.

If your project covers multiple Indian languages, compare your methodology with benchmarking NLP models for Telugu and Sanskrit, but do not assume that a benchmark designed for formal Indic text transfers directly to Tanglish.

Build a representative evaluation set

A benchmark is only as credible as its test data. Use a held-out test set that reflects the inputs your application will receive in India. Include variation across:

• Latin-script spelling conventions, such as “epdi,” “eppadi,” and phonetic alternatives.
• Tamil-script text, English-only text, and mixed-script messages.
• Different proportions of Tamil and English within one sentence.
• Formal, informal, abbreviated, emoji-rich, and speech-like writing.
• Regional vocabulary, names, films, food, public services, and local institutions.
• Short queries, long messages, misspellings, repeated characters, and punctuation-free text.

Avoid random row-level splits when examples from the same conversation, user, template, or source appear in both training and test data. Such leakage can produce impressive scores while failing in production. Prefer conversation-level, user-level, or time-based splits. Keep a private test set if the model or prompt will be repeatedly tuned against the public set.

Create a dataset card on Hugging Face that documents collection sources, licences, consent considerations, demographic limits, annotation instructions, and known gaps. Remove phone numbers, usernames, addresses, and other personal data. For sensitive domains, add a separate safety and privacy review rather than publishing raw examples.

Prepare a reproducible Hugging Face environment

Install the core libraries and pin versions in a requirements file or environment lock:

pip install transformers datasets evaluate accelerate scikit-learn pandas torch

For a classification model, load the tokenizer and model using the repository identifier. Always record the revision or commit hash, not just the model name:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "org/tanglish-model"
revision = "main"  # Prefer a fixed commit hash for published results

tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForSequenceClassification.from_pretrained(model_id, revision=revision)
dataset = load_dataset("org/tanglish-eval", revision=revision)

Confirm the model’s intended task, label mapping, tokenizer behaviour, maximum sequence length, and licence. A causal language model should not be compared directly with a sequence-classification model without defining the same prompt, decoding, and label-extraction procedure.

For GPU inference, use batching and disable gradients. Preserve the original text and a normalized copy so you can measure whether normalization helps or harms performance:

import torch

model.eval()
predictions = []
with torch.inference_mode():
    for batch in dataloader:
        inputs = tokenizer(
            batch["text"], padding=True, truncation=True,
            max_length=256, return_tensors="pt"
        ).to(model.device)
        logits = model(**inputs).logits
        predictions.extend(logits.argmax(dim=-1).cpu().tolist())

Record hardware, batch size, precision, sequence length, preprocessing, decoding settings, and runtime. Latency and memory matter when serving Indian-language applications, especially on modest cloud or on-premise infrastructure. Teams evaluating deployment options can also review how to deploy large language models locally.

Choose metrics that expose real weaknesses

Report overall scores and disaggregated results. At minimum, include:

• Macro-F1 and per-class F1 for imbalanced classification.
• Exact-match and normalized accuracy for intent or label tasks, with normalization rules disclosed.
• Precision, recall, and F1 by entity type for NER.
• chrF, BLEU, and character error rate for transliteration or translation, depending on the task.
• Recall@k, MRR, or nDCG for retrieval.
• Human preference, factuality, relevance, and safety rates for generation.
• Latency, throughput, peak memory, and cost per 1,000 requests for deployment decisions.

Run bootstrap confidence intervals where practical. A difference of one percentage point may not be meaningful on a small test set. For paired classification predictions, use a paired significance test or report per-example wins and failures rather than presenting a leaderboard rank as definitive.

Evaluate robustness by slicing results into script type, Tamil-English ratio, message length, spelling variation, domain, and demographic or regional categories only when those labels are ethically collected and sufficiently represented. This is often where a model’s practical quality becomes visible.

Compare models fairly

Create a fixed evaluation harness that runs every model on the same examples and applies task-specific post-processing. Do not silently correct one model’s output while leaving another untouched. For generative models, keep the system prompt, user prompt, temperature, top-p, maximum tokens, and stop conditions constant. Run multiple seeds when sampling is enabled; use deterministic decoding for a baseline.

A useful results table includes model revision, parameter count, licence, context length, dataset version, macro-F1 or task metric, worst-slice score, latency, memory, and cost. Include a strong non-neural or simple baseline where possible. A transliteration dictionary, TF-IDF classifier, or multilingual baseline can reveal whether a larger model provides meaningful value.

Do not judge a model solely against English benchmarks. Models developed for other Indic languages may provide useful transfer baselines, including the open-source small language models for Hindi, but Tanglish needs its own test slices and annotation standards.

Analyse errors and publish the benchmark

After scoring, inspect false positives, false negatives, malformed generations, and uncertain examples. Group errors into categories such as spelling variation, code-switch boundary, negation, sarcasm, named entities, transliteration ambiguity, toxic language, and out-of-domain input. Save representative examples with sensitive details removed.

Publish a machine-readable results file, evaluation script, dataset version, model revisions, and configuration. A good report answers three practical questions: What was tested? How reliable are the results? Where will the model fail? Include limitations—especially annotation disagreement, regional coverage, synthetic data, and possible contamination.

For teams building production systems, benchmark again after every tokenizer, fine-tuning, prompt, or retrieval change. Pair offline scores with a monitored pilot using consented, anonymised traffic. Track user corrections, fallback rates, latency, and safety incidents, while keeping a fixed regression suite so improvements in one Tanglish slice do not quietly damage another.