Apply for AI Grants India

Malayalam small language models should not be judged by one score or by English-first benchmarks translated after the fact. A useful evaluation asks whether a model understands Malayalam as people actually write and speak it: with inflection, compounds, code-mixing, spelling variation, regional vocabulary, transliterated text, and domain-specific terminology.

This guide presents a builder-focused evaluation process for models used in keyboards, search, education, translation, customer support, voice interfaces, and public-service applications. It is designed for teams working with limited data and modest hardware, where every test example and every millisecond of inference matters. For broader context on dataset design, tokenizer choices, and Indic-language constraints, see this low-resource Indic NLP guide.

Start with the deployment task

Define the product decision before choosing metrics. A Malayalam model for sentiment classification needs a different test plan from one that generates replies or powers speech recognition. Write down:

• Input and output: Malayalam script, Manglish transliteration, audio transcripts, or mixed Malayalam-English text.
• Users and domains: students, patients, government-service users, shoppers, journalists, or customer-support agents.
• Failure cost: a typo in autocomplete is inconvenient; an incorrect medical or legal answer can be harmful.
• Operating limits: device memory, CPU/GPU availability, latency target, context length, and cost per request.
• Success threshold: specify the minimum acceptable quality and the maximum tolerable error rate before testing.

Keep a locked test set that is never used for prompting, fine-tuning, model selection, or repeated manual inspection. Split data by document, speaker, author, or conversation—not randomly by sentence—so near-duplicates do not inflate results.

Build a Malayalam-first evaluation set

A representative test set should include both clean and naturally messy language. Balance the set across:

• Formal Malayalam from news, textbooks, policy documents, and official notices.
• Conversational Malayalam from messaging, forums, and customer interactions, with privacy-safe redaction.
• Regional and social variation, including vocabulary and constructions associated with different parts of Kerala and the wider Malayalam-speaking community.
• Code-mixed Malayalam-English and common transliteration patterns.
• Spelling errors, missing punctuation, repeated characters, abbreviations, and Unicode inconsistencies.
• Names, places, dates, currency, phone numbers, measurements, and Malayalam numerals where relevant.
• Long compounds, agglutinative forms, inflected words, and rare but valid vocabulary.

Annotate each example with task labels, difficulty, domain, script form, and known ambiguity. Use at least two Malayalam-proficient annotators for subjective tasks. Resolve disagreements with an adjudicator and report agreement rather than hiding uncertainty. A small, carefully reviewed test set is more valuable than a large web scrape with unknown duplication and label quality.

Measure quality by task

Generative language quality

Perplexity can compare models trained with the same tokenizer and evaluation corpus, but it is not a product-quality score. Tokenization differences make cross-model comparisons unreliable, and a lower perplexity model may still produce poor answers. Report perplexity separately for formal, conversational, code-mixed, and transliterated subsets.

For generation, evaluate factuality, relevance, instruction following, grammar, and repetition with human or expert review. Use a fixed prompt suite and blind model outputs. Score each response on a defined scale, record unacceptable errors, and include a “cannot answer safely” option. Automatic similarity metrics such as ROUGE or BLEU can help with constrained tasks, but they should not be the primary measure of open-ended Malayalam quality.

Classification and extraction

Use macro-F1, per-class precision and recall, and confusion matrices when labels are imbalanced. Accuracy alone can conceal failure on minority sentiments, dialectal forms, or rare entities. For named-entity recognition, report entity-level precision, recall, and F1, with separate scores for people, organisations, locations, dates, and mixed-script entities.

For structured extraction, check exact match and field-level accuracy. A model that extracts the right value but corrupts a date or currency symbol should not receive full credit. Test robustness to punctuation changes, spelling variants, and code-mixing.

Translation, summarisation, and rewriting

Use BLEU, chrF, or similar metrics only alongside human evaluation. chrF can be useful for morphologically rich languages because it considers character n-grams, but it still misses factual and stylistic errors. For summarisation, check coverage of key facts, unsupported claims, omission of names or numbers, and Malayalam fluency. Create targeted “must preserve” test cases for negation, honorifics, dates, quantities, and instructions.

Test robustness, safety, and fairness

A Malayalam model can perform well on standard text and fail on realistic inputs. Create challenge sets for:

• Orthographic variants and Unicode-normalisation differences.
• Malayalam-English code-mixing and transliteration.
• Dialectal vocabulary, informal speech, and social-media spelling.
• Long words, rare names, repeated text, and long contexts.
• Negation, sarcasm, politeness, honorifics, and ambiguous pronouns.
• Prompt injection, abusive requests, personal data, medical claims, and political persuasion.

Compare performance across demographic and regional slices where lawful and ethically collected. Do not infer identity from language style or publish sensitive subgroup results without safeguards. For high-impact applications, have Malayalam-speaking domain experts review failures. Safety evaluation should measure both over-refusal and unsafe compliance: a model that rejects ordinary Malayalam requests is not reliable, while one that confidently invents advice is dangerous.

Evaluate efficiency on the target hardware

Small models are often selected for deployment, not leaderboard performance. Measure:

• First-token and end-to-end latency at realistic input lengths.
• Tokens per second, peak memory, model size, and energy where relevant.
• Quality after quantisation, pruning, distillation, or adapter fine-tuning.
• Context-length degradation and performance under concurrent requests.
• Cost per 1,000 requests or per user session.

Run these tests on the actual Android device, edge server, or cloud instance you plan to use. A model with slightly lower quality but reliable offline performance may be the better choice for Kerala-focused field deployments. If the system includes speech, assess the complete pipeline rather than only the text model; voice interfaces require separate checks for recognition errors, turn-taking, and Malayalam pronunciation. See the related guide on open-source small language models for Hindi for a comparable Indic-model evaluation perspective, while remembering that Hindi results do not transfer automatically to Malayalam.

Use a repeatable evaluation workflow

1. Version the data and prompts. Store hashes, licenses, annotation guidelines, and split definitions.
2. Establish a baseline. Compare against a simple classifier, retrieval system, or existing open model—not only against previous fine-tunes.
3. Run automatic tests. Produce per-slice metrics, confidence intervals, and error distributions.
4. Review failures manually. Group errors by tokenisation, morphology, factuality, dialect, safety, or domain.
5. Test interventions. Compare better data, continued pretraining, instruction tuning, retrieval, and quantisation separately.
6. Run human preference or task studies. Use blinded pairwise comparisons with Malayalam-speaking evaluators and clear rubrics.
7. Pilot with monitoring. Log privacy-safe inputs, abstentions, latency, user corrections, and escalation events.
8. Re-test after every change. A gain on sentiment may come with worse hallucination, latency, or dialect coverage.

For reproducibility, publish the model version, tokenizer, decoding settings, prompt templates, dataset provenance, evaluation date, and hardware. Where possible, release synthetic or de-identified examples and the annotation rubric, not private user data.

Common mistakes to avoid

• Treating translated English benchmarks as a complete Malayalam test.
• Reporting one aggregate score without domain or script slices.
• Using BLEU or perplexity to judge open-ended conversation.
• Randomly splitting near-duplicate web text across train and test sets.
• Asking the same annotator to create and score the evaluation data.
• Ignoring transliteration, code-mixing, and spelling variation.
• Claiming improvement from a tiny test set without uncertainty estimates.
• Measuring model quality without latency, memory, and quantisation results.

A practical release standard

Before deploying, require a scorecard covering task quality, difficult slices, safety, efficiency, and known limitations. Set explicit go/no-go thresholds—for example, minimum macro-F1 for each critical class, maximum unsupported-answer rate, maximum p95 latency, and a zero-tolerance process for severe safety failures. Keep a Malayalam-speaking review panel involved after launch, because new names, slang, domains, and user behaviours will expose gaps that a static benchmark cannot.

Evaluation is not a final checkbox. For Malayalam small language models, it is the mechanism that turns scarce data and limited compute into a dependable product. Measure what users need, inspect where the model fails, and optimise the full system rather than chasing a single benchmark number.