Sarvam LLM evaluation is the process of measuring how well Sarvam AI’s language models perform for a specific application, language, workflow and risk profile. A useful evaluation goes beyond a single accuracy score: it tests Indic-language quality, factuality, instruction following, safety, latency, cost and reliability under realistic conditions.
For Indian AI teams, evaluation is especially important because performance can vary substantially across languages, scripts, dialects, code-mixed input and speech-to-text or translation pipelines. This guide presents a practical framework for evaluating Sarvam models before selecting a model, deploying an API, or comparing results with other large language models.
What Sarvam LLM evaluation should measure
A complete evaluation should answer five questions:
- Does the model understand the user’s language and intent?
- Does it generate accurate, useful and well-structured output?
- Does it follow application rules and return the required format?
- Is it safe and robust for the intended users?
- Can the system meet latency, throughput and cost targets?
The right test design depends on the use case. A customer-support assistant needs strong intent classification, retrieval-grounded answers and refusal behaviour. A government-services assistant needs multilingual coverage, accessibility and high factual accuracy. A developer tool may prioritise code quality, structured output and long-context performance.
Do not treat a public benchmark result as a substitute for application-specific testing. Your own evaluation set should represent the queries, spelling variations, languages and failure modes that the model will encounter in production.
Build an India-specific evaluation dataset
The foundation of Sarvam LLM evaluation is a clean, representative test set. Create separate development, validation and holdout sets so that prompt tuning does not contaminate final results.
Include language and script variation
For Indian-language applications, test more than formal written text. Include:
- Hindi, Bengali, Marathi, Tamil, Telugu, Kannada, Malayalam, Gujarati, Punjabi and Odia where relevant
- Romanised Indian languages, such as Hinglish or Tanglish
- Code-mixed prompts containing English and an Indian language
- Regional vocabulary, abbreviations and colloquialisms
- Spelling errors, missing diacritics and speech-recognition mistakes
- Native scripts alongside transliterated text
- Short queries, long descriptions and multi-turn conversations
A model can appear strong on clean benchmark prompts while failing on Romanised text or code-mixed requests. Report results by language, script and input type rather than publishing only an aggregate score.
Add realistic task categories
A balanced dataset may contain:
1. Question answering: factual, procedural and domain-specific questions.
2. Summarisation: news, policy documents, customer conversations and long reports.
3. Translation and transliteration: both directions, including named entities and technical terms.
4. Classification: intent, sentiment, urgency, eligibility or document category.
5. Information extraction: names, dates, amounts, addresses and application fields.
6. Generation: emails, explanations, notices, support responses and marketing copy.
7. Structured output: JSON, XML, tables or schema-constrained responses.
8. Conversation: clarification, context retention and escalation to a human agent.
Store each example with metadata such as language, domain, difficulty, expected output, source, risk level and whether external retrieval is required. This makes it possible to diagnose regressions instead of relying on a single score.
Core metrics for Sarvam LLM evaluation
Accuracy and task success
For classification and extraction, use exact match, precision, recall and F1 score. Exact match is strict but useful for fields that must be correct, such as an application ID. For generative tasks, define task-specific success criteria rather than using one generic metric.
Examples include:
- Correct answer rate for question answering
- Slot-filling accuracy for structured forms
- Valid JSON rate for API workflows
- Citation or evidence correctness for retrieval-augmented generation
- Successful resolution rate for support conversations
- Human-rated helpfulness for open-ended responses
Multilingual quality
Report quality separately for every supported language. A weighted average can hide weak performance in lower-volume languages. Useful measures include translation quality scores, language identification accuracy, terminology preservation and human ratings for fluency and adequacy.
For transliteration, check whether proper nouns, numbers, dates and domain terms remain unchanged or are rendered according to the product requirement. For code-mixed prompts, assess whether the response uses the requested language and maintains a consistent register.
Factuality and groundedness
A fluent answer can still be wrong. Build factuality tests using verified reference answers and, where applicable, a controlled knowledge base. Measure:
- Claim accuracy
- Unsupported-claim rate
- Retrieval recall
- Citation precision
- Abstention quality when evidence is unavailable
- Date sensitivity for changing information
For high-stakes applications, require the model to say that it does not know when the source material does not support an answer. Evaluate both false answers and unnecessary refusals.
Instruction following and format compliance
Test whether the model follows constraints such as language, length, tone, forbidden content and output schema. Include conflicting instructions, irrelevant prompt injection and malformed inputs.
For structured outputs, validate every response programmatically. Track schema-valid rate, missing-field rate, type errors and repair frequency. A model that produces excellent prose but invalid JSON may be unsuitable for an automated workflow.
Safety and responsible AI
Safety evaluation should reflect the product’s threat model. Test harmful requests, privacy leakage, harassment, discrimination, self-harm content, fraud, cyber abuse and attempts to bypass system instructions.
For Indian deployments, include sensitive scenarios involving caste, religion, gender, region, language identity, political persuasion and public benefits. Measure both unsafe compliance and over-refusal. A safe system must block genuinely harmful requests without refusing ordinary educational, medical-information or civic-service questions unnecessarily.
Design a reliable evaluation harness
Manual testing is useful during discovery but insufficient for repeatable model comparison. Build an evaluation harness that records the exact model version, endpoint, parameters, system prompt, tools, retrieved documents and timestamp for every run.
A practical pipeline is:
1. Load a versioned test set.
2. Send identical inputs to each candidate configuration.
3. Capture output, latency, token usage, errors and retry count.
4. Run deterministic validators for format, language and required fields.
5. Apply reference-based metrics where appropriate.
6. Use an LLM judge only as a secondary signal.
7. Send sampled outputs to trained human reviewers.
8. Store results in a dashboard and compare confidence intervals.
Keep temperature and other generation settings fixed during controlled comparisons. Test multiple runs for stochastic models because one pass can produce misleading results. For production readiness, conduct load tests with realistic concurrency and prompt lengths.
Human evaluation remains essential
Automated metrics often miss culturally inappropriate wording, subtle mistranslation, unnatural register and answers that are technically correct but unusable. Human evaluation should be performed by reviewers who understand the target language and domain.
Give reviewers a clear rubric with five-point or binary scales. For example:
- Correctness
- Completeness
- Relevance
- Fluency and naturalness
- Appropriate language and register
- Safety
- Evidence use
- Overall task success
Use at least two reviewers for high-risk samples and measure agreement. Resolve disagreements using an adjudication process. Blind the model identity where possible to reduce preference bias. Do not ask reviewers to rate dimensions they cannot reliably judge, such as legal correctness without domain expertise.
Compare Sarvam models fairly
A meaningful comparison controls variables that are not part of the model itself. Use the same:
- Test prompts and conversation history
- System instructions, unless model-specific syntax is required
- Retrieved context and tool outputs
- Output schema and token limits
- Sampling parameters
- Evaluation rubric
- Hardware or network measurement method
Separate model quality from application quality. If one system uses better retrieval, prompt engineering or post-processing, label the comparison accordingly. A strong production architecture may combine a smaller model, retrieval, deterministic validation and fallback routing rather than relying on a single large model.
Track a scorecard with quality, cost and operational metrics. For example, a deployment decision may require a minimum task-success rate, zero critical safety failures, a maximum p95 latency and a target cost per resolved interaction.
Latency, throughput and cost evaluation
Indian users may access AI services through variable mobile networks, so average latency is not enough. Measure p50, p95 and p99 latency separately for time to first token and full response completion.
Also test:
- Concurrent requests and rate-limit behaviour
- Long prompts and long outputs
- Streaming versus non-streaming responses
- Timeout and retry behaviour
- Regional network conditions
- Error rates and recovery paths
- Token consumption by language and script
Estimate total cost per successful task, not merely cost per request. Include retries, retrieval, translation, moderation, observability and human escalation. A cheaper model with a high correction or escalation rate may have a higher effective cost.
Common mistakes to avoid
Using only English benchmarks
English results do not predict performance in Kannada, Tamil, Hindi, code-mixed text or Romanised inputs. Always segment the evaluation by language and input form.
Relying on LLM-as-judge alone
An automated judge may share the same language or cultural blind spots as the evaluated model. Calibrate it against human ratings and use deterministic checks for objective requirements.
Testing only ideal prompts
Production users make typos, change topics, omit context and ask ambiguous questions. Include adversarial, incomplete and noisy inputs.
Optimising for average score
Averages hide catastrophic failures. Report worst-language performance, high-risk error rates, confidence intervals and examples of severe failures.
Changing prompts during comparison
If prompts, retrieved documents and decoding settings differ, the experiment no longer isolates model performance. Version every component.
A practical evaluation checklist
Before production, confirm that you have:
- A representative, versioned multilingual test set
- Holdout data that was not used for prompt tuning
- Language-, script- and domain-level reports
- Automated checks for schema, safety and groundedness
- Human review by qualified Indian-language evaluators
- Repeated-run testing for stochastic outputs
- Load, latency and failure-recovery tests
- Cost per successful task calculations
- Red-team tests for prompt injection and sensitive content
- Monitoring and regression alerts after deployment
- A rollback or fallback model strategy
After launch, continue evaluating live traffic using privacy-preserving sampling and redaction. Monitor distribution shifts, newly emerging user intents, language coverage and changes in source data. A model that passes pre-launch tests can degrade when users behave differently from the original dataset.
FAQ: Sarvam LLM evaluation
What is Sarvam LLM evaluation?
It is the structured testing of Sarvam language models for quality, multilingual capability, safety, latency, cost and reliability in a defined application.
Which languages should be tested?
Test every language your product supports, including native script, Romanised text and code-mixed queries. Results should be reported separately by language rather than only as an overall average.
Are benchmark scores enough to select a model?
No. Benchmarks are useful for initial comparison, but an application-specific holdout set, human review, safety testing and production-style performance tests are necessary.
How can teams evaluate factuality?
Use verified reference answers or trusted retrieved documents, measure unsupported claims and citations, and test whether the model abstains when evidence is missing.
How often should evaluation be repeated?
Run evaluations whenever the model, prompt, retrieval system, safety policy, data source or infrastructure changes. Continue monitoring after deployment for regressions and distribution shifts.
Apply for AI Grants India
Building an Indian-language AI product and need support for evaluation, research or deployment? Apply through AI Grants India to explore funding and opportunities for your startup.