Apply for AI Grants India

IndicGlue is useful precisely because it makes multilingual performance comparable across Indian-language NLP tasks. But a single leaderboard rank can hide important weaknesses: a model may score well overall while failing on low-resource languages, code-mixed text, named entities, or informal user-generated content.

Karpathy’s Autoresearch approach is best treated as an automated experiment loop, not a magic benchmark optimiser. You define a measurable objective, make one controlled change at a time, run the evaluation, retain improvements, and inspect failures. Applied carefully to IndicGlue, this workflow can reveal where a model is genuinely weak—and where a leaderboard gain is merely an artefact of data, prompting, or evaluation choices.

What Autoresearch means in this workflow

The practical idea behind Autoresearch is simple: turn model improvement into a repeatable search process. A script or agent modifies a training configuration, launches an experiment, records the result, and decides whether the change should be kept.

For IndicGlue, the loop should include:

• A fixed code version and environment.
• A declared model checkpoint and tokenizer.
• A reproducible dataset split and preprocessing pipeline.
• Task-level metrics, not only one aggregate score.
• Automatic logging of configuration, runtime, cost, and failures.
• A stopping rule that prevents endless hyperparameter search.

Do not assume that any tool marketed as “Autoresearch” has built-in knowledge of IndicGlue or access to private leaderboard data. In practice, you will usually need to connect an experiment runner to the benchmark repository, your training code, and a local results database.

Start with a leaderboard gap map

Before running experiments, create a table for every IndicGlue task and language you can evaluate. Record the current best result, your baseline, the metric, and the evaluation split. Also record whether the result is directly comparable: differences in model size, external pretraining data, task formulation, or test-time augmentation can make leaderboard numbers misleading.

A useful gap map includes:

• Absolute gap: best reported score minus your score.
• Relative gap: the absolute gap divided by the best reported score.
• Language spread: strongest language score minus weakest language score.
• Task spread: strongest task score minus weakest task score.
• Variance: mean and standard deviation across random seeds.
• Efficiency: quality relative to parameters, GPU hours, and inference cost.

This prevents a common mistake: optimising the easiest or most heavily represented task while ignoring the language where users experience the largest quality failure. If you are building an evaluation product, the same discipline used in an open-source speech arena leaderboard for India is valuable: publish the protocol, separate comparable runs, and expose uncertainty.

Build a reliable baseline first

Autoresearch cannot diagnose a gap if the baseline is unstable. Run at least three seeds for the initial configuration when compute allows, and save predictions—not just aggregate metrics. Check that labels, language identifiers, tokenisation, and task-specific formatting are correct.

For Indian-language experiments, inspect preprocessing closely:

• Preserve Indic Unicode correctly and normalise consistently.
• Test punctuation, numerals, spelling variation, and transliterated text.
• Identify code-mixed samples instead of silently dropping them.
• Measure sequence truncation by language and task.
• Verify that train and evaluation data do not contain near-duplicates.
• Keep a record of whether text was translated, transliterated, or generated.

A suspiciously high score can indicate leakage or an evaluation bug. A suspiciously low score can come from broken labels or truncation rather than model weakness. Run a small manual audit before launching a large search.

Design the Autoresearch experiment loop

Your runner should make every experiment a self-contained record. A minimal configuration can contain the checkpoint, learning rate, batch size, sequence length, training steps, data mixture, seed, and evaluation command. The runner then follows this cycle:

1. Select a proposed change from a constrained search space.
2. Create a unique experiment ID and immutable configuration file.
3. Train or fine-tune for a fixed budget.
4. Evaluate every IndicGlue task and language subset.
5. Save predictions, metrics, logs, and resource use.
6. Compare against the baseline and accept or reject the change.
7. Produce a failure report for the next proposal.

Start with one variable at a time. Early experiments might compare learning rates, maximum sequence lengths, sampling strategies, or instruction templates. Once the signal is clear, test interactions such as language-balanced sampling plus continued pretraining.

Use a composite objective only when it reflects your product goal. For example:

objective = macro_average(task_scores) - 0.25 × worst_language_penalty

The exact formula is less important than publishing it. If you care about low-resource languages, an unweighted average may hide regressions. Track both the headline objective and a guardrail metric such as the minimum language score or maximum degradation on any task.

Turn scores into diagnosable gaps

A leaderboard difference is a symptom, not an explanation. After each run, slice results by factors that matter in India:

• Language and script.
• Task type and label frequency.
• Code-mixed versus monolingual text.
• Formal versus conversational text.
• Short versus long inputs.
• Named entities, numerals, and spelling variants.
• Urban or domain-specific vocabulary where metadata is available.

Then inspect the actual errors. Cluster false positives and false negatives by cause: unfamiliar vocabulary, negation, ambiguous morphology, entity boundary errors, translation artefacts, or annotation disagreement. For classification tasks, examine the confusion matrix. For sequence labelling, calculate boundary and per-entity-type errors rather than relying only on token-level F1.

A useful diagnostic asks three questions: Does the model fail because it lacks knowledge, because the input is represented poorly, or because the labels are inconsistent? Each answer requires a different intervention. More pretraining will not fix a broken label mapping, and a larger model may not fix a tokenizer that fragments an important script excessively.

Prioritise experiments that can generalise

Once Autoresearch identifies a weak slice, test targeted interventions:

• Continue pretraining on licensed, representative Indic text.
• Add high-quality examples for the failing language or phenomenon.
• Rebalance batches without eliminating naturally frequent languages.
• Compare multilingual and language-specific adapters.
• Test script-aware or transliteration-aware preprocessing.
• Use hard-negative mining for confusing labels.
• Calibrate probabilities when deployment decisions depend on confidence.

Keep a held-out diagnostic set that is never used to choose experiments. Otherwise, the loop will overfit the visible benchmark. A model that wins on IndicGlue but fails on real support chats, public-service queries, or creator content has not solved the deployment problem. Builders choosing a new product direction should combine benchmark evidence with the kind of market validation described in Finding AI Business Ideas for India.

Reproduce, report, and avoid leaderboard gaming

A credible result includes the model version, training data sources, compute budget, seeds, preprocessing, prompt or fine-tuning format, and statistical variation. Report per-task and per-language results alongside the aggregate score. If you used synthetic data, translation, retrieval, external tools, or test-time voting, disclose it clearly.

Run the final configuration from a clean environment and confirm that the result survives a new seed. Compare against a strong, fairly matched baseline rather than only the public leaderboard leader. If the improvement is concentrated in one slice, describe it as a targeted gain—not a universal advance.

For collaboration, publish the experiment schema and failure categories so others can reproduce the analysis. An AI builders community in India can be useful for finding reviewers who understand regional language variation, annotation practice, and deployment constraints.

A practical 2026 checklist

Before trusting an Autoresearch result, confirm that you have:

• Defined the primary metric and guardrail metrics.
• Locked evaluation data and checked for leakage.
• Logged every configuration and random seed.
• Reported language- and task-level results.
• Saved predictions for error analysis.
• Measured compute, latency, and memory—not only quality.
• Tested improvements on an untouched diagnostic set.
• Documented data licensing and model-card limitations.

The objective is not to automate judgement away. It is to make judgement more informed: Autoresearch handles repetitive experimentation, while you decide which gaps matter for Indian users and whether an apparent gain is robust, equitable, and useful in production.