Tamil-language AI projects need more than translated prompts. They need examples grounded in the way Tamil-speaking users actually ask questions, interpret public services, and navigate Indian institutions. This guide explains how to create Tamil instruction tuning data from Indian public documents without treating every downloadable file as automatically reusable training data.
The goal is a dataset that is useful, auditable, and safe to fine-tune: each example should have a clear task, a defensible source, accurate Tamil, and enough context for a reviewer to verify the answer.
Define the dataset before collecting documents
Start with the model behaviour you want to improve. A narrow, well-evaluated dataset is usually more valuable than a large archive of loosely converted text.
Useful Tamil instruction categories include:
- Information extraction: identify dates, eligibility criteria, fees, offices, or required documents.
- Question answering: answer questions using a supplied government circular, scheme notice, or public report.
- Summarisation: produce short Tamil summaries for citizens, officials, or students.
- Translation and rewriting: convert formal administrative Tamil into plain language without changing meaning.
- Classification: route a request to a department, document type, or urgency category.
- Form assistance: explain fields and steps while clearly stating when the user must verify current instructions.
- Refusal and uncertainty: decline unsupported claims, expose missing information, and recommend an official source.
Write a dataset specification first. Record target users, domains, Tamil varieties, expected answer length, citation requirements, prohibited content, and evaluation metrics. If the project will fine-tune a model, align the specification with best practices for fine-tuning LLMs on custom data.
Select lawful, high-value Indian sources
“Publicly accessible” does not always mean “free to copy, redistribute, or use commercially.” Build a source register containing the URL, publisher, access date, licence or terms of use, language, document date, and permitted uses.
Good starting points include:
- Central and Tamil Nadu government department websites.
- Gazette notifications, scheme guidelines, public advisories, and annual reports.
- Public court judgments and tribunal decisions, subject to source terms and privacy review.
- Election, education, agriculture, transport, and municipal information published for citizens.
- Open government datasets and documents with explicit reuse terms.
- Public-domain or permission-based educational materials.
Avoid indiscriminate crawling. Respect robots.txt, rate limits, access controls, copyright, database rights, and website terms. Store a copy of the licence evidence alongside each source record. For sensitive domains such as health, use a stricter review process; ICMR-compliant medical AI data verification in India offers a useful reference point for provenance and validation discipline.
Download and preserve document provenance
Use stable, reproducible collection rather than manual copy-paste. For each document, preserve:
- Original URL and publisher.
- Retrieval timestamp and document publication date.
- File hash or version identifier.
- File type, language, and OCR status.
- Page numbers or paragraph identifiers.
- Licence, access conditions, and reviewer notes.
Keep raw files separate from processed text. Never overwrite the original. This allows the team to identify whether an error came from the source, OCR, cleaning, annotation, or model generation.
For PDFs, extract text with layout awareness. Government documents often contain tables, footnotes, headers, bilingual columns, and scanned pages. Run OCR only where necessary, and flag low-confidence pages for manual review. Do not silently “correct” names, numbers, dates, or legal terms during extraction.
Clean Tamil without erasing meaning
Tamil public documents can contain Unicode inconsistencies, legacy fonts, broken line wrapping, duplicated headers, and mixed Tamil-English terminology. Normalise Unicode carefully, remove repeated boilerplate, and retain meaningful punctuation and paragraph boundaries.
Create explicit rules for:
- Unicode normalisation and canonical character handling.
- Numerals, dates, currency, percentages, and measurements.
- Abbreviations and department names.
- Transliteration versus Tamil-script usage.
- Formal, colloquial, and domain-specific vocabulary.
- Tables, lists, annexures, and references.
Do not “improve” official wording merely because it sounds formal. Instead, create separate examples for faithful extraction and plain-language explanation. Keep the original passage linked to every generated example so reviewers can compare claims.
Generate instruction-response examples
Convert documents into tasks, not just question-answer pairs. A strong record normally includes the instruction, optional context, response, source citation, task type, language register, and quality labels.
Example:
{
"instruction": "இந்தத் திட்டத்திற்கு விண்ணப்பிக்க தேவையான ஆவணங்களைச் சுருக்கமாகக் கூறுங்கள்.",
"context": "[Source passage from an official Tamil scheme guideline]",
"response": "விண்ணப்பதாரர் அடையாளச் சான்று, முகவரி சான்று மற்றும் வருமானச் சான்றை சமர்ப்பிக்க வேண்டும். ஆவணங்களின் தற்போதைய பட்டியலை அதிகாரப்பூர்வ அறிவிப்பில் சரிபார்க்கவும்.",
"source_id": "tn_scheme_2026_014",
"task": "grounded_qa",
"citation_required": true
}Generate multiple difficulty levels from each source:
- Direct lookup of a clearly stated fact.
- Multi-step comparison across sections.
- Summarisation with a word limit.
- Ambiguous questions requiring clarification.
- Questions where the document does not contain the answer.
- Adversarial prompts that try to make the model invent eligibility, deadlines, or benefits.
Synthetic generation can accelerate drafting, but generated examples are proposals, not ground truth. Require human approval for every training record or use risk-based sampling for large collections.
Review for Tamil quality and factual grounding
Use at least two review layers for high-impact domains: a Tamil reviewer for fluency and register, and a subject reviewer for factual accuracy. Reviewers should check whether the response answers the instruction, preserves names and numbers, avoids unsupported claims, and distinguishes the source’s date from current policy.
Track common error categories:
- OCR substitutions and missing characters.
- Incorrect case markers or awkward machine-translated phrasing.
- Hallucinated details not present in the source.
- Overconfident answers to outdated documents.
- Unclear references such as “this office” or “the applicant.”
- Culturally inappropriate, stigmatizing, or discriminatory language.
Measure inter-annotator agreement on a sample before scaling. A simple rubric—grounding, Tamil fluency, completeness, safety, and citation quality—can use scores from 0 to 2, with mandatory rejection for factual or privacy failures.
Remove privacy, security, and licensing risks
Public documents may contain phone numbers, addresses, signatures, case details, or personal identifiers. Minimise or redact personal data unless it is essential to the task and legally justified. Exclude authentication information, private contact details, and unnecessary information about children or vulnerable people.
Also guard against prompt injection hidden in source documents. Treat document text as data, not instructions to the annotation model or training pipeline. Maintain a blocklist for secrets and run automated scans before release. Keep a deletion mechanism so a source or example can be removed from every dataset version.
Split, evaluate, and release the dataset responsibly
Prevent leakage by splitting at the document or source-family level, not randomly at the paragraph level. Otherwise, nearly identical passages may appear in both training and test sets. Keep a challenge set containing new documents, spelling variation, formal and spoken Tamil, tables, dates, and unsupported questions.
Evaluate both the base model and the tuned model on:
- Exact fact and number preservation.
- Citation and source attribution.
- Tamil grammatical and register quality.
- Abstention when evidence is missing or outdated.
- Robustness to code-mixed Tamil-English prompts.
- Performance across districts, domains, and user literacy levels.
Version the dataset with changelogs, source manifests, annotation guidelines, known limitations, and evaluation results. A veracity layer is especially important when models answer public-service questions; principles from data veracity infrastructure for high-stakes AI can help structure this audit trail.
For smaller teams, a simple repository with JSONL files, checksums, reviewer IDs, and automated validation is enough to begin. Build a feedback loop from real user queries, but sample and anonymise them before adding anything to training. If the project serves schools, public-service centres, or voice interfaces, connect dataset evaluation to the actual delivery environment rather than relying only on benchmark scores. Work on interactive live learning platforms for Indian schools and top-rated voice agent services for Indian businesses illustrates why channel-specific testing matters.
Practical launch checklist
Before fine-tuning, confirm that:
- Every source has provenance and a documented reuse basis.
- Raw, extracted, cleaned, and annotated data are kept separately.
- Tamil reviewers have approved language quality and terminology.
- High-risk content has subject-matter review.
- Personal data and secrets have been removed.
- Document-level splits prevent leakage.
- Unsupported and outdated questions are represented.
- Evaluation includes Tamil, code-mixed, and real-world task cases.
- Dataset versions can be reproduced, audited, and deleted when required.
The strongest Tamil instruction datasets are not simply the largest. They are grounded in traceable Indian sources, reviewed by people who understand Tamil in context, and designed to make models accurate about what they know—and honest about what they do not.