Telugu instruction-tuning data should do more than convert documents into question-and-answer pairs. A useful dataset captures Telugu’s grammar, script, formal and conversational registers, domain terminology, and the realities of Indian public services. It must also be traceable, legally defensible, and evaluated by people who understand the language.
This guide explains how to build that dataset in 2026, whether you are preparing data for an open model, a government-facing assistant, a voice system, or a domain-specific Telugu application.
Define the model behaviour first
Start with the outcomes you want from the model, not with a pile of documents. Write a short task specification covering:
- Users: citizens, students, researchers, customer-support agents, or government staff.
- Languages: Telugu only, Telugu-English code-switching, or Telugu with transliterated input.
- Tasks: summarisation, document question answering, extraction, translation, classification, rewriting, or conversational support.
- Risk level: general information is different from health, legal, financial, or welfare guidance.
- Output rules: script, tone, length, citations, refusal behaviour, and whether the answer must quote the source.
A balanced first release might include summarisation, fact extraction, question answering, simplification into plain Telugu, and translation of administrative terms. For model-training decisions, pair this workflow with best practices for fine-tuning LLMs on custom data, especially when deciding whether supervised fine-tuning is preferable to retrieval-augmented generation.
Select and document Indian public sources
Potential sources include Andhra Pradesh and Telangana government portals, district notices, public-sector reports, legislative material, court judgments, census publications, university resources, scheme guidelines, and public educational content. Do not assume that “publicly available” means “free to reuse.” Record the URL, publisher, retrieval date, licence or terms of use, language, document type, and any restrictions.
Prioritise sources that are:
- Authoritative: issued by a recognised public institution.
- Stable: available through a persistent URL or downloadable archive.
- Relevant: representative of the intended user’s real tasks.
- Diverse: covering urban and rural administration, education, agriculture, public health, transport, and social welfare.
- Readable: available as native Telugu text rather than low-quality scans whenever possible.
Create a source manifest before processing. Assign every document a stable identifier and retain the original file. This makes later audits, takedowns, corrections, and dataset versioning possible. For sensitive or high-stakes domains, establish a verification process similar to ICMR-compliant medical AI data verification in India, even when the final use case is not medical.
Extract Telugu text without destroying meaning
Indian public documents often arrive as PDFs, scanned circulars, web pages, spreadsheets, or images. Extraction quality is a major determinant of model quality.
Use a format-specific pipeline:
1. Extract native Unicode text from digital PDFs and HTML.
2. Apply Telugu-capable OCR to scanned pages.
3. Preserve page, section, table, heading, and paragraph boundaries.
4. Store OCR confidence and flag low-confidence pages for review.
5. Remove navigation menus, repeated headers, advertisements, page numbers, and boilerplate.
6. Keep tables as structured records rather than flattening them into unreadable strings.
Do not silently normalise every variation. Telugu punctuation, numerals, abbreviations, spacing, and spelling may vary across institutions. Store the raw text, a cleaned version, and any normalised version separately. This allows researchers to reproduce the pipeline and prevents irreversible editorial decisions.
Deduplicate at both document and passage level. Government sites frequently publish the same circular in multiple locations, while annual reports may repeat standard clauses. Near-duplicate removal reduces memorisation and prevents one source from dominating the training mix.
Design instruction examples grounded in documents
Each example should connect a clear instruction to an evidence-backed answer. A practical JSONL record might contain:
{
"id": "ts-000184",
"instruction": "ఈ నోటీసులో అర్హత ప్రమాణాలను సంక్షిప్తంగా వివరించండి.",
"input": "[document passage]",
"output": "[verified Telugu response]",
"source_id": "telangana-scheme-2025-014",
"task": "summarisation",
"language": "te",
"risk": "medium"
}Build several task families instead of generating thousands of superficial prompts:
- Extractive QA: answer only from a named passage.
- Abstractive summarisation: preserve conditions, dates, amounts, and exceptions.
- Plain-language rewriting: explain formal administrative Telugu for ordinary readers.
- Structured extraction: return fields such as deadline, eligibility, office, and required documents.
- Cross-lingual assistance: translate Telugu material into English or explain English administrative terms in Telugu.
- Refusal and uncertainty: say when the document does not contain the answer or is outdated.
For every answer, distinguish source-supported facts from generated interpretation. Require citations, document IDs, or quoted evidence for high-stakes tasks. Avoid asking a model to invent missing details simply to make an example look complete.
Make Telugu quality measurable
Human review is essential. Recruit Telugu language experts who can assess grammar, spelling, register, factual fidelity, and naturalness. Include reviewers familiar with public administration, because a linguistically fluent answer can still misread eligibility conditions or legal qualifiers.
Use a rubric with separate scores for:
- Faithfulness: does the answer match the source?
- Completeness: are important conditions and exceptions retained?
- Telugu quality: is the language grammatical and natural?
- Instruction following: is the requested format and length respected?
- Safety: does the answer avoid unsupported advice or personal-data exposure?
- Cultural and regional coverage: does the dataset avoid overrepresenting one register or locality?
Measure inter-annotator agreement on a sample before scaling. Adjudicate disagreements, record the decision, and update the annotation guide. Keep a challenge set containing OCR errors, mixed Telugu-English text, dates, currency, tables, government abbreviations, and deliberately unanswerable questions.
Evaluate against a held-out set whose source documents never appear in training. Test both Telugu prompts and realistic user inputs, including spelling variation and code-switching. If the system will support speech, test it separately with noisy transcripts; text-only scores do not guarantee good performance in a voice workflow. Related deployment considerations appear in top-rated voice agent services for Indian businesses.
Handle privacy, copyright, and contamination
Remove personal phone numbers, addresses, identity numbers, signatures, case-specific medical details, and other unnecessary personal data. Public records may still contain protected information. Apply automated detection followed by human review, and maintain a deletion log.
Keep licences and provenance with every record. Exclude material whose reuse rights are unclear, or obtain permission before release. Separate training, validation, and test sources by document or publisher to reduce leakage. Never use the same passage to create both a training answer and an evaluation question.
Package a reproducible dataset
Release more than a JSONL file. Include:
- source manifest and licence notes;
- extraction and cleaning code;
- annotation guidelines;
- schema and field definitions;
- dataset statistics by domain, task, and register;
- known OCR and coverage limitations;
- train, validation, and test splits;
- version history and correction policy;
- a datasheet describing intended and prohibited uses.
Track Telugu script coverage, average example length, duplicate rates, OCR confidence, human-review rates, and the percentage of examples with source citations. A small, verified dataset is usually more valuable than a large synthetic collection with weak provenance. Teams building open tooling can also study Indian open-source AI developer projects for practical approaches to publishing code and evaluation assets.
A practical pilot plan
For a first 2026 pilot, select 100–300 documents across three domains, process them with preserved provenance, and create several hundred expert-reviewed examples across five task types. Run a baseline model before tuning, then compare factuality, Telugu quality, refusal behaviour, and performance on unseen sources. Use the error analysis to improve extraction and annotation before expanding the corpus.
The objective is not merely to make a model speak Telugu. It is to make the model useful, honest about uncertainty, and reliable when handling the documents Indian users actually read. Strong source governance, careful Telugu review, and reproducible evaluation will matter more than raw dataset size.