Apply for AI Grants India

Indic speech systems fail for predictable reasons: noisy recordings, inconsistent transcripts, weak language metadata, and evaluation sets that do not represent how people speak. Fine-tuning Whisper can address these gaps, but only when the dataset pipeline is designed carefully. This guide explains how to use Hugging Face datasets for fine-tuning Indic Whisper models, with a focus on automatic speech recognition (ASR) for Indian languages.

Whisper is an encoder-decoder speech-to-text model. It consumes audio and generates a transcript; it is not a text-to-speech model. For production work, treat dataset design, transcription policy, and evaluation as seriously as the training code.

Choose the right starting model and dataset

Start by identifying three decisions:

• Language: Hindi, Bengali, Tamil, Marathi, Malayalam, Kannada, Telugu, Gujarati, Punjabi, Odia, Assamese, or a multilingual requirement.
• Audio conditions: telephone calls, meetings, classrooms, field recordings, broadcast audio, or close-mic speech.
• Output policy: native script, transliteration, punctuation, numerals, code-switching, and handling of names or English terms.

Use a multilingual Whisper checkpoint or an Indic-focused checkpoint that matches your language and licence requirements. Do not assume that a model labelled “Indic” performs equally well across all languages or dialects. Low-resource languages need especially careful sampling and validation; the principles covered in this builder’s guide to low-resource Indic NLP apply directly to speech projects.

Hugging Face Datasets can load public corpora, private repositories, and local files. Relevant sources may include Common Voice, language-specific community datasets, and your own consented recordings. Before downloading, check the dataset card for licensing, speaker permissions, language labels, transcript conventions, and permitted commercial use. For more options, compare low-resource language datasets for AI training in India.

Structure the dataset for audio training

A useful dataset needs, at minimum, an audio column and a transcript column. A language column, speaker identifier, duration, region, and recording condition are also valuable.

A typical loading workflow looks like this:

from datasets import load_dataset, Audio

# Replace with the dataset repository and configuration you have verified.
dataset = load_dataset("your-org/indic-speech-dataset")
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset)
print(dataset["train"][0])

Whisper commonly uses 16 kHz audio. Let the Audio feature decode and resample files consistently rather than resampling ad hoc in different scripts. Inspect several examples after loading: confirm that the audio opens, the duration is plausible, the transcript is not empty, and the language label matches the recording.

Remove or quarantine problematic examples instead of silently training on them. Useful filters include:

• Missing or corrupt audio
• Empty, duplicated, or extremely short transcripts
• Clips with excessive duration
• Transcript-audio language mismatches
• Severe clipping, long silence, or unintelligible background noise

Keep a rejected-records report. It makes data cleaning auditable and helps you recover examples when a filtering rule is too aggressive.

Define a transcription policy before preprocessing

Indic ASR datasets often mix native scripts, Romanised text, English words, punctuation, and multiple spellings for the same name. Decide the target format before tokenisation. For example, you might preserve Devanagari for Hindi, retain English brand names in Latin script, and remove punctuation that annotators apply inconsistently.

Do not apply blanket lowercasing: it is not appropriate for every Indic script, and it can damage mixed-script text. Normalise only what improves consistency. Document choices for:

• Unicode normalisation
• Punctuation and quotation marks
• Numerals and dates
• Filled pauses and partial words
• Code-switching
• Abbreviations, names, and proper nouns

If your product requires transliteration or translation, train and evaluate that output separately from native-script transcription. A low word error rate on a normalised target may conceal poor usability for search, subtitles, or government workflows.

Prepare the processor and dataset

Use the processor associated with the selected checkpoint so that the feature extractor and tokenizer remain compatible. The exact class can vary by Transformers version and model repository.

from transformers import WhisperProcessor

checkpoint = "openai/whisper-small"  # Replace after checking language support
processor = WhisperProcessor.from_pretrained(
    checkpoint,
    language="Hindi",
    task="transcribe",
)

def prepare_batch(batch):
    audio = batch["audio"]
    batch["input_features"] = processor.feature_extractor(
        audio["array"],
        sampling_rate=audio["sampling_rate"],
    ).input_features[0]
    batch["labels"] = processor.tokenizer(
        batch["text"],
        add_special_tokens=True,
    ).input_ids
    return batch

prepared = dataset.map(
    prepare_batch,
    remove_columns=dataset["train"].column_names,
)

For large corpora, use num_proc carefully and cache intermediate results. Precomputing features can reduce repeated CPU work, but it increases storage requirements. If audio is private or sensitive, control cache locations and avoid uploading generated artifacts to a public repository.

Use a data collator that pads feature arrays and labels independently. Whisper training scripts from Transformers provide reference implementations; adapt them to your model rather than copying configuration blindly. In particular, verify decoder start tokens, forced language tokens, and suppression settings for the checkpoint you selected.

Split by speaker, not just by clip

Randomly splitting clips from the same speaker creates leakage and overstates performance. Build train, validation, and test sets at the speaker level. If you are measuring regional robustness, also hold out districts, accents, devices, or recording environments where possible.

A credible test set should contain the conditions your Indian users will encounter: mobile microphones, overlapping speech, code-switching, names, varying speaking rates, and background noise. Keep the test set frozen. Every later experiment should use the same benchmark so that improvements remain comparable. For broader evaluation planning, see this guide to Indian-language benchmark datasets.

Track more than one aggregate score. Report character error rate (CER) for script-sensitive analysis and word error rate (WER) where tokenisation is meaningful. Break results down by language, region, speaker gender where ethically appropriate, duration, noise level, and code-switching. Review a sample of errors manually; metrics alone cannot distinguish spelling-policy problems from genuine recognition failures.

Fine-tune conservatively

Begin with a small pilot run before committing to a full corpus. Confirm that training loss decreases, validation quality improves, and decoded predictions are intelligible. Common controls include:

• A low learning rate, often around 1e-5 as a starting point
• Gradient accumulation when GPU memory is limited
• Mixed precision on compatible hardware
• Gradient checkpointing for larger checkpoints
• Early stopping when validation quality stops improving
• Checkpoint retention and experiment logging

Avoid training for a fixed number of epochs without inspecting outputs. Overfitting is common when a language has limited labelled audio or when the same speakers dominate the corpus. Augmentation such as noise mixing, gain changes, or speed perturbation can help, but it should reflect real deployment conditions rather than introduce unrealistic audio.

Hardware planning matters. You can compare deployment choices with fine-tuning large language models on local hardware, but speech training has its own storage and audio-decoding bottlenecks. Track GPU memory, data-loader throughput, wall-clock time, checkpoint size, and inference latency—not only the final metric.

Evaluate, package, and deploy

Decode predictions on the untouched test set and compute metrics using the same normalisation policy used for references. Save the processor with the model; deploying weights without the matching tokenizer or feature-extractor configuration is a common source of errors.

from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("your-checkpoint")
model.save_pretrained("indic-whisper-finetuned")
processor.save_pretrained("indic-whisper-finetuned")

Create a model card that records the training languages, dataset licences, speaker consent, exclusions, transcript policy, known failure cases, evaluation splits, and intended use. For serving, benchmark batch and streaming inference on the hardware you plan to use. Quantisation can reduce cost, but re-test accuracy on names, numbers, and code-switched speech after optimisation. If you need a managed endpoint, review platforms for hosting custom fine-tuned models.

A practical pre-release checklist

Before shipping an Indic Whisper system, verify that:

• Audio is consistently sampled and successfully decoded.
• Speaker leakage is absent from validation and test sets.
• The transcript policy is documented and applied consistently.
• Licences and consent cover training and deployment.
• CER and WER are reported by language and important conditions.
• Human reviewers have examined representative errors.
• The processor, tokenizer, model, and generation configuration are versioned together.
• Monitoring captures drift, new accents, and user corrections.

Fine-tuning is only one part of the system. Better annotation, representative evaluation, and responsible data governance often produce larger gains than changing the checkpoint. For teams building broader Indic language systems, the same discipline used in training LLMs on Indian datasets is equally valuable here.