How to Fine-Tune a Model on Indian Railway FAQs

Choose the right problem before training

Fine-tuning a model on Indian Railway FAQs can produce a useful question-answering assistant, but it should not be treated as a live railway information system by default. Timetables, fares, platform numbers, cancellations, seat availability, and operational notices change frequently. A model trained on static FAQs can explain policies and procedures; it cannot reliably provide real-time information unless it is connected to an approved, current data source.

For most teams, the practical goal is a FAQ assistant that answers questions such as:

How do I cancel a ticket?
What documents are required for a concession?
How does Tatkal booking work?
What is the refund rule for a waitlisted ticket?
Where can I find rules for luggage or boarding?

Before writing code, decide whether you need extractive question answering, retrieval-augmented generation, or supervised instruction tuning. The best practices for fine-tuning LLMs on custom data are especially relevant when the dataset is small, repetitive, or likely to contain outdated policy language.

Prepare a trustworthy FAQ dataset

Use official Indian Railways, IRCTC, railway-zone, or government sources wherever possible. Check the terms of use before scraping or redistributing content. Do not include passenger names, PNRs, phone numbers, payment details, or other personal information in a training repository.

A useful dataset should contain more than a question and an answer. Include fields such as:

{
  "question": "How can I cancel an e-ticket?",
  "answer": "Use the authorised booking service before chart preparation, subject to applicable rules.",
  "source_url": "https://example.gov.in/faq",
  "last_verified": "2026-01-15",
  "language": "en",
  "category": "cancellation"
}

Data quality checks matter more than dataset size. Remove duplicate questions, resolve contradictory answers, preserve important exceptions, and record the source and verification date. Split near-duplicates across train and validation sets carefully; otherwise, evaluation will look strong even though the model has simply memorised the wording.

For multilingual use, add human-reviewed Hindi or regional-language versions rather than relying entirely on machine translation. If your product needs voice interaction, pair the text system with the design principles covered in top-rated voice agent services for Indian businesses, including escalation and confirmation for high-impact requests.

Select a model and task format

For a controlled FAQ assistant, begin with a small encoder model such as DistilBERT or another suitable multilingual checkpoint. An extractive question-answering model is appropriate when every answer can be found in a supplied context passage. For a multilingual Indian deployment, compare models that support the languages your users actually speak and test them on code-mixed queries such as “ticket cancel kaise karna hai?”.

A generative model may produce smoother answers, but it can also invent rules. Retrieval-augmented generation is often safer: retrieve the relevant, current FAQ and ask the model to answer only from that evidence. Fine-tuning should teach the model the task and tone; retrieval should supply changing facts.

Install the Hugging Face stack

Create an isolated Python environment and install compatible versions of the core libraries:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\\Scripts\\activate
pip install -U torch transformers datasets evaluate accelerate

If you use a hosted GPU, confirm the CUDA and PyTorch versions before training. A modest dataset may fit on a consumer GPU, while parameter-efficient methods such as LoRA can reduce memory requirements for larger instruction models. Do not assume that a GPU is necessary for data cleaning, tokenisation, or a first evaluation run.

Format and tokenise extractive QA data

For extractive question answering, each record needs a question, a context, an answer string, and the answer’s character position in the context. A simplified record looks like this:

{
  "id": "faq-001",
  "question": "How can I cancel an e-ticket?",
  "context": "Passengers can cancel an e-ticket through the authorised booking service, subject to applicable timing and refund rules.",
  "answers": {
    "text": ["cancel an e-ticket through the authorised booking service"],
    "answer_start": [13]
  }
}

The answer_start value must point to the exact answer span. Validate offsets programmatically after normalisation; punctuation or whitespace changes can otherwise make labels invalid. Long contexts require a sliding window and an appropriate doc_stride so that answers near the boundary are not lost.

from datasets import load_dataset
from transformers import AutoTokenizer

model_name = "distilbert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
dataset = load_dataset("json", data_files={
    "train": "train.json",
    "validation": "validation.json"
})

def prepare_features(batch):
    return tokenizer(
        batch["question"],
        batch["context"],
        truncation="only_second",
        max_length=384,
        stride=128,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length"
    )

In production code, map answer offsets to token start and end positions, retain the overflow-to-example mapping, and remove offset metadata before passing features to the trainer. The Hugging Face question-answering documentation provides the exact preprocessing pattern for this step.

Fine-tune and track experiments

Load the model for question answering and train with a validation set that reflects real user queries. Start conservatively: a low learning rate, two or three epochs, and early stopping are safer than aggressive training on a small corpus.

from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer

model = AutoModelForQuestionAnswering.from_pretrained(model_name)
args = TrainingArguments(
    output_dir="./railway-faq-model",
    learning_rate=3e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    report_to="none"
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_validation,
    tokenizer=tokenizer
)
trainer.train()
trainer.save_model("./railway-faq-model")
tokenizer.save_pretrained("./railway-faq-model")

Library argument names can change between Transformers releases, so pin tested versions in requirements.txt and record the base model, dataset commit, hardware, seed, and hyperparameters. Uploading the model to Hugging Face Hub is useful for collaboration, but set the repository visibility and licence deliberately, especially if the data is derived from a government or commercial source.

Evaluate usefulness, not just loss

Report exact match and token-level F1 for extractive QA, but add operational tests that metrics miss:

Test spelling variations, abbreviations, Hindi-English code-mixing, and low-quality mobile input.
Include unanswerable questions and verify that the system refuses instead of guessing.
Check answers against the source URL and verification date.
Measure latency, memory usage, and cost on the intended deployment hardware.
Review errors by category: booking, cancellation, refund, concessions, luggage, accessibility, and safety.

Create a small, human-reviewed “golden set” of real queries before training. Keep it private from the training process and rerun it after every dataset or model change. For customer-facing systems, provide a clear escalation path to an official channel. A model should never request or expose OTPs, passwords, payment credentials, or full passenger records.

Deploy with freshness and safeguards

A practical architecture combines a versioned FAQ index, retrieval, the fine-tuned model, and an answer policy. Show the source and last-verified date with each response. Add a confidence threshold: below it, return a carefully written fallback rather than a plausible-sounding answer. Refresh the index when official rules change, and retrain only when evaluation shows that fine-tuning adds value.

If you are building a broader public-service assistant, study open-source vision-language models for Indian languages only when images, scanned notices, or multilingual documents are genuinely part of the workflow. For text-only FAQs, adding model complexity rarely improves reliability by itself.

Finally, monitor unanswered questions, policy conflicts, language gaps, and user corrections. Use this feedback to improve source coverage and retrieval first. Fine-tune again only after removing stale or contradictory examples. This approach keeps the assistant useful, auditable, and safer for Indian railway passengers.

Apply for AI Grants India