Start with the support job, not the model
A Bengali railway assistant should solve clearly defined passenger problems before it attempts broad conversation. Begin by mapping the highest-volume intents and the consequences of a wrong answer. A useful first release might cover:
- Train running status and delay explanations
- PNR and booking guidance, without exposing private passenger data
- Fares, quotas, concessions, refunds, cancellations, and boarding rules
- Platform, coach, station-facility, and accessibility information
- Lost-and-found, security, medical emergencies, and complaint registration
- Escalation to an official agent when a case needs account access or human judgement
Do not train the model to invent live railway information. Use the fine-tuned model for language understanding, intent classification, tone, and response structure; retrieve changing facts from approved railway systems or a controlled knowledge base. Teams building a wider multilingual stack can also review best practices for fine-tuning LLMs on custom data.
Build a Bengali-first dataset
The quality of the dataset will matter more than small changes to the learning rate. Collect de-identified conversations, call-centre transcripts, help-desk tickets, FAQs, and synthetic examples reviewed by Bengali-speaking staff. Obtain permission for every source and remove names, phone numbers, PNRs, payment details, addresses, and other personal information before annotation.
Represent the language passengers actually use. Bengali queries may contain English railway terms, transliterated Bengali typed in Latin script, spelling variation, abbreviations, and code-mixed Hindi or English. Include examples such as Bengali script, Banglish, and short mobile messages. Preserve meaningful regional usage, but do not let the dataset encode stereotypes about travellers or locations.
Each example should include fields such as:
- User message: the original Bengali or code-mixed query
- Intent: for example, refund status, train delay, or platform information
- Entities: train number, station, date, class, PNR, and journey direction
- Answer policy: what the assistant may state, request, or refuse
- Grounded response: an approved answer template or supporting document
- Escalation label: whether a human or official workflow is required
Create separate training, validation, and test sets by conversation or passenger case—not by randomly splitting near-identical messages. Keep a difficult, human-reviewed test set containing noisy spelling, dialect variation, code mixing, and ambiguous requests. This prevents data leakage and gives a realistic measure of production performance.
Choose the right adaptation strategy
For intent classification, entity extraction, and routing, a compact multilingual encoder may be sufficient and cheaper to operate. For conversational answers, start with a Bengali-capable instruction model and test parameter-efficient fine-tuning, such as LoRA or QLoRA, before considering a full-weight update. This reduces compute requirements and makes rollback easier for Indian teams.
Fine-tuning should teach the model how to respond, not memorise every timetable or fare. Pair it with retrieval-augmented generation for schedules, station facilities, policies, and operational notices. Keep source documents versioned, dated, and tagged by authority. If your product needs voice support, evaluate speech recognition and text-to-speech separately; a capable text model does not guarantee accurate Bengali audio. A review of voice agent services for Indian businesses can help frame the wider channel architecture.
Prepare and train the model
Normalise Unicode carefully without destroying Bengali characters or punctuation. Retain original user text alongside a normalised form so that evaluation reflects real input. Avoid blindly removing stop words: Bengali postpositions, negation, politeness markers, and short function words can change the meaning of a request.
A practical training process is:
1. Establish a zero-shot or base-model baseline.
2. Fine-tune on intent, entity, and response examples separately where possible.
3. Use a low learning rate, early stopping, and class-weighting or balanced sampling for rare but important intents.
4. Compare LoRA/QLoRA adapters with a full fine-tune on the same held-out test set.
5. Inspect errors by script, dialect, code-mixing, intent, and confidence band.
6. Keep the best checkpoint based on safety and task performance—not training loss alone.
Use approved refusal examples for requests involving unauthorised PNR access, payment credentials, guaranteed compensation, emergency decisions, or unsupported claims. The assistant should state when it cannot verify live information and direct passengers to an official channel.
Evaluate for railway reliability
Accuracy alone is inadequate. Report metrics per intent and per language form, not just one aggregate score. Useful measures include:
- Intent macro-F1: prevents high-volume status queries from hiding poor performance on safety or refund cases.
- Entity precision, recall, and F1: checks extraction of train numbers, stations, dates, and PNR-like identifiers.
- Grounded answer rate: measures whether responses are supported by the current approved source.
- Unsupported-claim rate: tracks fabricated timings, rules, fares, or guarantees.
- Escalation recall: measures whether risky or unresolved cases reach a human.
- Response latency and cost: confirms that the service works under expected Indian traffic patterns.
- Human ratings: assess Bengali fluency, politeness, clarity, usefulness, and cultural appropriateness.
Run adversarial tests for misspelled station names, multiple trains in one message, contradictory dates, code-mixed text, abusive language, prompt injection, and requests for another passenger’s information. Have Bengali language experts and railway-domain reviewers inspect failures before launch. Open-source work on AI projects for Indian languages may provide useful evaluation ideas, but production claims still need your own data.
Deploy with retrieval, controls, and monitoring
Put a policy and routing layer before the model. It should identify emergencies, redact sensitive values, check whether a request requires authentication, and route transactions to official APIs or human agents. The generation layer should receive only the minimum context required for the answer. Never treat a user-provided document or prompt as an authoritative railway instruction.
At launch, use a limited pilot with shadow evaluation or human approval for high-risk intents. Log model version, adapter version, retrieved sources, confidence, language form, latency, and escalation outcome—while enforcing retention and access controls. Monitor drift after timetable changes, policy updates, seasonal demand, and new station or service names. Refresh retrieval content first; retrain only when errors show a persistent language or behaviour gap.
A voice channel can be valuable for passengers who prefer speaking Bengali, but provide transcription confirmation and a keypad or human fallback. For broader multilingual deployment, open-source vision-language models for Indian languages may help with image-based queries such as tickets or station signs, provided privacy and verification controls are in place.
A practical launch checklist
Before production, confirm that you have:
- Consent, lawful processing, de-identification, and documented data retention
- Bengali and Banglish examples reviewed by native speakers
- A held-out, adversarial test set and per-intent results
- Versioned official knowledge sources and a retrieval freshness process
- Authentication and redaction for PNR, booking, and payment workflows
- Human escalation for emergencies, disputes, low confidence, and unsupported requests
- Monitoring for hallucinations, bias, latency, cost, and user complaints
- A rollback path for both the model and its knowledge base
The strongest Bengali railway assistant is not the one that answers every question. It is the one that understands passengers reliably, uses current official information, communicates clearly, protects personal data, and knows when to hand a case to a person.