LLM applications fail in ways that conventional application monitoring does not capture. An API can return a 200 response while producing an incorrect answer, exposing sensitive data, calling the wrong tool, or quietly increasing inference costs. Building custom LLM observability tools means creating the evidence needed to understand those failures and improve the system without guessing.
For Indian teams, this often means observing more than a model endpoint. A production workflow may combine a multilingual prompt, retrieval from internal documents, a hosted model, a voice or chat interface, business rules, and several external APIs. Your observability layer should connect these steps into one trace while respecting privacy, data residency, and cost constraints.
What LLM observability should answer
A useful system answers four questions for every important interaction:
- What happened? Which user request, prompt version, model, tools, retrieved documents, and output were involved?
- How well did it work? Was the answer correct, grounded, safe, relevant, and useful?
- Why did it fail? Was the issue caused by retrieval, prompting, tool execution, model behaviour, latency, or upstream data?
- What should change? Can the team reproduce the case, test a fix, and verify that quality improved?
This is different from storing every prompt and response in a dashboard. Observability should support an operational loop: detect, investigate, correct, evaluate, and deploy.
Design the telemetry model first
Start with a canonical event schema rather than choosing a dashboard or database. A practical trace can include:
- Request metadata: trace ID, tenant or application ID, timestamp, region, language, channel, and environment.
- Model information: provider, model name, version, temperature, token limits, system-prompt version, and fallback path.
- Workflow spans: retrieval, reranking, moderation, tool calls, model generation, post-processing, and human review.
- Performance data: queue time, time to first token, total latency, input and output tokens, retries, and timeout causes.
- Quality signals: evaluator scores, citations, refusal category, user feedback, escalation, and task completion.
- Cost data: per-call price, cached-token usage, tool charges, and estimated cost by tenant or feature.
Use stable IDs to connect a user session, workflow run, model call, and tool invocation. OpenTelemetry-compatible traces are a sensible foundation, but LLM-specific attributes still need to be defined by your team. Keep prompt templates, model configuration, evaluation datasets, and application releases versioned so that a quality regression can be tied to a concrete change.
Instrument the full LLM workflow
Instrumentation should sit at the boundaries where decisions and failures occur. Capture a parent trace for the user request, then create child spans for each significant operation. For a retrieval-augmented generation application, the trace might be:
1. Receive and classify the request.
2. Detect language, intent, and sensitive entities.
3. Generate an embedding and retrieve candidate passages.
4. Rerank and filter the context.
5. Call the model and record streaming timings.
6. Execute any tools or business APIs.
7. Validate the output for schema, safety, and grounding.
8. Return the answer or route the case to a human.
This level of detail is valuable for applications that use agents or distributed services. Teams designing those systems can also review the architecture principles in building distributed systems with AI agents. For Indian-language applications, log language and script as controlled fields rather than inferring them later from raw text.
Do not treat raw prompts as mandatory telemetry. Default to redaction, token-level masking, hashing, or sampled storage. Store full content only when there is a documented debugging or evaluation need, with access controls, retention limits, and audit logs. Separate operational metadata from sensitive payloads so engineers can diagnose latency without automatically seeing customer conversations.
Measure more than latency and uptime
A production dashboard should combine system, model, and business metrics.
System metrics
Track p50, p95, and p99 latency; time to first token; throughput; queue depth; error and timeout rates; retry counts; and provider availability. Break these down by model, endpoint, geography, language, device, and tenant. A single average latency number hides the experience of users on slower networks or during provider failover.
Cost metrics
Measure input and output tokens, cache-hit rate, cost per successful task, and cost by workflow. Set budgets at both application and tenant level. A cheaper model that requires repeated retries or produces more escalations may cost more overall.
Quality metrics
Use a mixture of automated and human signals:
- Groundedness and citation correctness for retrieval workflows.
- Structured-output validity and tool-call success rate.
- Relevance, completeness, and instruction adherence.
- Hallucination, toxicity, prompt-injection, and sensitive-data leakage rates.
- User corrections, thumbs-down events, abandonment, escalation, and repeat queries.
Automated judges are useful for triage, not as unquestioned truth. Maintain a labelled evaluation set containing real Indian usage patterns, code-switching, spelling variation, regional terminology, and difficult edge cases. Where the model supports a specialised domain, pair generic quality scores with task-specific acceptance criteria. For example, a fintech onboarding assistant should be assessed on correct document requests and compliant escalation, not merely conversational fluency.
Build an evaluation and regression loop
Observability becomes valuable when it changes engineering decisions. Sample production traces into an evaluation queue, remove or mask personal data, and run checks against a fixed test set. Compare the current release with the previous one across quality, safety, latency, and cost.
Use three evaluation layers:
- Offline tests: deterministic checks for schemas, retrieval, prompt-injection resistance, and known failure cases.
- Continuous evaluations: scheduled or release-triggered scoring on representative datasets.
- Online signals: user feedback, task completion, escalation, and statistically monitored drift.
Segment results by language, customer type, workflow, and model route. A model can improve its overall score while degrading sharply for a smaller language group. If your application depends on custom domain knowledge, pair observability with best practices for fine-tuning LLMs on custom data, especially when deciding whether a failure needs better retrieval, fine-tuning, or a revised business rule.
Alert on actionable conditions
Avoid alerts for every small score fluctuation. Define thresholds tied to user or business impact, such as:
- p95 latency exceeding the agreed service objective for 10 minutes.
- Tool-call failures crossing a workflow-specific limit.
- Groundedness falling below the release baseline.
- Token cost per completed task rising by a defined percentage.
- A sudden increase in refusals, unsafe outputs, or sensitive-data detections.
- A quality drop concentrated in one language, region, or customer segment.
Each alert should include the affected release, model, prompt version, sample trace IDs, likely contributing span, and a recommended owner. Route operational incidents to the existing on-call system rather than creating a separate notification silo.
Choose a practical architecture
A lean first version can use an application middleware layer, an OpenTelemetry collector, object storage for sampled payloads, a relational database for metadata, and a dashboard or query interface. Stream high-volume events through a queue, then aggregate them into low-cardinality metrics. Keep detailed traces available for investigation without forcing the analytics store to index every token or full response.
Build the minimum useful slice first: one critical workflow, one dashboard, one evaluation set, and three alert rules. Add agent graphs, semantic search over traces, and advanced drift detection only after the basics are reliable. Open-source components can reduce vendor lock-in, but teams should budget for schema maintenance, security hardening, storage costs, and evaluator validation.
Privacy, security, and governance
Treat observability data as production data. Apply role-based access, encryption, retention schedules, tenant isolation, and deletion workflows. Record consent and purpose where conversations may contain personal or financial information. Avoid sending raw customer content to an external evaluator unless contractual and security requirements permit it.
For systems serving India’s next wave of users, privacy and accessibility must be designed alongside scale. The practical lessons in building AI apps for the next billion users in India are relevant here: unreliable connectivity, multilingual input, shared devices, and low-cost inference all affect what should be measured and how results should be interpreted.
A 30-day implementation plan
- Week 1: Map one workflow, define the event schema, classify sensitive fields, and select baseline quality cases.
- Week 2: Add tracing middleware, model and tool spans, latency metrics, token accounting, and redaction.
- Week 3: Build dashboards, offline evaluations, trace sampling, and three actionable alerts.
- Week 4: Review failures with product and operations teams, tune thresholds, document incident response, and establish a release gate.
Success is not the number of charts. It is the time required to move from “users report bad answers” to a reproducible trace, a verified root cause, and a tested fix.
Frequently asked questions
Should a startup build custom observability or buy a platform?
Start with a platform or open-source foundation when speed matters. Build custom components where your workflow, privacy requirements, language coverage, or cost model is not well served by generic tooling.
Should every prompt and response be stored?
No. Use redaction, sampling, retention limits, and access controls. Keep full payloads only for defined debugging, evaluation, or compliance purposes.
How do I detect hallucinations automatically?
Use retrieval-grounded checks, citation validation, structured assertions, domain rules, and sampled human review. No single evaluator is reliable across every task or language.
What is the first metric to implement?
For most teams, begin with end-to-end latency, error rate, token cost, trace completeness, and a small labelled quality set. These establish an operational baseline before more sophisticated scoring.
Support AI development in India
Teams building evaluation, reliability, and monitoring infrastructure for Indian-language or domain-specific AI can explore AI Grants India for potential funding and ecosystem support.