A useful AI agent needs more than a large context window. It must retain relevant facts, recover after failures, distinguish current instructions from stale information, and forget data when a user or organisation requests deletion. That is the difference between a chatbot that generates plausible replies and a dependable system that can support customers, employees, and operational workflows.
This guide explains how to build AI agents with memory for production applications in India. The focus is not on adding every conversation to a vector database. It is on designing a controlled memory system with clear write rules, retrieval boundaries, durable state, and measurable behaviour.
Start with a memory model, not a database
Treat memory as several different forms of state. Each has a different lifetime, access policy, and storage method:
- Working memory: Tool results, intermediate decisions, and the current task plan. It usually lives only for one run.
- Conversation memory: Recent messages and the active session summary. It helps the agent resolve references and maintain continuity.
- Semantic memory: Durable facts that may be useful later, such as a customer’s preferred language or an approved vendor.
- Episodic memory: Significant past events, such as a failed payment attempt or a completed service request.
- Procedural memory: Rules, policies, and repeatable workflows. Store these as versioned configuration or documents rather than personal memories.
This separation prevents a common design error: treating an old chat transcript, a user preference, and a compliance policy as interchangeable text. A voice agent handling restaurant bookings, for example, may need to remember a preferred language and dietary requirement, while retrieving the latest menu from a controlled knowledge source. See the practical architecture in how to build a voice agent for a related deployment pattern.
Define what the agent may remember
Before choosing embeddings or a framework, create a memory policy. For every memory type, specify:
- Purpose: What future task does this memory improve?
- Owner: Which user, organisation, account, or workspace does it belong to?
- Sensitivity: Does it contain identity, financial, health, or confidential business information?
- Retention: When should it expire or be reviewed?
- Source and confidence: Who supplied it, when was it observed, and how reliable is it?
- Correction path: How can a user amend or delete it?
A good write rule is conservative: store only information that is likely to improve a future interaction and is appropriate to retain. Do not automatically save every model-generated inference. “The user may be travelling next month” is not the same as “the user prefers aisle seats.” Ask for confirmation when a fact affects money, eligibility, healthcare, legal decisions, or access control.
Build short-term memory with durable state
The current run needs a compact state object rather than an ever-growing transcript. A practical schema might include:
{
"conversation_summary": "Customer is resolving an invoice mismatch.",
"open_tasks": ["Verify invoice number", "Request purchase order"],
"recent_messages": [],
"tool_results": [],
"user_id": "user_123",
"policy_version": "billing-v4"
}Keep the last few turns verbatim for precision, maintain a rolling summary for older context, and record open tasks separately. Summaries should be regenerated from trusted conversation data, not repeatedly summarised summaries, which can compound errors.
For multi-step agents, persist checkpoints after meaningful transitions: before a tool call, after a successful result, and whenever human approval is required. A graph-based orchestrator such as LangGraph can model these transitions, but the principle is framework-independent. The state must be serialisable, versioned, and recoverable if a worker, API, or network connection fails. This matters especially for distributed systems with AI agents, where retries and duplicate tool calls are normal failure modes.
Add long-term memory through retrieval
Long-term memory usually combines structured records with semantic search. Use a relational database for facts that require exact filtering—customer ID, consent status, language, account tier, or deletion state. Use vector search for fuzzy recall of relevant notes, summaries, and events. PostgreSQL with pgvector is often a strong starting point because it keeps metadata, access controls, and embeddings in one system.
A robust retrieval flow looks like this:
1. Classify the request and identify the user or workspace scope.
2. Apply hard filters for tenant, permissions, data type, and retention status.
3. Retrieve a small candidate set using keyword, vector, or hybrid search.
4. Re-rank candidates using recency, source quality, confidence, and task relevance.
5. Remove duplicates and contradictions before prompt injection.
6. Present concise memory snippets with source and timestamp metadata.
Do not retrieve memories solely because they are semantically similar. A five-year-old preference may be less useful than a confirmed preference from yesterday. A simple scoring model can combine similarity, recency, confidence, and importance. The agent should also be able to answer “I don’t know” when no memory clears a minimum relevance threshold.
Use structured memory for important facts
For preferences and account details, a JSON record is usually safer than an embedding alone:
{
"key": "preferred_language",
"value": "Hindi",
"source": "user_confirmed",
"confidence": 1.0,
"observed_at": "2026-02-10",
"expires_at": null
}Maintain one canonical value where possible, retain an audit trail for changes, and resolve conflicts explicitly. If a user says they now prefer English, the system should update the structured profile rather than retrieve two contradictory notes and ask the model to guess.
For sensitive sectors, isolate memory by tenant and purpose. A hospital assistant should not expose one patient’s history to another workflow, and a legal assistant should not mix one client’s documents with another’s. The privacy concerns discussed in private AI chatbots for lawyers are equally relevant to any agent storing confidential long-term context.
Control memory writes and tool actions
The model should propose a memory write, but application code should validate it. A memory service can enforce schemas, redact prohibited fields, attach consent and retention metadata, and reject unsupported claims. Keep write operations separate from retrieval so you can audit what was saved and why.
Use idempotency keys for tool calls and memory updates. If a request is retried, it should not create duplicate bookings, tickets, or memories. Require human approval for irreversible actions, high-value transactions, medical decisions, or changes to permissions. Memory should improve decision-making; it must not silently become an authority that overrides current user instructions or system policy.
Privacy and security for Indian deployments
Long-term memory can contain personal data, so design for the Digital Personal Data Protection Act and applicable sectoral obligations from the start. Practical controls include:
- Obtain clear, purpose-specific consent where required.
- Minimise collection and avoid storing raw transcripts by default.
- Encrypt data in transit and at rest; protect embedding stores as sensitive data.
- Enforce tenant-level access controls and keep immutable audit logs.
- Provide export, correction, and deletion workflows.
- Define retention periods and automated expiry jobs.
- Redact tokens, passwords, health details, and unnecessary identifiers before indexing.
- Keep production data out of prompts, logs, and evaluation sets unless authorised.
For healthcare, map the memory design to the organisation’s clinical governance and security requirements rather than assuming a generic vector database is sufficient. Voice and multilingual interfaces also need careful consent and transcript handling; multilingual voice agents for restaurants in India illustrates why language preference and channel continuity should be treated as explicit product requirements.
Evaluate memory like a product feature
Test memory with scenarios, not just response quality. Build a dataset covering:
- Correct recall of a confirmed preference.
- Rejection of an unverified inference.
- Conflict between old and new information.
- Cross-tenant isolation.
- Deletion and expiry behaviour.
- Recovery after a failed tool call.
- Prompt injection inside retrieved memories.
- Requests where the correct answer is to ask for clarification.
Track recall precision, stale-memory rate, unsupported-memory writes, retrieval latency, token cost, deletion completion time, and task success. Sample production traces with privacy controls and review both false memories and missed memories. A memory system that recalls less but accurately may be more valuable than one that fills every prompt with plausible noise.
A practical implementation path
For a first production version, use PostgreSQL for structured profiles and task state, pgvector or a managed vector store for selected semantic memories, and a graph or state-machine orchestrator for checkpointing. Begin with one narrow workflow—such as customer support follow-up or appointment scheduling—then add long-term memory only after you can measure its benefit.
A sensible sequence is:
1. Implement session state and conversation summaries.
2. Add structured, user-confirmed preferences.
3. Introduce hybrid retrieval with strict tenant filters.
4. Add checkpointing, retries, and idempotent tools.
5. Implement deletion, retention, and audit controls.
6. Evaluate on real task scenarios before expanding memory scope.
The goal is not an agent that remembers everything. It is an agent that remembers the right information, for the right period, with the right evidence, and can explain or remove that memory when required.