Agent Memory Architecture: From Chat History to Reusable Long-Term Experience

Context management within a single session is relatively mature in agent applications. However, once a task spans multiple sessions, several problems quickly emerge. An agent may repeat an experiment even after validating the same conclusion in a previous session. It may forget a rule that the user explicitly corrected. Interaction logs may continue to grow without becoming reusable experience.

Addressing these problems requires more than accumulating historical conversations. Memory must become an engineering system that can be written, replayed, validated, injected into context, and governed.

One useful design approach is inspired by memory consolidation during sleep and rest. While interacting with users, an agent captures important information quickly. During idle periods, it replays recent experiences, extracts valuable patterns, and corrects errors. In an agent system, this corresponds to running background “dreaming” jobs between interactions. Through repeated replay and recombination of historical records, the system can build a long-term memory loop that works across sessions.

This article examines the architecture from several perspectives, including memory layers, offline replay, key-event tagging, the learning loop, evaluation, observability, and engineering governance.

1. Defining the Boundary: What Counts as Agent Memory?

Before designing a memory architecture, it is important to define its scope.

Agent memory generally refers to information acquired through task execution and user interaction that can persist across sessions and later be retrieved as plain-text context or another searchable representation. The defining characteristic is that the information can be called upon later. It is not merely temporary context used once.

Three mechanisms should be clearly excluded:

1. Model weight updates: Memory should not be confused with continued training or fine-tuning. Weight updates are expensive, difficult to delete at a fine-grained level, and unsuitable for frequently changing user preferences and task-specific experience.
2. In-session context learning: Keeping information in the context window during one conversation is short-term context management, not cross-session long-term memory.
3. Replacing old conversations with summaries: Summarization helps control context-window size, but a summary may not be validated or traceable. It cannot replace structured memory by itself.

With this boundary in place, the goal becomes clearer: convert observations, rules, facts, failure causes, and corrective conclusions from interactions into external experience that can be retrieved, verified, and audited later. The objective is not to preserve every conversation without distinction.

2. A Layered Memory Architecture

A practical memory architecture typically consists of four layers. Offline “dreaming” connects and processes all of them.

Layer 1: Raw Records

The raw-record layer stores original session logs, tool outputs, user feedback, task states, and other source materials.

Its primary value is traceability. When a later memory conflicts with another memory or proves to be incorrect, the system can return to the original evidence for verification. Raw records should not normally be injected into the context in full, because doing so creates noise and wastes the context budget.

Layer 2: Memory Notes

During or after a session, the agent converts important observations into concise notes, such as:

• “This type of query requires validating field types first.”
• “The user prefers the conclusion before the step-by-step explanation.”
• “This rule does not apply under a particular condition.”

The notes layer is an intermediate state between raw records and long-term experience. It emphasizes readability, editability, and source attribution.

Layer 3: Memory Representations

Memory notes are organized into structured categories, such as rules, facts, preferences, failure patterns, and operating procedures.

This layer can combine vector retrieval, keyword indexes, time decay, and importance weights. Its purpose is to answer a practical question: how can the system quickly find experience relevant to the current task?

Layer 4: Injected Context

When a new task begins, the system retrieves memories related to the current objective, trims them according to a fixed budget, and injects the selected items into the model context.

The injection layer must control both quantity and relevance. Injecting a large number of low-value memories can distract the agent and reduce task quality.

The “dreaming” process runs across these layers. It reads raw records and notes, generates candidate experience, validates that experience, updates the representation layer, and influences future tasks through context injection. This creates a loop from recording to consolidation, from consolidation to retrieval, and from feedback to correction.

3. Offline “Dreaming”: Replay, Recombination, Validation, and Consolidation

During live interaction, the agent’s primary goal is to complete the task. It has limited opportunity to perform large-scale reflection. Offline replay uses idle compute between interactions to process historical records a second time.

The main steps are:

1. Replay: Retrieve recent sessions, failure cases, user corrections, and tool results.
2. Recombination: Combine scattered observations into rules or hypotheses and identify common patterns across similar tasks.
3. Validation: Check candidate experience against external answers, tool execution results, user feedback, or consistency checks.
4. Consolidation: Write validated conclusions into the memory catalog while recording their sources, confidence levels, validity periods, and conflicts.
5. Pruning: Reduce the weight of, merge, or delete memories that are outdated, low-confidence, duplicated, or contradictory.

A simplified pseudocode example is shown below:

on_session_end(session):
    raw = persist_interaction(session)
    key_events = extract_key_events(session)
    notes = update_memory_notes(raw, key_events)
    enqueue_offline_job(raw, notes)
offline_reflect(job):
    candidates = replay_and_recombine(job.raw, job.notes)
    validated = validate_against_evidence(candidates)
    memory = merge_with_conflict_policy(validated)
    publish(memory)

The important point is not the specific function names, but the engineering principles behind them:

• Offline jobs should be asynchronous, retryable, and auditable.
• Candidate experience must be validated before entering long-term memory.
• Memory merging requires an explicit conflict policy rather than simple overwriting.

4. Key-Event Tagging: Predicting What May Matter Later

The human brain uses emotional and chemical signals to tag important experiences, giving them priority during later consolidation. An agent can adopt a similar mechanism by tagging events that appear important during a session or writing them to an external memory catalog for prioritized offline processing.

However, this creates a difficult question: how can an agent predict which event will matter in the future?

The answer is not to seek perfect prediction. Instead, the system can use multiple trigger signals:

• Explicit feedback: User corrections, rejections, and newly added constraints should receive high priority.
• Failures and anomalies: Tool errors, validation failures, and repeated attempts should be recorded as potential failure patterns.
• Task completion: Important steps and verified conclusions from successful tasks may be worth preserving.
• Rule conflicts: When a new observation contradicts an existing memory, the system should trigger replay and verification.
• Periodic sampling: Low-priority logs should be sampled for replay to reduce the risk of missing useful patterns.

Writing a note is itself a prediction about the future. Recording a rule predicts that the rule may be useful again. Recording a verified fact predicts that the system will not need to validate it from scratch the next time it is used.

For this reason, memory writing should not optimize for completeness. It should optimize for reuse, verifiability, and traceability.

5. The Learning Loop: From Scattered Guesses to Verified Rules

Long-term memory can be evaluated with database lookup and question-answering tasks. In such a setup, the agent retrieves information to answer questions. After each answer, the system reveals the correct result. The agent can then store the experience in a memory catalog for use in later sessions.

Comparing a stateless baseline with a system that stores memory makes it possible to observe metrics such as cumulative reward and repeated-error rate.

Agents at different levels of maturity often exhibit different learning patterns:

• Early stage: The agent generates unverified guesses. The memory catalog contains fragmented theories, and repeated errors remain common.
• Intermediate stage: The agent begins integrating notes and combining similar observations into rules, but still requires human or external validation as a safeguard.
• Mature stage: The agent can distinguish facts, rules, and hypotheses, record sources and confidence levels, and continuously resolve conflicting memories through replay.

This shows that the value of a memory system lies not only in storing information, but also in correcting mistakes.

Without validation and pruning, memory becomes a noise amplifier. With replay, validation, conflict resolution, and feedback from context injection, the system can form an autonomous memory-learning loop.

6. Engineering Considerations and Governance

When implementing this architecture, consider the following areas:

1. Storage separation: Store raw logs, memory notes, and structured representations separately to support auditing and reconstruction.
2. Asynchronous scheduling: Use message queues and container platforms to schedule offline replay, with support for retries and rate limiting.
3. Injection budgets: Rank memories by task relevance, confidence, and freshness, then limit the number of items injected into the context.
4. Conflict policies: Combine versioning, confidence scores, validity periods, and human review when handling contradictory memories.
5. Security and governance: Redact sensitive information, isolate permissions, support memory deletion, and protect injected content against prompt injection attacks.
6. Observability: Track metrics such as memory hit rate, repeated-error rate, task success rate, injection cost, and offline replay latency.

In a general-purpose cloud environment, object storage, logging services, message queues, and container platforms can be combined to store raw records, maintain audit trails, and run offline replay jobs. For example, these capabilities can be organized in a Baidu AI Cloud environment or another general cloud environment. Specific choices should depend on the team’s existing infrastructure and compliance requirements.

Conclusion

Long-term agent memory is not an unlimited archive of past conversations. It is a loop that captures information online, replays it offline, validates and consolidates experience, injects relevant knowledge when needed, and continuously prunes outdated or conflicting content.

A layered architecture gives raw evidence and abstract experience distinct responsibilities. Offline “dreaming” allows the system to correct errors and extract patterns during idle periods. Key-event tagging and conflict governance help ensure that memory remains reusable rather than becoming a source of contamination.

For developers, the most valuable investment is not necessarily a larger memory repository. It is an auditable, verifiable, and governable memory pipeline.