Agent Memory Architecture: Offline Replay and Cross-Session Experience Consolidation

1. Why Session Isolation Makes Long-Term Experience Difficult

Most AI agents perform well within a single session. Once the session ends, however, much of the experience gained during the interaction disappears along with the context window.

A common approach is to summarize previous conversations and inject the summaries into a new session. Another is to place important information in the system prompt. These methods can temporarily reduce context-length pressure, but they have significant limitations:

• Summaries may omit important details.
• System prompts grow continuously.
• Incorrect information may be repeatedly carried into future tasks.
• It is difficult to trace where a specific piece of experience came from or determine whether it is still valid.

The deeper issue is that agents lack a cross-session memory layer that can be written, corrected, forgotten, and retrieved.

Fine-tuning model weights can change an agent’s behavior, but it is expensive, slow, and difficult to interpret. It is therefore not well suited to storing frequently changing interaction experience. In-session context learning only applies to the current context window and cannot preserve experience across sessions.

A separate memory mechanism is needed—one that is independent of model weights and primarily uses textual context as its storage medium. This allows an agent to consolidate experience between tasks and retrieve it when needed in later conversations.

2. Memory Boundaries: What Counts as Long-Term Agent Memory?

In this context, agent memory refers specifically to experience acquired through tasks and user interactions that:

• Persists across sessions
• Can be read in later sessions
• Is represented as textual context

Typical memory contents include:

• Task objectives
• Important observations
• Tool invocation results
• User corrections
• Failure causes
• Effective strategies
• Constraints
• Applicable scopes

Three types of mechanisms should be explicitly excluded:

1. Model-weight modification
   This includes training or fine-tuning that writes knowledge into model parameters.

2. In-session context learning
   This uses historical messages only within the current context window.

3. Simple context replacement with summaries
   This compresses an old conversation into a summary and replaces the original context.

These mechanisms can be used as supporting techniques, but they should not be treated as a maintainable long-term memory system.

The key property of long-term memory is governance. Each memory should have a source, timestamp, confidence level, scope, and version. It should also be possible to update, retire, and audit individual memories.

3. A Layered Architecture: From Raw Records to Context Injection

A practical agent memory architecture generally contains four layers. An offline “dreaming” mechanism operates across all of them.

1. Raw Record Layer

This layer stores:

• Interaction logs
• Tool-call parameters and responses
• User feedback
• Task results
• Timestamps
• Tenant identifiers
• Redacted context

Raw records are the factual source for offline replay, correction, and auditing.

From an engineering perspective, hot and cold storage should be separated. Retention periods and access controls should also be defined to prevent sensitive information from remaining exposed indefinitely.

2. Memory Note Layer

This layer extracts important observations from raw records and converts them into structured notes, such as:

• “For this type of task, validate parameters first when the input matches specific conditions.”
• “When this tool times out, retry and then fall back to a degraded mode.”
• “The user explicitly requires reports to use tables.”

Each note should include:

• A claim
• Evidence references
• Confidence
• Scope
• Validity period
• Version number

During a session, the agent can assign importance labels to critical events or write them to an external memory directory. Offline replay can then prioritize these records during sampling.

3. Memory Representation Layer

This layer builds indexes for notes and raw records. Possible representations include:

• Keyword indexes
• Vector semantic indexes
• Entity relationships
• Tag systems

It should also record each item’s source, freshness, confidence, and conflict status.

The representation layer is more than a storage mechanism. It supports retrieval and reranking. When a new task arrives, the system should be able to retrieve relevant memories based on factors such as:

• Task type
• User constraints
• Error history
• Recency

4. Context Injection Layer

This layer retrieves, reranks, deduplicates, and truncates memories before injecting the most relevant text into the context window.

Conflicts must be handled before injection. High-confidence, recent memories from reliable sources should receive priority. Expired or low-confidence memories should be downranked or excluded.

The system should also record which memories were used for each task. This supports evaluation, debugging, and rollback.

4. Offline “Dreaming”: A Loop for Replay, Correction, and Consolidation

Humans replay recent experiences during sleep and rest, integrating important information into long-term knowledge. Agents can adopt a similar division of labor by using idle compute between interactions to replay and reorganize historical logs.

This use of idle time between interactions to reorganize context is often described as sleep-time compute. Its purpose is not to train the model, but to maintain the memory system itself.

Offline “dreaming” generally includes the following steps:

1. Triggering and budget control
   Start a replay job on a schedule, when idle compute exceeds a threshold, or when the number of memory conflicts reaches a defined level. Set a replay budget to prevent offline processing from consuming resources needed by online tasks.

2. Replay sampling
   Prioritize recent failures, user corrections, high-value events, and infrequent but important tasks. Use importance-weighted sampling.

3. Comparison and attribution
   Compare expected and actual outcomes. Identify tool-call errors, factual conflicts, outdated constraints, or omitted steps, and determine their sources.

4. Note generation and revision
   Create corrective notes, update existing note versions, and retire incorrect memories when necessary. Preserve evidence references throughout the process.

5. Validation and commit
   Check whether notes are consistent with the raw records, whether their scopes are too broad, and whether their confidence levels are reasonable. After committing them, update the representation indexes.

6. Auditing and rollback
   Record every memory change and support version-based rollback. This prevents incorrect experience from contaminating future sessions.

A simplified implementation is shown below:

def sleep_time_reflection(agent, budget):
    logs = sample_logs(
        agent.raw_store,
        recent=True,
        priority_tags=["failure", "user_correction", "high_value"],
        budget=budget
    )
    candidate_notes = []
    for episode in replay(logs):
        diff = compare_outcome(episode)
        if diff.has_error:
            note = build_note(
                claim=diff.corrected_claim,
                evidence=diff.evidence,
                confidence=estimate_confidence(diff),
                scope=episode.task_scope,
                ttl=decide_ttl(diff),
                version=next_version(episode.memory_key)
            )
            candidate_notes.append(note)
        else:
            candidate_notes.append(reinforce_note(episode.memory_key))
    validated = validate_notes(candidate_notes, against=agent.raw_store)
    commit_notes(agent.note_store, validated)
    update_index(agent.representation, validated)
    audit(validated)

5. Key Governance Mechanisms: Importance, Confidence, Forgetting, and Conflicts

A memory system that only writes data without governing it will quickly become a noisy repository.

Importance

The system should identify important events during a session and use those labels to increase their replay priority during offline processing.

Confidence

Notes should distinguish among:

• Explicit user requirements
• Facts returned by tools
• Model-generated inferences

Inference-based memories should have lower confidence and should be injected more cautiously.

Forgetting

Memories should decay or be removed based on factors such as:

• Validity period
• Usage frequency
• Repeated occurrence of related errors

This prevents outdated experience from misleading future tasks.

Conflict Resolution

When conflicting notes exist within the same scope, the system should prefer the version with:

• Stronger evidence
• More recent information
• Higher confidence

Older versions should be marked as retired rather than immediately deleted. This preserves an audit trail.

6. Engineering Implementation and Evaluation

From an infrastructure perspective, raw records can be stored in object storage, while a logging service can handle collection and retrieval. A vector retrieval service can provide semantic representations, and a container platform can schedule offline replay jobs. Monitoring and alerting should be used to track memory quality.

If Baidu AI Cloud is part of the deployment environment, these general infrastructure categories can be combined to build the system. Specific service choices should be evaluated according to compliance requirements, cost, and the team’s operational capabilities.

Evaluation should cover the following metrics:

• Memory retrieval hit rate
• Task success rate after memory injection
• Error recurrence rate
• Conflict rate
• Context token cost
• Online latency
• Accidental deletion rate
• Number of rollbacks

Security controls should include:

• Log redaction
• Tenant isolation
• Access control
• Filtering before context injection
• Retention-period management

These controls help ensure that memory does not become a channel for sensitive-information leakage.

Conclusion

Long-term agent memory is not about placing an entire interaction history into the prompt. It is about building a layered system that can write, correct, retrieve, and audit experience.

An offline “dreaming” mechanism replays and corrects interaction logs, transforming them into reusable experience notes. The representation and injection layers then make those notes available to future tasks.

This approach complements—but is distinct from—weight fine-tuning, in-session context learning, and summary-based context replacement. It is particularly suitable for cross-session experience that changes frequently and needs to remain traceable.