AI Agent Cross-Session Memory: Capture While Working and Dream-Based Consolidation

1. The Core Problem: Writing-Time Importance Is a Decision Made with Incomplete Information

The intuitive approach to giving an AI agent long-term memory is straightforward: during a task, write seemingly important conclusions into a note file, and read that file at the start of the next session. This works reasonably well in demos, but in production it usually degrades within one to two weeks. The problem is not the storage medium, nor the precision of vector retrieval, but the fact that the moment of writing occurs exactly when the available information is least complete.

During a session, the agent only has a local view. It does not know what the user will ask next, which intermediate step will become a critical prerequisite, or which conclusion is a temporary compromise versus a stable fact. In other words, it must estimate the long-term value of an item while lacking information about its future utility. This is a decision where expected benefit cannot be observed, and the result is that errors occur in both directions: important clues are missed, and irrelevant noise is carefully recorded.

The second structural problem is write-as-finalized. Most implementations decide item granularity, field structure, and classification at write time, but these aspects are often still unstable while the session is ongoing. By the time the structure becomes clear, earlier entries have already become misplaced. The usual remedy is to keep appending patches, which eventually turns the memory into a “historical sediment layer” that no one dares delete and no one fully trusts.

The conclusion is clear: high-cost judgment should be moved to a stage where information is complete and where it does not consume the user’s waiting time. This is the core motivation behind the design of capture-while-working plus offline consolidation.

2. Three-Layer Memory Model: Separate Working, Episodic, and Semantic Memory

Cognitive science’s layered view of memory has direct engineering value. It is useful to explicitly separate three layers and allow data to flow between them in one direction:

• Working memory: everything inside the current context window. It is capacity-limited, bound to the session lifecycle, and discarded when the session ends. Do not try to push an entire conversation into long-term storage and expect retrieval to compensate for that decision later.
• Episodic memory: the raw, post-session trace that preserves replayability. It is the foundation for correction. Without raw records, the offline stage cannot verify whether any note is correct. This layer should be as complete, cheap, and read-only as possible.
• Semantic memory: stable conclusions distilled from episodic memory, such as user preferences, project constraints, domain facts, and validated operational procedures. This is the main entry point for retrieval and also the layer that must be repeatedly revised and forgotten.

The key point is this: the raw layer should not be revised; it should only be appended to and annotated. The semantic layer may be revised, but it must preserve version history. The first ensures traceability, while the second ensures correctability. Mixing both into one table is one of the most common architectural mistakes in this kind of system.

As a result, the online-stage task can be reduced to the minimum: write fragments worth preserving, together with their context, into the episodic layer, and perform a lightweight initial annotation such as type, confidence, and involved entities. Do not deduplicate, abstract, or perform final classification at this stage. This step must remain low-latency and low-cost, because most sessions do not actually produce valuable long-term memory.

3. The “Dreaming” Pipeline: Replay, Reconcile, Consolidate, Forget

The background consolidation process triggered after a session can be divided into four stages. It works because it has two advantages that the online stage does not have: it can see the complete session, and even a cross-session view, and it is not on the user’s critical path, so it can freely use longer contexts, slower reasoning, and batch processing.

1) Replay

Select the episodic records from the current session and actively recall relevant existing memories. This step is often overlooked. If you only look at the current session, you cannot detect conflicts between new information and old conclusions. The goal of replay is to bring “newly written content” and “old content that may be overturned” into the same view.

2) Reconcile

Compare each new claim against existing semantic entries and determine one of four relationships:

• Duplicate: semantically equivalent; merge directly and preserve both source references.
• Conflict: later information is more reliable; revise the old entry rather than appending a new one.
• Stale: the conclusion still holds, but the described environment has changed; update the applicable scope.
• Unrelated: do not enter the semantic layer; keep it in the episodic layer.

The reliability of correction comes from provenance. Each semantic entry must record which episodic fragments it came from, including the session identifier and fragment index. When an entry is suspicious, read back the original fragments before making a judgment. This constraint effectively suppresses the risk that “the offline stage itself is hallucinating.” The correction process itself must also be verifiable.

3) Consolidate

Cluster scattered entries by topic and rewrite them into a more regular, more retrieval-friendly form: merge similar facts, eliminate duplicate wording, turn one-off operations into reusable procedures, and aggregate scattered user preferences into stable profile fields. What is produced here is a new version of an entry. The old version is marked as superseded and retained rather than overwritten or deleted.

4) Forget

This is the other half of long-term usability. It requires explicit mechanisms, not vague “periodic cleanup”:

• Time decay: downrank items by last access time; items that have not been retrieved for a long time should be moved to a lower-priority partition.
• Hard expiration: entries such as session credentials, temporary paths, and one-off computation results should have short time-to-live values.
• Explicit deletion: when the user requests deletion, remove the data from both the semantic and raw layers and write a tombstone record to prevent later offline tasks from “resurrecting” it.
• Quota: each namespace should retain a fixed budget. When the limit is exceeded, trigger lossy compression such as merging, lowering precision, or keeping only a summary.

Example Data Structure

# Semantic-layer memory item: revisable, source-backed, and versioned
@dataclass
class MemoryItem:
    id: str                       # stable identifier; unchanged across revisions
    namespace: str                # user/tenant/project isolation boundary
    kind: str                     # preference | fact | constraint | procedure
    statement: str                # normalized one-sentence statement
    scope: dict                   # applicability conditions, e.g. {"project": "...", "env": "prod"}
    confidence: float             # derived from source count and consistency
    sources: list[str]            # episode fragment IDs; must be reread during correction
    revision: int
    superseded_by: str | None     # replacement item; old version is preserved
    created_at: str
    accessed_at: str
    ttl_days: int | None
    tags: list[str]

Offline Consolidation Task Skeleton

def dream(session_id: str) -> None:
    episodes = load_episodes(session_id)              # 1. replay
    candidates = extract_claims(episodes)             # lightweight annotated candidate claims
    neighbors = recall_existing(episodes, top_k=20)   # recall old items that may be affected
    for claim in candidates:
        relation, target = classify_relation(claim, neighbors)
        if relation == "duplicate":
            merge_into(target, claim)
        elif relation == "conflict":
            if verify_against_sources(claim) and verify_against_sources(target):
                revise(target, claim)                 # revise instead of appending
            else:
                quarantine(claim)                     # suspend if evidence is insufficient
        elif relation == "stale":
            update_scope(target, claim)
        else:
            create_or_append_topic(claim)
    consolidate_topics(session_id)                    # 3. clustering and rewriting
    apply_decay_and_ttl(namespace_of(session_id))     # 4. forgetting
    mark_done(session_id)                             # idempotency marker; supports resume after interruption

Note the quarantine branch: when evidence is insufficient, suspend the claim instead of forcing a binary decision. This removes the online-stage pressure to “reach a conclusion immediately” and leaves room for later sessions to provide more signal.

4. Retrieval Must Be Designed Alongside Writing; Otherwise, Writing Is Wasted

The value of consolidation is only realized at read time. The retrieval path should at least do the following:

• Hybrid retrieval: perform structured filtering first, such as by namespace, type, applicable scope, and freshness, then apply a mix of keyword and vector ranking over the candidate set. Pure vector retrieval almost inevitably recalls wrong entries when constraints are present.
• Progressive disclosure: first provide low-level topic summaries and an index so the model can decide which part to expand, then load the full text of the specific entries. Stuffing all memory into the context at once consumes working space and significantly reduces instruction-following quality.
• Provenance-rich output: provide the matched entries and their source fragments to the model. When conflicts exist, explicitly mark that “two mutually exclusive conclusions exist,” rather than letting the model guess.

You should also impose a budget on reading. Set an upper limit on the amount of memory that can be injected into each session. If the limit is exceeded, truncate entries according to confidence × freshness decay × access frequency, and record which items were truncated for later evaluation.

5. Engineering Considerations

• Idempotence and reentrancy: consolidation tasks should be organized by session and advanced through checkpoints. When a task is executed repeatedly, use stable item identifiers to ensure idempotent writes and avoid merging the same conclusion into duplicate entries.
• Cost and throughput: offline tasks naturally fit batch processing and graceful degradation. A hybrid trigger strategy is recommended: run periodically during idle periods, trigger when accumulated entries or conflict signals reach a threshold, and trigger when the user explicitly requests consolidation. Set upper limits on tokens and duration for each task.
• Privacy and compliance: memory is a highly sensitive data asset. The raw layer needs redaction and field-level encryption. The semantic layer should support per-user export and complete deletion, including offline copies. In cross-border scenarios, isolate storage and computation by region, and retain audit logs for all read, write, and revision operations.
• Multi-tenant isolation: the namespace is a mandatory boundary. Retrieval, consolidation, and deletion paths must all verify ownership. Cross-namespace recall and merging are prohibited.
• Observability: at minimum, collect consolidation success rate, number of corrected entries and the proportion of corrected entries, conflict quarantine rate, retrieval hit rate, cost per consolidation run, and the memory-size growth curve. A correction ratio that remains zero for a long time usually means the correction stage is not actually working, rather than that there are no errors.
• Cloud component mapping: this architecture can be mapped directly to general-purpose cloud capabilities. Object storage can hold raw session records, a message queue can drive asynchronous consolidation tasks, a container platform can run batch processing and tuned model inference, vector and full-text search services can support hybrid recall, and logging and monitoring services can cover audit and metrics. This general combination can be organized without introducing proprietary coupling.

6. How to Evaluate: Four Categories of Metrics

• Factual accuracy: use the original session as the baseline and verify, manually or automatically, whether semantic entries are consistent with their sources. Pay special attention to the error-correction rate for revised entries.
• Conflict-resolution quality: construct sequences of test cases in which old and new information are mutually exclusive, and check whether the final correct conclusion is retained and whether traceable history is preserved.
• Retrieval effectiveness: measure recall and precision on cross-session tasks, as well as the noise-injection rate of whether irrelevant memory is injected into the context.
• Forgetting correctness: check whether expired information stops participating in retrieval, and whether deleted information still fails to reappear after several consolidation cycles.

It is recommended to maintain an offline evaluation set of continuous multi-session tasks. Run the same batch of tasks under three configurations: “no memory,” “online writes only,” and “online writes plus offline consolidation.” Only then can you see the true gain of the consolidation stage.

7. Common Anti-Patterns

• Full-volume writing: storing the entire conversation directly into the long-term layer causes cost to grow linearly with the number of sessions while usability continues to decline.
• Append-only without revision: conflicting entries coexist, retrieval randomly hits one of them, and the output becomes self-contradictory.
• Making final judgments online: letting the model in an active session decide entry granularity and classification leads to premature structural decay.
• Entries without provenance: if a memory item is wrong, you cannot locate its source or safely delete it, so the only option is to clear everything.
• No forgetting mechanism: memory size grows without bound, eventually overwhelming the context budget and allowing stale conclusions to continuously pollute the output.

8. Summary

The engineering value of cross-session memory comes from scheduling work at the most appropriate time: the online stage pursues low-cost, complete capture, while the offline stage handles high-cost judgment that requires a full view. After a session ends, reviewing raw records, correcting mistaken notes, and restructuring scattered information into retrievable structures is equivalent to adding a “consolidation” stage to the agent. This allows temporary experience to have a chance to settle into stable knowledge, and allows erroneous conclusions to be revised before they are widely used.

When implementing this, focus on four things: separate the raw and semantic layers while preserving provenance, prefer revision over append-only writing, design retrieval with progressive disclosure and budget control, and implement explicit forgetting and deletion mechanisms. With these in place, the design can move from a demo to sustainable operation.