Layered Memory and Offline Dreaming: Engineering Cross-Session Learning for AI Agents

1. The Starting Point: Memory Debt in Long-Horizon Tasks

AI agents often exhibit a common degradation pattern when executing long-horizon tasks across multiple sessions: more sessions do not necessarily lead to better performance, and in some cases, the agent becomes less effective over time.

The root cause is usually not a decline in model capability. It is the accumulation of three types of memory debt.

• The context window is a hard constraint. Putting the entire history into the context window quickly hits the limit. Even when it fits, irrelevant details dilute attention, and the model may latch onto the most recent sentence while ignoring conclusions that were already validated earlier.
• Errors lack a retraction mechanism. A wrong coordinate, an invalid table name, or an expired path that is written into a log unchanged will be referenced again in later turns. Errors compound over time.
• Logs are not knowledge. Raw interaction logs are readable but often unusable. They record what happened, but they do not answer how the information should be used later.

A useful analogy comes from biological systems. Humans do not retain all raw sensory input indefinitely. During sleep, short-term memories are replayed, filtered, merged, and written into long-term storage. AI agents can perform a similar process during idle periods between interactions. This capability depends less on stronger reasoning and more on an explicit offline consolidation pipeline.

---

2. Splitting Memory into Four Layers: Responsibility Boundaries Matter More Than Storage Medium

Many failed attempts treat memory as a single component: either everything is stuffed into the prompt, or everything is sent to retrieval. A more maintainable approach is to divide memory by lifecycle, with each layer responsible for one thing.

Layer	Content Form	Typical Lifecycle	Write Mode	Main Purpose
L1 Session Log	Event-level and tool-call-level records	Session-level; only summaries retained long term	Automatically appended at runtime	Auditing, replay, and offline source material
L2 Memory Notes	Partial conclusions and unverified hypotheses	Days to weeks	Actively marked and written by the model	Carrying forward unfinished work
L3 Knowledge Representation	Rules, policies, plans, and structured facts	Months or longer	Distilled and solidified during the dreaming stage	Core assets for cross-session reuse
L4 Injected Context	A compact set assembled for a specific task	Single interaction	Retrieval plus budget-based pruning	Helping the model recall what it needs

Several design points are easy to overlook.

1. L1 provides evidence, not decisions. The value of session logs is reproducibility. When a rule in L3 is disputed, it should be possible to trace it back to the logs that support it. Therefore, logs should preserve source identifiers and timestamps, but they should be summarized and compressed before archiving. The goal is to reduce “every observed frame” into “key decision points.”

2. L2 is a staging area, not a knowledge base. It may contain unverified or low-confidence entries, but they must be explicitly marked as hypothesis and must not be directly injected into L4. Otherwise, the staging area becomes a source of contamination.

3. L3 is the layer that requires careful governance. It should have a schema, versions, ownership, and confidence. It should be able to answer: who established this knowledge, and under what evidence?

4. L4 is a budget-constrained assembly layer. More injected memory is not always better. Under a limited token budget, the system should select memories based on task relevance, confidence, and reuse frequency, rather than simply stacking recent memories in reverse chronological order.

A canonical memory record can be defined as follows:

{
  "memory_id": "rule-order-quarter-normalize-001",
  "layer": "L3",
  "type": "strategy | rule | plan | fact | episode",
  "content": "Before aggregating the sales database by quarter, order timestamps must first be normalized to the baseline time zone; otherwise cross-region orders will be assigned to the wrong quarter.",
  "provenance": {
    "source_sessions": ["s-9f21", "s-a3c7"],
    "evidence": ["retry_3", "validation_pass_12"],
    "created_at": "2026-03-02T08:15:00Z"
  },
  "confidence": 0.86,
  "stability": "high",
  "reuse_hint": ["nl2sql", "reporting"],
  "hit_count": 7,
  "last_hit_at": "2026-04-11T02:20:00Z",
  "ttl_days": 180,
  "supersedes": null
}

The stability field determines whether a record can enter L3. Volatile state should never receive a long-term identity.

---

3. The Dreaming Mechanism: A Four-Step Loop During Idle Time

“Dreaming” does not mean letting the model generate freely. It means using idle compute for a deterministic pipeline.

def dream_cycle(agent, idle_budget_s: int = 60):
    """Run memory consolidation during idle windows between interactions."""
    for log in agent.store.unconsolidated_logs(budget_s=idle_budget_s):
        # 1) Replay: split the event stream into meaningful segments and extract candidate conclusions
        episodes = agent.parse_events(log)
        candidates = [distill(e) for e in episodes]
        # 2) Verify: verifiable candidates must be rechecked against real state; recollection does not automatically become fact
        candidates = [c for c in candidates
                      if c.stability == "volatile" or agent.sandbox.verify(c)]
        # 3) Distill and reconcile: align with existing knowledge, resolve conflicts, and generate supersede relationships
        merged = reconcile(candidates, agent.store.load(candidates[0].namespace)
                           if candidates else [])
        # 4) Consolidate and forget: write, compress, and evict low-value entries according to budget
        for item in merged:
            if item.action == "update":
                agent.store.supersede(item.supersedes, item)
            elif item.action == "insert":
                agent.store.insert(item)
        agent.store.prune(keep_top_k=agent.policy.top_k,
                          min_confidence=agent.policy.min_conf)
        agent.store.compress_logs(log)   # Reduce dimensionality before L1 archiving

Each step solves a specific problem.

Replay solves the question of where the material comes from. In terms of strategy, priority sampling should be used. Segments containing failures, retries, human corrections, and task termination should be replayed first because they carry the highest information value per unit.

Verification solves the question of how errors can be retracted. This is the most practical capability of the dreaming mechanism. Typical examples are externally verifiable facts such as coordinates, paths, and identifiers. During idle periods, the model can re-read the real state, such as the file system, API responses, or configuration center. If the recorded value differs from the current value, the model should rewrite it and preserve the correction trace. Subjective judgments and preferences cannot be re-verified in the same way, so they must rely on confidence decay.

Distillation solves the question of how to move from process to regularity. A good memory is not “this time I used parameter X,” but “under this type of constraint, parameter X should be used because of Y.” The latter is transferable knowledge.

Consolidation and pruning solve the question of where capacity comes from. Memory must have a budget limit. L3 is often trimmed using three constraints: namespace, top-k, and minimum confidence. Entries that have not been hit for a long time, meaning those with stale last_hit_at and low hit_count, are downgraded or deleted.

One practical engineering principle is: the dreaming stage is a write-heavy batch process, while the interaction stage is a read-heavy hot path. If these two are mixed, online latency and offline governance will drag each other down.

---

4. Write Criteria: Memory Is Essentially a Prediction About the Future

A useful criterion from memory research is that writing a memory is equivalent to the model placing a bet that it will rely on that information again in the future. Based on this idea, an explicit value score can be designed instead of relying on the model’s feeling that something is important.

score = P(future reuse) × benefit (saved inference/execution cost) × stability × verifiability − maintenance cost

This leads to three executable guidelines.

1. Prefer storing strategies, plans, and tested rules. This type of content does not drift over time, has a high reuse rate, and can be re-verified. It is a high-stability asset.

2. Refuse to store volatile transient state. The current frame image, the current cursor position, a temporary process ID, or a one-second API response snapshot are all volatile. They may look concrete and therefore important, but their lifecycle is extremely short. Writing them creates noise.

3. Few and precise is better than many and comprehensive. An observation from several handheld game benchmarks is instructive: an implementation that maintained only about 10 memory files achieved 4 achievements, while an implementation that wrote 413 files obtained only 1. The latter recorded a large number of frame states and temporary coordinates, while the real strategies and plans were drowned out. This shows that the signal-to-noise ratio of memory retrieval directly determines whether memory is effective.

A related rule is the evidence threshold. A candidate memory needs at least two types of support, such as two independent successes or one success plus one failure contrast, before it can be promoted to an L3 rule. Observations with only a single accidental success should remain in L2.

---

5. Conflict Resolution and Consistency Governance

The most common way cross-session memory becomes corrupted is the accumulation of contradictions. If three mutually exclusive conclusions exist under the same namespace, retrieval may randomly hit any one of them, causing the model’s behavior to jitter. Governance requires four mechanisms.

• Key-based uniqueness. Design a stable primary key for rule-type memory, such as domain plus object plus constraint. Updates should use supersede rather than append, ensuring that a fact has only one currently valid version.
• Temporal validity and versioning. For knowledge with a TTL, specify when it begins to expire. Facts such as table schemas, API signatures, and dependency versions should be bound to observable environment versions.
• Conflict detection. Before writing, perform nearest-neighbor comparison using semantic retrieval plus field-level diff. If the same primary key has different values, trigger re-verification rather than direct overwrite. If re-verification still cannot resolve the conflict, downgrade it to L2 and mark it for human confirmation.
• Source credibility weighting. Memories from human corrections should be weighted higher than model self-evaluation. Memories from re-verifiable tools should be weighted higher than content from a single generation.

It is also worth emphasizing the positive value of forgetting. In a memory system with fixed capacity, the benefit of deleting an incorrect rule is often greater than the benefit of adding a correct rule, because deleting the incorrect rule removes an ongoing systematic bias.

---

6. Evaluation: How to Prove That Memory Is Actually Useful

Memory capability must be measured with cross-session metrics. Single-session metrics cannot reveal the difference. There are three feasible evaluation designs.

• Baseline comparison. Run the same continuous-learning benchmark with the same model under two configurations: with memory and without memory. In cross-session database question-answering tasks, implementations with a memory mechanism can achieve an average score of around 0.83, while the no-memory baseline is around 0.65. The gap mainly comes from not having to repeat mistakes that were already encountered in previous sessions.
• Capacity-benefit curve. The horizontal axis represents the number of memory entries, and the vertical axis represents task success rate. A healthy curve usually rises quickly at first and then reaches a plateau. If the curve continues to decline, it means that noise injection has exceeded knowledge gain.
• Correction rate and regression rate. Track the number of facts actively corrected during the dreaming stage and whether those corrections introduce new errors. This is a core metric for judging whether the offline pipeline is trustworthy.

For the evaluation environment, it is recommended to use a containerized platform as an isolated sandbox. The dreaming stage needs to validate candidate rules on a controlled copy to avoid contaminating the main environment. Logs and execution traces should be delivered to a log service and object storage for offline replay and metric aggregation. Monitoring and alerting can be used to observe long-term trends such as memory store size, injection hit rate, and the number of conflicting entries.

---

7. Deployment Architecture and Engineering Trade-offs

A minimal viable memory subsystem can usually be assembled from six general-purpose capabilities, without requiring proprietary components.

1. Object storage. Store L1 raw logs and L3 knowledge snapshots, supporting versioning and prefix-based archiving.
2. Message queue. Decouple events such as session end, idle trigger, and threshold trigger into asynchronous tasks, smooth traffic peaks, and ensure that dreaming jobs can be retried.
3. Container scheduling. Launch offline consolidation jobs according to budget, limiting CPU, memory, and concurrency to avoid preempting online service resources.
4. Logging and metrics service. Record every write, hit, downgrade, and deletion of memory to form an auditable chain.
5. Retrieval service. Provide semantic nearest-neighbor recall and structured filtering by namespace, type, stability, and confidence. This is the foundation of L4 assembly.
6. Access control and key management. In a multi-tenant environment, memory must be isolated by subject, and cross-tenant recall is prohibited. Writing processes must desensitize personal information.

For trigger timing, a “three-choice OR” strategy is recommended: normal session end, idle duration exceeding a threshold, or pending material volume exceeding a threshold. The first two ensure timeliness, while the third prevents material from piling up.

Several common pitfalls should be avoided:

• No context budget pruning. Directly concatenating all recalled results turns retrieval noise into attention noise.
• Synchronous distillation on the online path. This causes tail latency to spike. Distillation should be strictly offline.
• No human takeover channel. Low-confidence conflicting entries must be able to enter a human confirmation queue. Otherwise, errors can only be corrected by the model itself, which may become self-consistent but incorrect.
• Only adding, never subtracting. The prune step should be included in release checks from the beginning.

---

8. Memory, Retrieval-Augmented Generation, and Parameter Training

These three approaches are often confused, but their boundaries are clear.

• Retrieval-augmented generation (RAG) is aimed at world knowledge. External documents are relatively stable, and updates are the responsibility of the data source. The agent does not need to write this information.
• The memory system is aimed at experiential knowledge. It comes from the agent’s own interaction process, including pitfalls encountered, corrections received, and validated strategies. It must support writing and forgetting.
• Parameter fine-tuning is aimed at capability consolidation. Only when a type of experience is highly stable, abundant, and broadly applicable across tasks is it worth solidifying into parameters.

For most teams, the optimal path is to first make the memory system solid. It is fast to validate, easy to roll back, and auditable. Moving too early toward parameter-level consolidation can multiply the cost of error correction.

---

9. Conclusion

Letting an agent “dream” essentially means separating two things that cannot both be handled well during online reasoning: the hot path is responsible for responding, while the cold path is responsible for learning and self-correction. In practice, four key judgments matter.

1. Layering is the prerequisite. Logs, notes, knowledge, and context each have their own role, and the staging area must not be directly injected.
2. Verifiability comes first. Only facts that can be re-verified by the environment deserve to enter long-term memory. Subjective speculation should remain in a low-confidence area.
3. Few and precise. Memory capacity is a scarce resource. Strategies and rules are more valuable than transient state, and deleting an error is better than adding another correct item.
4. Offline loop. Replay, verification, distillation, and consolidation must be completed as batch processes during idle time and continuously evaluated.

Once this mechanism works, the performance of an agent in long-horizon tasks no longer depends only on the amount of information in a single conversation. It depends on whether the agent has truly learned from its past failures.