Dreaming Memory for LLM Agents: A Cross-Session Long-Term Memory Design Inspired by the Human Brain

Why Agents Need Cross-Session Memory

Agent performance has improved considerably in recent years, but agents still have almost no real long-term memory. When a session ends, its interaction history is either discarded or compressed into a summary. The agent cannot accumulate experience the way people do and reuse it in future tasks.

Recently, a memory system design inspired by the human brain has been publicly described. Between interactions, the model uses idle compute to “dream” in the background. It replays and reorganizes its history to fix mistakes and distill experience, which enables memory that persists across sessions. This article provides a technical explanation and extension of that architecture.

In a typical agent application, the context window serves as working memory. It keeps a single conversation coherent, but it has three clear limitations:

1. Limited capacity. Even a large context window cannot hold the volume of interaction logs that a long-running agent produces.
2. Short lifespan. Context is destroyed when the session ends, so the agent cannot remember lessons from previous tasks.
3. High cost. Placing all past records into every request slows inference and increases call costs.

In this article, agent memory refers specifically to experience that a model gains from tasks and user interactions, that persists across sessions, and that is read in later sessions as plain-text context. It excludes three concepts that are often confused with it:

• Changes to model weights. This belongs to training or fine-tuning.
• In-context learning within a single conversation. This lasts only for the current session.
• Compressing old conversations into a summary that replaces the context. This is still temporary context management, not retrievable knowledge that carries over between sessions.

Modeling the Human Brain: From Hippocampal Buffering to Cortical Consolidation

Cognitive science has produced a well-known two-stage model of human memory. While awake, the hippocampus quickly captures current experiences and forms short-term memories that are immediately usable but limited in capacity. When the brain goes offline during sleep or rest, the hippocampus replays recent experiences. It extracts the important content and integrates it into the cerebral cortex, where it becomes stable long-term knowledge. In addition, emotional intensity or chemical signals can tag certain experiences at key moments. Tagged experiences receive more replay during sleep and are consolidated preferentially.

This division of labor maps naturally onto agent systems:

• Buffer layer: During daily work, an agent produces a large volume of raw interaction logs. Like hippocampal short-term memory, these logs are complete but redundant and noisy.
• Consolidation layer: Memory notes and memory representations correspond to long-term knowledge in the cortex. They are filtered, distilled, and structured.
• Tagging mechanism: Important interaction events receive higher consolidation priority. Examples include critical turning points in failed tasks, explicit user preferences, and operation paths that have been repeatedly verified as effective.

The “Dreaming” Mechanism: Offline Processing During Sleep-Time Compute

Earlier research introduced the concept of “sleep-time compute.” Under this idea, a model should not only work when it is called. It should also use the time between interactions to actively organize and restructure existing context. The memory architecture described here puts this idea into practice as a dreaming mechanism. The workflow has five steps:

1. Trigger: When the agent is idle, with no pending requests or during low-load periods, a background task starts.
2. Replay: The model rereads recent raw interaction logs and reviews them segment by segment, similar to hippocampal replay during sleep.
3. Correction: During replay, the model identifies errors in earlier reasoning or execution, such as wrong tool calls or plans that the user rejected. It reevaluates these from a hindsight perspective and records the correct conclusions.
4. Distillation: The model writes valuable observations into the memory note layer. Examples include user preferences, task templates, environmental constraints, and common pitfalls.
5. Reorganization: The model periodically merges, deduplicates, and abstracts scattered memory notes into higher-level memory representations, producing a more concise knowledge structure.

The following simplified pseudocode illustrates the process:

def dream(memory_store):
    logs = memory_store.fetch_recent_logs()
    for log in logs:
        issues = self_review(log)          # Retrospective review to find errors
        insights = distill(log, issues)    # Distill reusable experience
        memory_store.write_notes(insights) # Write to the memory note layer
    memory_store.consolidate()             # Merge and deduplicate; build high-level representations

Dreaming uses compute that would otherwise sit idle. This is a form of off-peak computing that does not compete with online inference for resources, which makes it an attractive design from a cost perspective.

Layered Memory Architecture: From Raw Records to Context Injection

A complete memory system typically has four layers, and the dreaming mechanism runs through all of them:

1. Raw record layer: This layer stores complete interaction logs. It is the factual foundation of memory and is kept unprocessed.
2. Memory note layer: The dreaming process writes key observations directly from raw records into this layer. It is the first round of distillation. Notes are small in granularity and more time-sensitive.
3. Memory representation layer: This layer further abstracts and merges notes into stable, structured long-term knowledge. Examples include user profile summaries and libraries of task execution patterns.
4. Context injection layer: When a new session starts or during task execution, the system retrieves relevant content from the upper memory layers based on the current task and injects it into the context window for the model to use directly.

The core value of this layered design is that each layer reduces noise and improves retrieval efficiency. Context injection does not need to scan raw logs. It only needs to match distilled memory representations. When higher-level memory conflicts or is missing, the system can trace back to raw records for verification, which keeps the memory auditable.

Engineering Considerations

Putting this memory architecture into practice requires handling several engineering problems:

• Write policy: Not every interaction is worth remembering. A salience assessment can help. Following the emotional tagging idea, the system can give high priority to events such as failed retries, user corrections, and explicit preferences.
• Forgetting and decay: More memory is not always better. Notes that are not retrieved for a long time should gradually lose weight or be archived, so the memory store does not grow until retrieval quality declines.
• Consistency: When new experience conflicts with existing memory representations, an arbitration mechanism is needed. Typically, newly verified conclusions should override older ones, and the conflict record should be kept for traceability.
• Privacy and compliance: Cross-session memory involves persistent storage of user data. Users must be able to view, delete, and control their memory, and the system should follow the principle of storing the minimum necessary data.
• Retrieval quality: The effectiveness of memory injection depends heavily on retrieval accuracy. Vector retrieval and keyword matching can be combined, and task context can be used to rerank results by relevance.

Conclusion

The value of the dreaming memory mechanism is that it moves memory consolidation from a synchronous, expensive online process to an asynchronous offline process that uses idle compute. Modeled on the brain’s consolidation process, it provides a complete layered system that runs from raw records to context injection. For teams building agent platforms, this approach points to an important direction: improving agent capability depends not only on stronger models but also on better memory management. As sleep-time compute strategies mature, agents may become more capable while “asleep,” and wake up ready to handle the next task with richer experience.