Agent Long-Term Memory Architecture: Offline "Dreaming" to Distill Experience

1. The Starting Point: Why Agents Can’t “Remember”

Large-model-driven conversational agents have an inherent flaw: they are stateless. In every task and every conversation turn, the model only sees what has been assembled into the context at that moment. Once the context window is cleared, all prior exploration paths, pitfalls, and errors the user corrected cease to exist.

Three common early remedies each have obvious shortcomings:

• Full replay: stuffing the raw conversation history into the context. Cost rises linearly or even superlinearly with the number of sessions, quickly hits the context-length limit, and noise drowns out the key information.
• Rolling summarization: compressing old conversations into a summary and replacing the original. Information loss is irreversible, and once a wrong conclusion is compressed into the summary, it becomes an “established fact” for later reasoning, hard to recognize again.
• In-session in-context learning: letting the model adjust its behavior from feedback within the current session only. It works immediately, but everything is lost when the session ends and cannot be deposited as an asset.

What really needs to be solved is cross-session experience distillation: an agent should perform better on the second, twentieth, or two-hundredth run of a similar task than on the first, and this improvement must be persistently stored, retrievable, and correctable. That is exactly the scope long-term memory must cover.

2. A Biologically Inspired Mapping: Decouple “Encoding” and “Consolidation” in Time

The way the human brain handles memory gives this architecture its most direct inspiration. The massive stream of daily experiences is first captured quickly by the hippocampus in the moment, forming immediately usable but volatile temporary memory. When the brain enters offline states such as sleep and rest, the hippocampus repeatedly replays recent experiences, distills and reorganizes the important parts, and integrates them into the cortex, turning them into stable long-term knowledge.

Three key design insights follow, each mappable directly onto an agent system:

1. Decouple writing from organizing. Encoding happens at interaction time and optimizes for low latency; consolidation happens at offline time and optimizes for controlled quality and cost. The two do not have to run on the same path.
2. Offline periods are usable idle resources. Between two interactions, an agent usually has large blocks of compute sitting idle — time that is well suited to organizing work.
3. Replay is a prerequisite for distillation. A single experience is rarely enough to support a reliable conclusion; only by replaying multiple experiences together can you see commonalities and differences, and discover that “doing it that way last time was wrong.”

There are already explorations in the field that formalize this idea, using concepts like “sleep-time compute” to describe a model reorganizing existing context between interactions. Engineering that idea is what is commonly called the offline dreaming mechanism.

3. First, Define It Clearly: What Counts as “Agent Memory”

Conceptual ambiguity is the leading cause of failure for these systems. Here we adopt a strict definition: agent memory is experience acquired during interaction with tasks and users that persists across sessions and is read by later sessions as plain-text context.

By this definition, three other mechanisms are explicitly excluded:

Mechanism	Time scale	Storage medium	Modifies weights	Main failure mode
Model fine-tuning / parameter update	Training time	Model weights	Yes	High training cost, no instant updates, catastrophic forgetting
In-session in-context learning	Single session	Context window	No	Lost when the session ends
Conversation history compression	Single session/task	Summary text	No	Information loss, errors get entrenched
Cross-session agent memory	Cross-session, long-term	External retrievable text	No	Memory pollution, bloat, imprecise retrieval

The engineering significance of this distinction: the memory system is a data subsystem that lives outside the model and is observable, editable, and rollback-capable. It has its own read/write interfaces, storage medium, and lifecycle, so it can be evaluated and governed independently without touching the training pipeline.

4. The Four-Layer Memory Architecture

A practical memory system can be divided into four layers, progressing from “raw factual state” to “injectable state.”

Layer 0 — Raw episodic log (raw episodic log)
Append-only, never rewritten. Each record contains a timestamp, session/task ID, user input, model output, tool-call trace and arguments, execution result, and explicit feedback (user corrections, task success/failure determination). This is the single source of truth for all upper layers and the basis for correction and traceability. The typical carriers are logging services and object storage, partitioned by time series.

Layer 1 — Memory notes (memory notes)
Atomic, reusable conclusions distilled from the raw records. A good note usually has four elements: the applicable condition, the conclusion itself, a source reference (pointing to a Layer 0 record ID), and a confidence level or verification count. Tagging by type helps downstream filtering. Common types include operational rules (“tasks of this kind must validate X first”), factual knowledge (“the true semantics of a certain field in this data source”), lesson-type negative examples (“calling it the Y way fails because of Z”), and user preferences (output format, language, level of detail). This stage can write directly from a session without waiting for the offline period.

Layer 2 — Memory representation (memory representation)
The number of notes will quickly exceed what can be injected in full, so indexing and structuring are required. At minimum it should provide: keyword/full-text search, vector semantic retrieval, filtering by type and entity, and statistical aggregation (a rule successfully cited N times, falsified M times). This layer is the key to whether memory can actually be used.

Layer 3 — Context injection (context injection)
The part that actually enters the model context. Three things must be done: recall by relevance to the current task, trim and rank within the token budget, and resolve conflicts among contradictory memories. Injected content should explicitly label its source and time so the model knows the “shelf life” of a given piece of experience.

5. Key Event Marking: Solving the “What’s Worth Remembering” Prior Problem

The brain has an elegant mechanism: emotional or chemical signals mark experiences for salience at key moments, so those fragments get more replays during the subsequent sleep. This suggests that the core challenge of a memory system is not “how to remember” but “what to remember” — the agent cannot predict in advance which of the events happening now will matter in the future.

In engineering, three kinds of signals are typically mixed:

1. Explicit writing (agent’s own decision). Allow the agent to call a write interface during a session, dropping observations it considers important directly into a memory file, or tagging them. The advantage is flexibility; the risk is that the model over-writes large amounts of low-value notes.
2. Rule-triggered (deterministic events). Errors thrown, tool-call retries, explicit user corrections, tasks judged as failures, the same question asked repeatedly — all of these are high-salience events, and a hit raises the replay priority.
3. Feedback-driven (external validation). Task success criteria, correct answers from a verifier, human sample scoring, whether the user adopted a given suggestion. These signals are the highest quality but also the most delayed.

When fusing the three, set a “salience score” that decides how many times an item gets replayed and its priority during the offline dreaming phase. Raw logs default to a low score, feedback-verified items get a high score, and items repeatedly cited but never verified should get a medium score — they are worth testing.

6. “Memory Is a Prediction About the Future”: How the Learning Loop Closes

One point worth stressing: taking notes is itself a prediction about the future.

• When the agent writes a rule, it is predicting “this pattern will be useful again later”;
• When it records a verified truth, it is predicting “future me won’t need to re-check it when using it”;
• When it saves a failed path, it is predicting “next time conditions look similar, this path should be skipped.”

Once memory is treated as prediction, the structure of the learning loop becomes clear: predict → used in a later session → result observed → confidence or content of the prediction corrected. Closing the loop requires three conditions:

• Observable usage traces: which memory was recalled in which session, and whether it affected the output;
• Decidable result signals: task success/failure, verifier judgment, user adoption or rejection;
• Executable correction actions: raise confidence, downgrade, rewrite, merge, delete.

If any link is missing, the system degrades into a dead repository that “only writes, never reads” or “only reads, never evaluates.” This is also the root cause of many memory systems that appear to be live but are in fact ineffective.

7. How to Actually Implement the Offline “Dreaming” Flow

The dreaming phase is the process of turning Layer 0 low-value raw material into high-quality Layer 1/2 assets. It is recommended to break it into six clear steps:

1. Trigger and scheduling. Use idle compute as the trigger: scheduled windows, after the task queue drains, during system off-peak hours. In implementation, deliver “dream task sets” via a message queue, consumed by an independent offline worker pool to avoid competing with online inference for resources.
2. Candidate sampling. Do not replay everything. Sample with a combined score of “recency + high salience + previously retrieved but poorly used.” This sampling strategy directly determines the return per unit of compute.
3. Replay and recombination. Put multiple trajectories of the same task type together and look at them jointly: extract common steps, divergence points, and stable conclusions. A single trajectory yields only factual notes; multiple trajectories yield rule-type notes.
4. Correction and conflict handling. Detect contradictions between new and existing notes. The handling principle: a higher-confidence, newer source overrides the old conclusion, while keeping the old conclusion and the reason it was overridden, for auditing. If evidence is insufficient, keep both and explicitly mark the conflict, leaving the online phase to prefer the more recent by update time.
5. Abstraction and writing. Output structured notes with source IDs, applicable conditions, and confidence; simultaneously update the index and the statistical aggregation.
6. Maintenance. Deduplicate, merge synonymous entries, compress verbose phrasing, evict by utility score, and periodically rebuild the index.

Illustrative implementation (pseudocode, expressing control flow only):

def run_dream_cycle(mem_store, budget_tokens):
    # 1. Sample: balance recency, salience, and historical hit performance
    candidates = mem_store.sample_episodes(
        strategy="recency_x_salience_x_poor_hit",
        limit=200,
    )
    # 2. Replay: cluster by task type, look at multiple trajectories together
    for cluster in group_by_task_signature(candidates):
        facts, rules, conflicts = reflect(cluster)
        # 3. Correct: let newer, higher-confidence sources override old conclusions
        for rule in rules:
            existing = mem_store.find_conflicting(rule)
            if existing and existing.confidence < rule.confidence:
                mem_store.supersede(existing, rule, keep_reason=True)
            else:
                mem_store.upsert(rule, source_ids=cluster.episode_ids)
        # 4. Record conflicts with insufficient evidence; decide in the online phase
        for c in conflicts:
            mem_store.mark_pending_conflict(c)
    # 5. Maintain: dedupe, merge, evict by utility, rebuild index
    mem_store.dedup_and_merge()
    mem_store.evict(by="utility_score", keep_top_k=5000)
    mem_store.reindex()

The read path on the online side matters just as much. It is recommended to cap the injection size and explicitly carry metadata:

def build_context(task, mem_store, max_notes=8):
    notes = mem_store.retrieve(
        query=task.description,
        filters={"task_type": task.type, "status": "active"},
        limit=max_notes,
    )
    return "\n".join(
        f"[{n.id}] ({n.updated_at}, verified {n.hit_count} times): {n.text}"
        for n in notes
    )

8. Evaluation: How to Prove It “Remembered” Rather Than “Memorized”

Evaluation design is more easily overlooked than the model itself. An effective benchmark needs three characteristics: multiple independent sessions (each round is a fresh context, ensuring only memory can carry information across), a decidable correct answer per round (providing the feedback signal that drives learning), and later rounds requiring experience from earlier rounds (otherwise memory has no role to play).

Offline lookup tasks are a good vehicle: have the agent query answers within a data structure and answer a series of questions, announce the correct answer after each, and allow experience to be written to a memory directory for later sessions. This kind of task has objective feedback, clear judgment, and can run long sequences automatically.

Evaluation should look at four categories of metrics:

• Effectiveness: answer accuracy of the no-memory baseline vs. the memory-enabled system over the same number of rounds, plus the cumulative reward curve. The baseline is a mandatory control; otherwise you cannot distinguish “the model got stronger” from “memory is doing the work.”
• Retrieval: recall hit rate, proportion of ineffective notes, average number of injected items.
• Cost: extra tokens and offline compute consumed per unit of accuracy improvement.
• Health: memory store growth rate, proportion of overridden entries, size of the conflict backlog.

One thing worth knowing in advance from practice: different model configurations exhibit completely different learning shapes. Some show a cumulative reward that keeps rising, indicating the learning loop is closed; some plateau midway, a common cause being noisy feedback signals or imprecise retrieval; some oscillate, often because mutually contradictory entries coexist in the memory store and get recalled alternately. When you see a plateau curve, investigate the retrieval and feedback layers first rather than switching models.

Three ablation groups are also recommended: keep only the raw log, remove the dreaming phase and do online writing only, and disable retrieval to inject everything. Only then can you localize which link the gains actually come from.

9. Engineering Risks and Governance

• Memory poisoning (error entrenchment). The most dangerous failure mode: a wrong conclusion gets written, then repeatedly retrieved and cited, forming a self-reinforcing loop. Countermeasures: enforce source labeling and confidence, allow new evidence to override old conclusions, periodically re-verify high-frequency entries, and keep a “falsified” state rather than deleting outright.
• Memory bloat. An unconstrained memory store simultaneously drives up storage and injection costs and degrades retrieval precision. Countermeasures: evict by utility score, set per-user/per-tenant caps, and merge and compress same-topic entries.
• Privacy and compliance. Raw logs inevitably contain personal information and sensitive business data. You need identification and redaction before writing, set retention periods, and enforce strict per-tenant isolation; in particular, note the executability of deletion requests — deletion must cascade to the raw records, notes, index, and all derived representations, and deleting only one layer is not enough.
• Explainability and auditability. Every injection should be traceable to its source session and time; otherwise, when the model behaves abnormally, you cannot tell whether it is a reasoning problem or a memory problem.
• Concurrent write consistency. Online writes and offline dreaming may modify the same entry simultaneously, so use idempotency keys, version numbers, or optimistic concurrency control to avoid offline conclusions being silently overwritten by online writes.

10. Go-Live Checklist

1. Are the boundaries of memory clearly defined, with responsibilities distinguished from fine-tuning and in-session learning?
2. Is there an append-only raw-record layer as the single source of truth?
3. Does the write path carry all three signal types: explicit, rule-triggered, and feedback-driven?
4. Does offline dreaming run on an independent worker pool, with a tuned sampling strategy?
5. Does conflict handling follow “new evidence overrides old conclusions + preserve history”?
6. Does the retrieval layer have a baseline control, and is the injection size subject to a budget cap?
7. Does evaluation use a multi-session independent-context benchmark, observing the shape of the cumulative reward curve?
8. Are utility statistics, eviction policy, and index-rebuild capabilities in place?
9. Are redaction, retention, tenant isolation, and cascading deletion closed off end to end?
10. Is every injection traceable to its source and update time?

Designing memory as an independent data subsystem — with a clear read/write protocol, lifecycle, and quality metrics — is the fundamental reason this approach can actually run. The value of offline dreaming lies not in “making the model smarter,” but in systematically turning the cheap byproducts an agent produces during real tasks into reusable, correctable, auditable long-term assets.