Agent Memory Architecture: Sleep-Time Compute and Brain-Like Consolidation

Why Agents Need Memory

Today’s large-model agents often look strong inside a single session and almost amnesic across sessions. Yesterday’s mistakes can repeat; patterns summarized last week are not automatically retained. Extending the context window is not enough for agents that must run for a long time. They need a dedicated memory layer.

Agent memory has a clear boundary. It should not be confused with three related ideas:

1. Updating model weights. Fine-tuning or continued pre-training writes knowledge into parameters. That process is costly, slow, and prone to catastrophic forgetting. It is not the memory layer.
2. In-session in-context learning. The model may pick up user preferences or task patterns in the current conversation, then lose them when the session ends. That is not durable memory.
3. Summary-as-context. Compressing old dialogue into a summary and stuffing it back into the window is lossy transport, not independently retrievable, evolving experience entries.

True memory is experience the agent acquires from tasks and interaction, keeps across sessions, and later reads and reuses as plain text. It is independent of model parameters and of the current context window. External storage can hold it so it can be retrieved, edited, or deleted at any time.

Sleep-Time Compute: Idle Time as Learning Time

Concept

Sleep-time compute is the idea that a model should not think only when it is called. Idle time between interactions is also compute. The analogy is human sleep: the brain does not stop; it replays the day at lower energy cost, strips noise, and consolidates important memories.

For an agent, this offline pass can be thought of as “dreaming”: reviewing interaction logs, checking earlier reasoning errors, and rewriting or summarizing memory entries. Advantages include:

• Controllable cost. Offline jobs have no latency SLA, so they can use a longer thinking budget and more self-checks.
• Safer error correction. Online mistakes can fail a task; offline reflection can overturn old conclusions without that risk.
• Higher memory quality. Multiple recall–evaluate–rewrite cycles turn raw logs into concise rules and facts.

Typical Idle-Time Jobs

In practice, sleep-time work often includes:

1. Log replay: scan recent session logs and find critical success/failure points
2. Error correction: find stale or contradictory memory entries, mark them, and rewrite
3. New notes: distill unsaved observations into structured experience entries

The memory store should support versioned writes, ideally with logging and storage so every entry has a traceable source and can be rolled back.

Brain-Like Memory Tagging

Emotional Signals and Priority

Neuroscience shows that emotional or chemical signals tag memories at key moments so they are replayed more during sleep and consolidated more strongly. The brain does not treat all experience equally; it has a priority scheduler.

Agents can copy this: when an event looks special—task failure, explicit user correction, a new pattern—tag it high priority or write it straight into an external memory store. During sleep-time, tagged items are replayed and consolidated first.

A Simple Record Shape

A simplified memory entry:

{
  "id": "mem_20240601_017",
  "content": "For long documents, split by section then summarize each part to avoid truncation from a single read",
  "priority": "high",
  "origin": "session_8823",
  "created_at": "...",
  "last_reviewed": "...",
  "confidence": 0.86
}

priority is the analog of an emotional tag. confidence tracks how often the experience has been verified and how consistent it is. last_reviewed supports expiry during sleep-time.

The Hard Problem: Predicting What Will Matter

How does the agent know which current event will matter later? Human tagging has a biochemical basis; models only have their own judgment. Practical approximations include:

• Surprise detection. Events that conflict with existing memory or expected outcomes often carry new information.
• Correction events first. Explicit “that’s wrong; do it this way” from the user is a strong signal of future usefulness.
• Repeated patterns. If a situation appears more than once, it is worth turning into a rule.

Memory as a Bet on the Future: Closing the Learning Loop

Taking Notes Is Prediction

Writing a note is a prediction: the agent is betting that the rule will be useful later. That view turns memory from a passive store into a predict–verify loop:

1. Predict. Form a hypothetical experience during interaction and write it to memory.
2. Verify. Retrieve related memory on later tasks, act on it, and see whether the outcome matches.
3. Strengthen or drop. Successful items raise confidence; repeated failures are down-weighted or deleted in sleep-time.
4. Generalize. Similar items are merged into more abstract, reusable rules.

Engineering the Loop

Three capabilities make the loop work:

• Reliable retrieval. Written memory must be recalled in the right situation, typically via a mix of semantic vector search and keyword indexes, so items are not stored and then forgotten.
• Conflict resolution. When old and new experience clash, timestamps, confidence, and verification counts decide what to keep so the store does not contradict itself.
• Periodic audit. Sleep-time “dreaming” audits in batch: replay, cross-check, and keep quality usable.

Implementation Notes and Outlook

Start simple: plain-text memories plus tags. Add vector retrieval and confidence models only after the value is proven. Isolate sleep-time jobs from the online service; use idle compute and async queues. Keep an audit trail and rollback for every write, and monitor store health.

The point of a memory system is to move agents from starting from scratch every time to getting smarter with use. Brain-like tagging, replay, and consolidation are a long-evolved design sketch; sleep-time compute is a practical path. As the pattern matures, long-running agents can genuinely improve themselves over time.