From “413 notes losing to 9” to sleep-time compute

In research on long-term memory for AI agents, the authors observed a counterintuitive result: an agent that kept writing hundreds of notes into its memory store performed worse on follow-up tasks than a version that had distilled only a handful of refined memories. The lesson is that the value of a memory system is not determined by how much it stores, but by whether what it stores is useful and can be retrieved accurately.

Behind this finding is a direction that more and more teams are starting to pay attention to: sleep-time compute. The core idea is that a model does not have to perform inference only during the “awake” moments when it is responding to a user request. It can also use the idle time between interactions to reorganize, compress, and consolidate its existing context and memory — much as the human brain consolidates memory during sleep.

What sleep-time compute is

In the traditional inference paradigm, almost all compute is consumed at test time: the model thinks harder on the spot when it faces a concrete problem, with longer chains of thought and more sampling. Sleep-time compute moves part of that work earlier, into offline gaps:

• Awake phase (online): focused on the current request, optimizing for low-latency, immediate responses.
• Sleep phase (offline): uses idle compute to review past interactions, tidy up, abstract, and denoise memory, so that high-quality context is ready for future requests in advance.

At its core, this is a reallocation of compute along the time dimension: a controlled offline cost is traded for shorter, more accurate context in the online phase, which reduces the burden of every single inference.

Why “remembering more” leads to “remembering worse”

To understand the experiment in which 413 notes lost to 9, it helps to recognize three mechanisms.

1. Signal-to-noise degradation
Every low-value, repetitive, or outdated note is noise for future retrieval. Once noise dominates, the model is more easily distracted by irrelevant information at recall time and more likely to draw the wrong associations.

2. Declining retrieval precision
The more memory there is, the larger the candidate set for vector or keyword search, and the higher the probability that Top-K results include content that looks relevant but is actually useless. A small deviation at recall is then amplified step by step by downstream reasoning.

3. Context pollution and “lost in the middle”
Stuffing a large number of notes into the context window dilutes the information that truly matters and triggers the well-known “lost in the middle” problem in long contexts. The longer the context, the more scattered the model’s attention to core cues becomes.

In one sentence: memory is a system that needs active maintenance, not a warehouse where things only go in.

Four core strategies for organizing memory

For a “dreaming” agent to actually improve its recall, the offline consolidation phase should cover at least the following four types of operations.

1. Compression and summarization

Distill long, verbose raw interactions into structured, high-information-density memory units. Keep conclusions, constraints, and key facts; discard process noise.

2. Deduplication and merging

Merge memories that are highly similar in meaning so the same fact does not appear repeatedly in multiple versions. When merging, keep the version that is the most complete and the most recent.

3. Semantic layering

Organize memory into a hierarchy by topic, time, or task (for example “user preferences,” “project background,” “past decisions”), so retrieval can first locate the right partition and then match precisely within it.

4. Active forgetting

Assign recency weights to memories so that expired or long-unreferenced content is down-weighted or even evicted. Forgetting is not a defect; it is the capability that keeps the memory store healthy over the long term.

A simplified offline consolidation flow

The following generic pseudocode shows how the “sleep phase” turns raw notes into high-quality memory.

def consolidate_memories(raw_notes, gain_threshold=0.6, sim_threshold=0.85):
    """
    Offline memory consolidation pipeline:
    1. Filter out notes with low information gain
    2. Semantic deduplication and merging
    3. Structured summarization and indexing
    """
    consolidated = []
    for note in raw_notes:
        # 1. Discard low-value content (no new information or already outdated)
        if note.information_gain < gain_threshold:
            continue
        # 2. Compare against existing memories by semantic similarity
        duplicate = find_semantic_duplicate(note, consolidated, sim_threshold)
        if duplicate:
            merge_into(duplicate, note)   # merge instead of appending
        else:
            consolidated.append(note)
    # 3. Layered summarization + build a searchable index
    return build_indexed_summary(consolidated)

The point of this flow is not “write more” but “select carefully”: filter first, then deduplicate, then structure. After one such round of “sleep,” a memory store often converges from hundreds of entries to a few dozen that are genuinely useful — which is exactly what the “413 loses to 9” result points to.

How to evaluate a memory system

Memory quality should not be judged by capacity alone. Useful dimensions include:

• Retrieval hit rate: whether the system recalls the relevant memories accurately when facing real tasks.
• Context efficiency: the number of context characters required to solve the same problem (the leaner, the better).
• Resistance to noise: how much task performance drops after noisy memories are injected.
• Freshness maintenance: whether expired information is down-weighted or removed in time.

A healthy memory system balances information completeness against retrieval precision, instead of accumulating without limit.

Engineering practice recommendations

• Control the write threshold: not every interaction deserves to be remembered; judge its information gain first.
• Trigger consolidation periodically: run the consolidation logic in batches during idle time rather than processing it synchronously on every write.
• Tiered storage and retrieval: use generic cloud building blocks to carry long-term memory — object storage for archiving, vector search for recall, log services for auditing — and keep the online phase read-only over the condensed context.
• Observability: build monitoring and alerting for the memory store, tracking metrics such as volume growth, hit rate, and the proportion of contaminated entries.

Closing thoughts

“An AI that dreams actually remembers better” captures an engineering truth that is easy to overlook: the core competitive advantage of a memory system lies in organizing it, not hoarding into it. Sleep-time compute offers a clear path — invest part of the compute in offline memory consolidation, replace accumulation with curation, and replace blind recording with denoising. For teams building long-term-memory agents, rather than chasing “remember everything,” it is better to design well for three things: how to forget, how to merge, and how to retrieve.