Why Agent Memory Degrades in Long Runs — and How Offline Consolidation Fixes It

1. Where It Starts: Why Agent Memory Turns Into a Junk Drawer

Anyone who has run an agent for a long time, across many sessions, knows the curve: the first few sessions are full of useful memory, and each session after that is less useful than the last. The cause is not mysterious.

Agent memory is written locally, incrementally, and in place. While executing a specific task, the agent decides that a given piece of information is valuable for that task and writes a record. From the perspective of the moment of writing, every decision is reasonable:

• “Tests in this project run through the test script bundled with the package manager, not npm test“ — true at the time.
• “The id field on that endpoint is a string” — also true at the time.

The problem is that writing costs almost nothing, while curation is never paid for. As sessions accumulate, the memory store therefore evolves in a predictable way and develops three kinds of pathology.

1. Duplication. The same fact gets written many times in different wording. Three records describe one thing, and semantic retrieval recalls all three.

2. Contradiction. The environment itself changes. When the build tool is swapped out midway through a project, the old and new conclusions stand in direct opposition, and both wear the appearance of “high confidence.” The agent’s behavior becomes unpredictable: one retrieval surfaces the new conclusion, the next surfaces the old one.

3. Staleness. Details recorded early no longer hold, but no field records that they have expired. They sit quietly in the store, available at any time to mislead a decision.

The collateral damage of these three problems is usually underestimated:

• Signal-to-noise collapse. Half of the top-K retrieval results are duplicates, so the density of useful information is cut in half.
• Wasted context budget. What gets injected is redundant text rather than new knowledge; the tokens are spent standing in place.
• Reduced interpretability. When the agent makes a wrong decision, you cannot tell whether the reasoning was flawed or the memory was dirty.
• Higher cost. Longer context means higher inference overhead, for worse results.

In other words, this is not a “not enough memory” problem. It is a never maintained problem.

2. The Borrowed Idea: Memory Consolidation, Not Longer Context

Neuroscience has a well-established concept here: memory consolidation. The brain does not archive the day’s experiences verbatim; during sleep it completes replay, pruning, and transcription, integrating fragmented short-term traces into structured long-term knowledge while clearing redundant connections.

Transplanted to agents, this yields a clear dual-channel architecture:

Phase	Human	Agent
Online encoding	The day’s experiences	Incremental memory writes during a session
Offline consolidation	Sleep-time replay and integration	Batch memory curation between sessions
Retrieval	Cue-based recall	Semantic retrieval + confidence weighting

The key insight is that writing and curation must be decoupled. The online path optimizes for low latency and never blocking the task; the offline path optimizes for a global view and high-quality adjudication. They run at different tempos, fail with different blast radius, and require different observability. Packing both into the same synchronous flow is a root cause of memory system rot on many teams.

This is also why the gap between sessions is a natural consolidation window: no active task is latency-sensitive, and enough new experience has accumulated to be worth analyzing.

3. The Offline Consolidation Pipeline, Stage by Stage

Below is a pipeline design you can put into practice directly. Three core principles: append-only (versioned), auditable, idempotent.

Stage 0: Trigger Policy

Do not use a fixed timer. Combine conditional triggers with a fallback schedule:

Enqueue when any of the following holds:
- Number of newly written memory entries >= N (e.g., 20)
- Time since the last consolidation run > T (e.g., 24 hours)
- A suspected conflict appears in retrieval results
  (multiple highly similar records under the same key)
- A session ends normally and today's write count is non-zero
Scheduling fallback:
- Low-priority batch job in an idle window, so it does not contend
  with online inference for resources

Stage 1: Candidate Extraction (Experience Replay)

The input is not the full conversation transcript but a structured session trace: the task goal, a summary of execution steps, tool calls and their results, the final outcome (success or failure), and any explicit user corrections. The extraction target is “reusable facts and experience,” typically falling into these categories:

• Project conventions (factual): test commands, directory layout, naming rules.
• Environment facts (factual): dependency versions, field types, behavior of external APIs.
• Operational procedures (procedural): a reliable path for fixing a class of errors.
• Preferences and constraints (preference): requirements the user stated explicitly.
• Lessons from failures (negative): the reasons a given approach does not work in this environment.

Negative memory is the easiest to overlook, yet its value in reducing repeated trial and error is often higher than that of positive memory.

Stage 2: Normalization and Deduplication

Normalize first, then merge by similarity:

def normalize(entry):
    # 1) Subject extraction: turn prose into "subject -> predicate -> value"
    # 2) Term normalization: synonym map + case folding + path separators
    # 3) Fill in environment qualifiers: project / branch / env / date
    # 4) Strip modifiers that carry no information
    return canonical_form(entry)
def dedup(cluster):
    # Vector similarity + exact match on subject key -> merge directly
    # Between the 0.85 similarity band and the threshold -> send to an LLM
    # to decide whether the two express the same proposition
    return merged_or_new(cluster)

One easily missed detail: every memory entry must carry environment qualifiers. “Run tests with command X” is a fact inside a single repository, but a source of contamination inside a multi-repository toolchain. Missing qualifiers are the number-one cause of cross-project memory bleeding.

Stage 3: Conflict Detection and Adjudication

For multiple assertions under the same subject, compare them pairwise and classify the relation as consistent / refinement / conflicting / unrelated. Adjudication uses the following priority order:

1. Explicit user correction — the highest weight, which can effectively override directly.
2. Recent verification evidence — the result of the most recent tool call that genuinely succeeded.
3. Evidence count — an assertion reproduced across multiple independent sessions is more trustworthy.
4. Recency — when the environment has changed, the newer usually beats the older.

An adjudication result is not a deletion. The losing entry is tagged with superseded_by and its history is retained. That way you can roll back, and when the agent makes a choice that contradicts a historical conclusion you still have an audit trail.

Stage 4: Aging and Decay

Maintain a usability score per memory entry, decaying with time and use:

score = w1 * confidence          # confidence of the assertion
      + w2 * recency_decay        # exp(-lambda * dt), lambda varies by category
      + w3 * retrieval_utility    # lift in task success rate after retrieval historically
      + w4 * evidence_count       # number of independent reproductions (log)
      - w5 * contradiction_penalty

Decay rates should differ by category. Project conventions change slowly, so λ is small; external API behavior and dependency versions change fast, so λ is large. A single uniform half-life is a common design flaw.

The retrieval_utility term deserves the investment: let the effectiveness of using a memory decide whether it survives. Entries that are repeatedly recalled but have never improved a task outcome should be downweighted or archived.

Stage 5: Abstraction and Compression

This is the part of the “dreaming” stage closest to intelligence: deriving a higher-level regularity from many concrete experiences.

Concrete experiences:
  E1: pnpm test works in this repository
  E2: pnpm test -- --coverage produces a coverage report
  E3: running the test files directly with node fails,
      because alias resolution is missing
Compression result (procedural memory):
  "Tests in this repository always run through the package manager's test
   script entry point; bypassing that entry and calling node directly
   loses the path alias configuration."

The result is a three-tier structure: episodic (raw experience) → semantic (facts) → procedural (operational rules). Retrieval hits the higher abstraction first and drills down into the concrete experience only when detail is needed, which substantially shrinks the volume of context you inject.

Stage 6: Write-back and Version Control

• Run an idempotency-key check before writing (subject key + qualifiers), so repeated runs do not create duplicate entries.
• Take versioned snapshots of the whole memory store; each consolidation run produces a rollbackable version.
• Keep a diff view: how many entries were added, merged, downweighted, or overturned this run. That diff is the first place to look when diagnosing “the agent suddenly got dumber.”

4. Storage Model Sketch

{
  "id": "mem_8f31...",
  "type": "project_convention | env_fact | procedure | preference | lesson",
  "subject": "repo:checkout:test_command",
  "scope": { "project": "checkout", "branch": "*", "env": "ci" },
  "statement": "Tests run through the package manager's test script entry point",
  "value": "pnpm test",
  "confidence": 0.86,
  "evidence": [
    { "session": "s_102", "verified_at": "2026-07-11", "outcome": "success" },
    { "session": "s_117", "verified_at": "2026-07-28", "outcome": "success" }
  ],
  "contradicts": ["mem_2a09..."],
  "superseded_by": null,
  "score": 0.79,
  "last_retrieved_at": "2026-08-19",
  "consolidated_at": "2026-08-20T03:12:00Z"
}

The scope and evidence fields are what many homegrown memory systems are missing, and they are exactly the foundation of conflict adjudication and auditability.

5. Cooperating With Online Retrieval

Offline consolidation fixes a dirty store; the retrieval side fixes “getting the right one.” The two must be designed together:

• Dual-channel retrieval. Hot memory (the current session plus high-scoring long-term entries) is injected directly; cold memory (low-scoring, historical, archived) is drilled into only when needed.
• Best-of-subject selection. For the same subject, take only the highest-scoring entry that has not been superseded, which eliminates duplicate injection at the source.
• Budget control. Give long-term memory a fixed token quota and truncate by score when it is exceeded, so memory does not crowd out reasoning space.
• Hit feedback. Record retrieval_utility on every recall and feed it back to the consolidation stage as a decay weight.

6. How to Prove It Actually Works

Do not judge “memory got better” by feel. Use measurable indicators:

Dimension	Metrics
Store health	Share of duplicate entries, number of unresolved conflicts, share of expired entries
Retrieval quality	Information density in top-K, no-result rate, false-recall rate
Task effect	Cross-session task success rate, average retry count, first-attempt success rate
Efficiency	Context tokens per task, end-to-end latency, inference cost
Consistency	Agreement rate for the same question across different sessions

For method: run offline evaluation against a fixed multi-session regression set, then run a small live A/B test — one group with offline consolidation enabled, one group with incremental writes only — and compare the metrics above. Pay particular attention to cross-session consistency. It expresses the value of memory maintenance most clearly, and it is the hardest capability to fake with prompt engineering.

7. Engineering Pitfalls

1. Over-generalization. The abstraction stage easily promotes a special case into a rule, turning one sporadic failure into “this approach does not work.” Fix: an abstraction conclusion requires at least two independent pieces of evidence, and references to counter-examples must be retained.

2. Cost and latency. Fully replaying the memory store every round is unaffordable. Fix: incremental processing, model tiering (a lightweight model for normalization, a strong model for conflict adjudication and abstraction), and batching windows.

3. Isolation and compliance. Memory naturally accumulates user data. You must enforce hard isolation along session and tenant dimensions, support deletion and export by subject (the right to be forgotten), and make the provenance of each memory traceable.

4. Security. Memory is persistent state that can be poisoned. If external content (web pages, documents, tool returns) is written into memory without a trust label, you have effectively installed a cross-session injection persistence channel in the agent. Every write must record source and trust level, and content from external sources must not be promoted directly to high-confidence fact.

5. Rollback. Take a snapshot before every consolidation run so you can return to the previous version in one step when something goes wrong. Automated curation without rollback capability is a higher risk than its benefit.

6. Infrastructure selection. The requirements here are fairly modest: raw traces and the memory store can live in object storage or a versioned storage service; consolidation jobs are decoupled from their triggers by a message queue; batch runs execute as scheduled or elastic tasks on a container platform; quality metrics and diffs are observed through a logging service and monitoring and alerting. On Baidu AI Cloud, these capabilities can be composed directly from general-purpose product categories, so you do not need to build a dedicated system for memory maintenance.

8. What This Means for Agents

The significance here is not the anthropomorphic framing that “agents dream.” It is that this reveals a shift in engineering focus.

Over much of the past year, improvements in agent capability came mainly from stronger single-turn inference: larger models, longer context. But once agents start running continuously for days and weeks at a time, the primary constraint becomes state management: what the agent remembers, whether it remembers correctly, and whether it can update its understanding after the environment changes. This is the same transition software engineering went through when it moved from “writing functions” to “managing state” — the hard part was never computation, it was invariants.

Making memory consolidation an independent subsystem delivers three long-term benefits:

1. Maintainability. The memory store gains a schema, versions, a diff, and health metrics, and turns from a black box into an operable object.
2. Auditability. Every decision can be traced back to a specific memory entry and its evidence chain, which is a hard requirement in enterprise settings.
3. A higher capability ceiling. Cross-session consistency and long-term preference adherence are prerequisites for long-running agents to become reliable.

In the short term, offline consolidation is just a background batch job: low cost, small surface area of change, worth prioritizing. In the long term, “how memory is organized” is likely to become an independent and central discipline within agent engineering, much as “how data is modeled” became one for databases. Whoever gets this chain solid first is the one whose agents can actually do work over the long run.