Long-Term Agent Memory: Designing Background Consolidation

An agent that has completed a complex task often starts the next session from scratch. A larger context window still cannot hold the full history, and pasting entire logs back into the prompt only buys noise at additional cost. Writing experience directly into the system prompt may look simple, but it goes stale within months, and no one is willing to change it. The real problem of long-term memory is not whether a file still exists. It is deciding which statement to trust now and which one to carry into the next inference.

The problem: archives are not memory

Session transcripts are archives. They are append-only, suitable for auditing, replay, and dispute resolution. Memory must be editable: errors need correction, duplicate experience needs merging, and outdated conventions need demotion or removal. If the working agent also acts as an archivist while executing, three failure modes appear.

First, the write timing is wrong. Judgments made mid-task are often local and have not yet been falsified by later steps. Writing them immediately into long-term memory freezes intermediate assumptions. Second, the retrieval target is wrong. If memory is piled up according to “what happened” instead of “what is most likely to be needed next time,” injecting it into context crowds out the constraints that actually matter. Third, maintenance ownership is unclear. Without an independent verification step, errors are repeated across sessions and gradually look like facts.

A safer division of labor is therefore: execution produces only traceable raw records and a small number of candidate notes; consolidation reads those records during idle time and decides what to keep, correct, merge, or discard.

Design principles: organize for future tasks, not recall everything

The structure of memory should follow the agent’s responsibilities rather than forcing every agent into the same schema.

Coding agents need project boundaries, directory responsibilities, interface contracts, test entry points, and confirmed pitfalls. This information reduces repeated exploration, while “a preference the user mentioned in passing last week” is less relevant. Personal assistants need stable preferences, long-term goals, and confirmed constraints such as output language, data that must not be touched, and fixed reporting cadence. Agents that process large document collections are better served by an encyclopedic structure: concept pages, entity pages, decision records, and cross-references, rather than chronological chat summaries.

Three practical filtering rules:

• Keep only information that changes future action. Purely procedural trial and error does not need to enter the resident context once its conclusion has been absorbed into a convention.
• Every memory item must point back to evidence. Assertions without a session or document source can only be treated as items pending verification.
• Distinguish “fact,” “convention,” and “assumption.” An assumption can guide one attempt, but it must not be injected at the same level as a confirmed convention.

Four-layer structure

Four layers can separate append-only archives from memory that must change. The names describe responsibilities, not a proprietary interface from any product.

Raw record layer. This layer stores session transcripts, summaries of tool-call results, and decisions the user explicitly confirmed. In principle it is not rewritten; it is append-only. It provides the evidence for checking memory. Object storage or log services are suitable, indexed by session and time, with a retention period.

Note layer. Candidate lessons are extracted from a single session or a small number of sessions: wrong conclusions, effective practices, and user corrections. Notes may change, but each change should record the reason, for example “conflicted with a later session; demoted.” Notes answer “what was learned this time,” not yet “what should be the default from now on.”

Knowledge layer. This layer holds structured knowledge intended for reuse. In coding scenarios it can be architecture descriptions and code conventions; in document scenarios, topic pages and entity relationships; in assistant scenarios, preferences and goals. Knowledge pages should be short, citable, and mergeable. When multiple notes point to the same conclusion, they are promoted into one knowledge item, with the source of confidence noted as either “repeatedly consistent” or “explicitly confirmed by the user.”

Injection layer. What actually enters the next inference context is only a small slice of the knowledge layer plus a few notes relevant to the current task. Injection content needs a budget: rank by relevance, freshness, and confirmation level, and discard low-priority items when the limit is exceeded. The resident system prompt should contain only a very small number of stable principles, avoiding volatile details welded into the prompt.

Movement between layers is promotion and demotion, not copy-paste. Notes are promoted to the knowledge layer after repeated validation. When knowledge conflicts with new records, it is demoted back to notes or marked stale rather than silently overwritten.

Background consolidation: move curation off the execution path

Consolidation jobs run in idle windows after sessions end or as scheduled jobs, without competing with user requests for the same inference turn. They read raw records, existing notes, and knowledge pages within a time window and perform four kinds of actions.

Verification. Compare notes against raw records. If a note claims “the user requires a certain format at all times” but the record contains only one example, it should be demoted to an assumption. If the user later explicitly rejects it, correct the note and preserve the rejection itself so the old conclusion is not promoted again.

Distillation. Merge repeated patterns across sessions. For example, if the same boundary condition is hit in the same module multiple times, write it as an executable convention: trigger condition, correct approach, and counterexample. Remove conversational tone and irrelevant details during distillation, and keep testable statements.

Reorganization. Rearrange the directory according to future tasks. Coding knowledge is organized by module and convention type; personal preferences by decision impact; document knowledge by topic and entity. Chronological order is only a secondary index, not the main structure.

Injection scheduling. Adjust which knowledge pages enter the default injection set based on the distribution of recent tasks. Items that have not been used for a long time and are not safety constraints are removed from the default set but remain in the knowledge layer for retrieval. Safety, permission, and data-boundary constraints take precedence over habit preferences.

The consolidation process itself should also be auditable: each run records which sessions were read, which pages were changed, and why. If memory becomes distorted, it can then be replayed rather than only cleared and rebuilt.

A minimal pseudocode sketch can express responsibilities without implying a runnable product interface:

for session in recent_transcripts:
    notes = extract_candidates(session)      # produce candidates only
    conflicts = diff_against(notes, knowledge)
    apply_corrections(conflicts, evidence=session)
promote(notes, rule="repeated_or_user_confirmed")
demote(stale_or_contradicted_pages)
injection = select_for_next_tasks(knowledge, budget)

Comparison with common workarounds

Putting the entire history into context gives precise retrieval but is expensive, and old errors sit next to new conclusions with no way for the model to judge priority. Hardcoding experience into the system prompt is initially simple but lacks a verification loop; once it goes stale, maintenance risk outweighs the benefit. Relying only on vector retrieval of similar snippets can recover original text but does not merge contradictions; similarity is not correctness.

Background consolidation fills the gap of “governance after writing”: raw records are responsible for truth, knowledge pages for brevity, and the injection layer for restraint. Long-term memory becomes usable only when all three hold at the same time.

Engineering considerations for production

Isolation and permissions. A consolidation job may read only the sessions and documents authorized for that agent; it must not scan across tenants. Sensitive fields in memory pages should be classified. Default injection should avoid secrets, identity numbers, and other content that should not be repeated; when necessary, retain only “a constraint exists” without the original text.

Idempotency. When the same session is scanned repeatedly by a scheduled job, it should not generate duplicate knowledge pages. Use session identifiers and content hashes for deduplication, and base knowledge-page edits on version numbers so two consolidation jobs do not overwrite each other.

Conflict policy. The user’s latest explicit instruction takes precedence over model-generated induction; safety policy takes precedence over convenience preferences. When a conflict cannot be resolved automatically, mark it as pending confirmation. Do not use voting-style merging to blur opposite conclusions into one vague sentence.

Evaluation. Do not look only at “how many items were remembered.” More useful metrics include whether repeated exploration declines, whether corrected memories disappear in the next turn, whether injection tokens remain stable, and whether wrong memories are cited fewer times. Build a fixed task set and compare three settings: no memory, raw-text retrieval only, and layered consolidation.

Failure fallback. A failed consolidation job should not block online sessions. The online path reads only the last published injection set; a new version switches over after validation. If a knowledge page is too large, references are broken, or it conflicts with safety policy, reject publication and raise an alert.

A coding-agent walkthrough

Suppose an agent fixes defects in the same codebase over several days. Raw records preserve the scope of each change and review comments. The note layer records “a certain directory must not access storage directly; it must go through the existing data-access wrapper.” After the same point is repeated in a third review, the consolidation job promotes it to a module convention and links it to the relevant review summaries. The next session’s injection layer carries only this convention and the directory responsibilities, not the long back-and-forth trial and error from previous days. If a later architecture change invalidates the convention, the new record will demote the old page in the next consolidation, preventing two opposite conventions from remaining resident at the same time.

Personal assistants and document-oriented agents follow the same logic; only the skeleton of their knowledge pages differs. The former uses preferences and goals, while the latter uses concepts, entities, and decision records. Do not merge the three skeletons into one large table.

Conclusion

Long-term memory is worth building, but it should not become a second, unbounded chat log. Separate execution from curation, archives from memory, and then decide what enters context based on future tasks. Verification, distillation, reorganization, and injection scheduling during idle periods are the main mechanisms that turn “remembering” into “remembering correctly and usefully.” On cloud infrastructure, object storage can retain raw records, scheduled jobs can trigger consolidation, and monitoring and alerting can observe job failures and injection size. These are generic capabilities, and the architecture can be explained without depending on any proprietary interface.