Hierarchical and Idle-Time Consolidation for Cross-Session Memory in AI Agents

Four Layers Instead of One Large Notebook

Engineering-wise, it is practical to decompose memory into four distinct layers, rather than letting a single store handle every kind of semantic need.

Session logs serve as the factual ledger. They record user instructions, tool returns, key intermediate results, and anomalies in chronological order. The emphasis is on traceability, not conciseness. Logs can be long, but by default they do not enter the next prompt. They are the raw material for later review.

Memory notes are short, future-oriented entries. During a session, the model or an external process flags events that might be useful later: confirmed business rules, stable user preferences, or verified workarounds after a failure. Notes should be searchable, overridable, and markable as invalid—they are not an append-only stream.

Knowledge representations are the structured layer. After merging and summarizing notes, they form rule tables, entity relationships, operation steps, or constraint lists. They answer “how are things usually done for this kind of situation” rather than “what was on the screen last time.” Once stable, these structures are more reusable and verifiable than scattered sentences.

Injected context is the runtime view. Every round, only a small number of relevant notes and knowledge fragments are selected, along with a necessary summary of recent logs. This layer is a budget layer: there must be limits on the number of entries, total length, and conflict priority. Otherwise, memory can crowd out the reasoning budget.

The four layers are connected by one-way refinement and two-way correction. Logs support notes; notes precipitate into knowledge; knowledge is injected on demand. If subsequent observations conflict with existing knowledge, the system should write back an invalidation marker instead of letting old conclusions permanently outrank newer evidence.

Memory Is Essentially Prediction

Not all information deserves cross-session retention. The value of a memory entry is based on how likely the model will depend on it in the future. Verified rules, recurring constraints, and expensive failure causes have high predictive value. Momentary screen coordinates, temporary counters, or speculative statements from casual conversation have low predictive value.

A simple heuristic can guide the decision: If this piece of information were missing in the next round or the next similar task, would it cause repeated exploration, violated constraints, or incorrect actions? If not, leave it in the logs and do not promote it. If yes, write it as a note and clearly state the basis and scope. Treating “might be useful” as “store everything by default” will cause retrieval to surface stale states, and the model will mistake them for current facts.

Idle-Time Consolidation: Moving Reinforcement Out of the Dialogue

Humans consolidate memories during sleep, not by rewriting all experiences in every waking moment. An agent can similarly perform heavy work during idle windows between interactions, when there is no pressure to respond in real time. In these windows, the system can replay logs, compare notes, discover contradictions, and update the knowledge base.

Idle-time consolidation should be a repeatable batch process, not an extension of the online dialogue. A typical sequence looks like this:

Review — Read logs and existing notes that have not yet been consolidated within a time window. Mark new facts, repeated facts, and contradictory statements.

Correction — Perform consistency checks on verifiable fields (coordinates, IDs, time ranges, units). If later tool results overturn an earlier record, update based on the newer observation and keep a brief note that the previous record was wrong, so the same error is not adopted again.

Refinement — Promote recurring, outcome-verified practices into rules. Write out the precondition, action, and counterexample. Unverified hypotheses are marked as “pending confirmation” and are not injected by default.

Update — Merge duplicate notes, delete or downgrade expired entries, and refresh the version of the knowledge representation. Online sessions only consume the consolidated results.

The following is a generic illustrative flow, independent of any specific vendor. It shows the control flow rather than a proprietary interface:

on_idle(session):
  raw = load_logs(since=last_dream)
  notes = load_notes(session)
  conflicts = find_conflicts(raw, notes)
  for c in conflicts:
    verified = recheck_with_source(c)  # only rewrite when verifiable
    if verified.overrides_old:
      mark_stale(c.old_note)
      upsert_note(verified.statement, evidence=verified.refs)
  rules = induce_rules(notes, min_support=2, require_test=True)
  save_knowledge(rules)
  set_checkpoint(session)

Idle consolidation must be bounded: limit the replay length, the number of rewrites per run, and a timeout. If consolidation fails, the previous version of the knowledge base should remain usable; do not inject half-finished rules into production. Logs, object storage, and task scheduling can be handled by generic logging services, object storage, and asynchronous jobs—there is no need to embed the consolidation logic inside the dialogue process.

Less but Better: Prefer Compact Strategy Over Large State Snapshots

A common failure in highly interactive tasks with rapidly changing state is to write every frame of the situation to a file. The state changes constantly, old snapshots are almost always outdated, and retrieval will treat stale snapshots as current facts. In contrast, keeping only a small number of tested strategies, plans, and rules occupies less space while guiding many future steps.

In practice, set a hard cap on notes and enforce eviction. Give priority to:

• Verified rules
• Unfinished plans and their prerequisites
• Explicit and stable user constraints
• High-cost errors and their corrections

Priority for removal:

• Ephemeral UI state
• Unreviewed numeric values
• Repeated paraphrases
• Background unrelated to current goals

Write notes with fixed fields instead of free-form prose: conclusion, scope, evidence pointer, confidence, expiration condition, and update time. For retrieval, first filter by task type, then sort by recency and confidence, then inject only the top few entries. In the prompt, state: “If in conflict with the latest tool results, defer to the tool results.”

Failure, Reflection, Verification, and Re-Summarization

A more robust learning loop is not “the model thinks it understands, so it writes.” Instead, it is failure-driven. When an action deviates from the goal or a tool returns an error, first record the failure phenomenon and the evidence available at the time. During the idle period or before the next step, propose a testable correction hypothesis. Verify it using subsequent observations or read-only checks. Only after verification passes, promote the hypothesis into a rule.

This avoids two extremes: completely memory-less trial-and-error, and unverified speculation being solidified. Rules should also carry counterexamples. If a rule fails in a new context, narrow its scope—do not delete the entire history, and do not let one old rule overwrite new evidence.

Boundaries in Implementation

Memory cannot replace permissions and audit. Notes should not contain secrets, unauthorized sensitive personal information, or implicit conclusions that cannot be explained to the user. Cross-session content must be disabled, exportable, and clearable. Every rewrite during idle consolidation should leave a diff record so it is traceable which review introduced an error.

For long-cycle tasks, watch three signals: whether repeated errors drop, whether injected context remains short, and whether most knowledge entries carry evidence rather than stale states. If task success rate improves but notes explode, the promotion standard is too loose. Tighten the “useful for the future” threshold instead of widening the injection window.

Treat logs as raw material, notes as predictions, knowledge as verifiable structures, and idle periods as consolidation windows. An agent can then maintain long-term consistency within a limited context. The goal of memory is not to retain everything, but to keep the part that remains valid next time and is worth relying on.