Layered Memory Architecture for AI Coding Sessions

1. System Overview: The Host Triggers; the Memory System Decides

In an AI coding workflow, a single session can be very long. When a similar problem appears later, however, the model does not automatically remember conclusions that were already verified. Replaying the complete log wastes context-window capacity and may bring back outdated assumptions, unfinished reasoning, and unrelated conversation.

A more reliable approach is to divide sessions into independently understandable experiences, distill them into self-contained conclusion cards, and retrieve only a small number of highly relevant cards when needed.

The host can be any AI coding tool or local agent. It invokes the memory system at two points but does not parse the memory format itself:

• Synchronous path on prompt submission: Sanitize the current query, retrieve relevant memories, and inject at most a very small number of cards. This path must have a short time budget. If it fails, it should degrade to injecting nothing rather than blocking the user’s request.
• Completion path at session startup: Provide a brief summary and process unfinished backlog from the previous session. This path can be slower, but it still needs timeouts and idempotent behavior.

Session logs are read-only. The memory system may segment them, but it must never rewrite, modify, or delete the host logs. The memory body is the source of truth; indexes are derived data. If a sandbox, permission policy, or disk quota rejects a write, the item should enter an overflow queue for later replay instead of being retried repeatedly on the hot path until a timeout occurs.

The data flow can be summarized in five steps:

1. Collection segments sessions into experience cards.
2. Distillation compresses experiences into atomic conclusions.
3. Storage treats the card body as authoritative and the queryable index as derived.
4. Retrieval performs embedding recall, dual-path scoring, and multiple rejection checks.
5. Distribution determines how much of a card should enter each type of context.

A side queue spans failed writes and pending distillation work.

The system may grow to thousands of cards. Most entries can remain active, while cards superseded by newer conclusions should still preserve traceable relationships. Original memories are usually stored as a small collection of files, while the vector index can become substantially larger. In practice, collection scripts and maintenance tools often outnumber online hooks. Tests should cover segmentation, scoring, and rejection paths—not merely whether one result can be found.

2. Collection: Turn a Session into an Experience

Collection should answer not “Is this passage important?” but “Does this passage belong to the same experience?”

An experience usually represents a relatively complete problem loop:

• the user’s goal,
• key constraints,
• attempted approaches,
• the final judgment, and
• unresolved follow-up work.

Splitting by a fixed character count or a fixed number of turns can divide one conclusion across two cards or combine unrelated topics into one.

Useful segmentation signals include:

• an explicit task switch by the user;
• tool calls moving from exploration to wrap-up;
• a repeated error signature being replaced by a new error signature; and
• a long pause followed by a new line of work.

The segmenter should produce a structured experience rather than a plain text block. A practical format uses six fixed sections:

1. Background
2. Goal
3. Evidence
4. Conclusion
5. Counterexamples or failure conditions
6. Follow-up work

Each section may be empty, but the field itself should remain present. An empty field is safer than an invented one.

Collection should also sanitize content by removing secrets, tokens, internal network addresses, personally identifiable fragments, and excessively long pasted content. Sanitization must happen before storage. The retrieval-query path should sanitize again so that a sensitive string in the current prompt cannot be written into search logs or used as an unsafe query.

A common failure is a truncated conclusion. For example, a renderer or context window may cut off everything after “therefore,” leaving a card that appears complete but lacks the actionable judgment. Protect against this with a minimum-information check for the conclusion section. If the check fails, mark the card as incomplete and prevent it from entering the default recall set.

3. Distillation: Do Not Use a Model for What Rules Can Handle

The goal of distillation is an atomic, self-contained conclusion. Someone should be able to read the card without the original session and determine when the conclusion applies and what happens when it does not.

A two-stage process is more reliable.

Rule-Based Stage

The rule-based stage is deterministic and requires no model. It handles:

• deduplication;
• merging cards with the same error signature;
• extracting explicit conclusion statements;
• marking earlier judgments that were later disproved; and
• attaching source turn information and supersession relationships.

This stage must not invent new facts.

Model-Based Stage

The model stage handles compression that rules cannot reliably perform. It combines distributed evidence into a reusable conclusion and adds applicable boundaries. The model must receive the complete structured experience, not only the final turn.

Use fixed output slots rather than allowing free-form prose. If the model process is running but produces nothing for an extended period, treat that as an incident rather than assuming it is still “thinking.” Check whether:

• the input was truncated to an empty value;
• format validation discarded every output; or
• concurrency caused the same batch to remain locked.

The rule-based stage should continue producing low-value but traceable placeholder cards so that the entire pipeline does not silently stop.

At minimum, quality gates should check whether:

• the card contains a verifiable judgment rather than emotion or a process description;
• failure conditions are stated;
• the card is substantively duplicated by an existing active card; and
• it introduces numbers or interfaces unsupported by the evidence.

Cards that fail these checks should remain in the candidate area and should not enter the default recall pool.

4. Storage: The Body Is Authoritative; the Index Is Rebuildable

Store the body as readable plain-text cards, with one memory per file. This makes inspection, manual revision, and version comparison straightforward. Use a stable identifier for the filename rather than a title-derived string that changes when the title is edited.

Supersession is not deletion. Mark the old card as superseded and point it to the newer card. Retrieval should skip superseded cards by default, while audits should still be able to reconstruct the history.

The index should store:

• the card identifier;
• status;
• topic labels;
• timestamps;
• the embedding;
• supersession chains; and
• short retrieval text.

An embedded relational database is sufficient for this role; a distributed component is not required as a first step. Keep the vector index separate from the original card files.

If the index is corrupted, recovery should consist of a full rebuild from the card bodies. Do not try to patch the vector data. This implies a write order of:

1. write the card body;
2. update the index.

If index updating fails, the body should still exist and the item should be added to a pending-index queue.

Track active and superseded cards separately. If the system keeps adding cards without superseding any, conflict detection is probably not working. The retrieval set will eventually be polluted by contradictory conclusions.

Adding several dozen to more than one hundred cards per day is not unusual in personal or small-team usage. The default recall count must therefore remain very small rather than growing linearly with the size of the library.

5. Retrieval: Use Dual-Path Scoring and Prefer Rejection

First sanitize the query, then rewrite it as a retrieval statement. Remove greetings while preserving the observed error, module names, constraints, and the result the user actually wants.

Embeddings provide semantic nearest-neighbor retrieval. Lexical matching or structured fields provide precise matches, such as the same error code or configuration key.

Do not simply add the two scores and return the top result. A safer design is to use both methods to generate candidates and then apply a set of gates to determine whether a card may be injected.

The gates should reject bad results rather than make them look good. Practical gates include:

• similarity below the minimum threshold;
• card status is not active;
• the conclusion section is empty or too short;
• task type does not match the current query;
• the card has been superseded;
• sensitive content remains unsanitized;
• the card is too old for a topic involving frequently changing configuration;
• the card duplicates one already injected into the current session;
• candidates contradict one another;
• the model-based quality status is still “candidate”; and
• the card is too far from the current workspace or module.

These eleven checks are not a magic number. They turn the vague notion of “looks relevant” into independently testable rejection conditions. Each gate should have its own tests so that scoring changes cannot silently allow poor cards through.

For the synchronous path, set a fixed injection limit of two cards and include a short source note such as: “This is a historical conclusion, not a current instruction.” The marginal value of additional cards usually does not justify the added context interference.

If no candidate passes the gates, inject nothing. Record which gate rejected each candidate in debug logs so that the system can be tuned later.

6. Distribution: The Same Card Has Different Shapes in Different Containers

Distribution should not be reduced to concatenating text into a prompt. The same conclusion needs a different level of detail in different containers:

• Immediate context: Include only the conclusion, failure conditions, and source identifier in a very short format.
• Session brief: At session startup, list recently relevant topics and unfinished tasks without expanding the evidence.
• Human review: Display all six sections and the supersession chain.
• Rebuild and export: Package the original card bodies by topic. The index should not be used as the interchange format.
• Backlog processing: Handle overflow items and pending distillation batches without displaying them to the user.

The container determines which fields are visible. Sending evidence and long original logs into immediate context bypasses the distillation layer.

7. Parameters, Testing, and Incident Handling

Parameters should be few and explainable:

• synchronous timeout;
• maximum number of injected cards;
• minimum retrieval score;
• minimum conclusion length for incomplete cards;
• backlog batch size; and
• index-rebuild concurrency.

When changing a parameter, update the expected behavior as well. Tightening the minimum score will increase the rate of empty injections. That is often a sign of a healthier system, not a regression.

At minimum, tests should verify that:

• segmentation does not truncate conclusions;
• the rule-based stage can mark incomplete cards when the model is unavailable;
• superseded cards are not recalled by default;
• sensitive strings do not enter card bodies or query logs;
• the index can be rebuilt after deletion from the original card bodies;
• the overflow queue replays each item only once after permissions are restored;
• each of the eleven gates can reject a candidate independently; and
• two contradictory cards are never injected together.

Three incidents deserve explicit entries in the runbook.

1. Renderer Truncation Produces a False Complete Card

A card may appear complete even though its conclusion was cut off. The response is to fail the conclusion-integrity check and keep the card out of the active set.

2. Cards Are Only Appended and Never Superseded

Retrieval begins to oscillate between conflicting conclusions. Enforce conflict checks for the same topic. If the system cannot resolve the conflict automatically, downgrade the cards to candidate status.

3. The Distillation Model Is Alive but Produces Nothing

Count format-validation failures and empty outputs explicitly. After a timeout, return the batch to the queue and raise an alert instead of retrying indefinitely on the hot path.

Implementation does not need to begin with vectors, models, and every container. The smallest useful loop is:

• read-only session logs;
• rule-based segmentation;
• body files as the source of truth;
• exact-field retrieval;
• injection of no more than two cards; and
• a failure queue.

Add embeddings and the model stage only after this loop is stable. The value of a memory system is not determined by how large its library becomes. It comes from ensuring that, when a similar problem appears again, the context contains only conclusions that are still valid, short enough to read, and traceable when they are wrong.