AI Coding Agent Memory Systems: Layered Architecture and Practical Implementation

Memory engineering is the upper bound of AI coding agent capability.

1. Why Memory Engineering Determines the Capability Ceiling of AI Coding Agents

1.1 The “amnesia” of stateless models

Large language models are stateless at inference: each call is an independent forward computation, and parameters do not change during inference. This means that after closing the terminal, restarting an IDE plugin, or switching to a new task, the model knows nothing about “which file was changed last time”, “why a previous approach was abandoned”, or “which directories must not be touched”.

Therefore, an agent’s so-called memory has only one physical implementation path: before each call, reassemble relevant information into the context window. All memory products, memory frameworks, and memory middleware essentially do the same thing: within a limited token budget, decide which information should appear, in what form, and when it should disappear.

1.2 Context budget as a form of scarce-resource economics

Context window length is increasing, but effective attention does not scale linearly. Three factors jointly determine that filling the window is not the same as using it well:

• Recall quality of information in the middle degrades, commonly called “lost in the middle”;
• Instruction-following degrades in long contexts;
• Cost and latency rise as context length increases.

A reasonable engineering practice is to manage context as a budget:

• Static, stable, behavioral content → resident
• Project-level norms and decisions → loaded at startup
• Concrete code facts, such as function signatures, call sites, and configuration values → retrieved on demand by tools
• Temporary constraints within a session → maintained in a rolling window

1.3 Three critical problems memory systems must solve

Problem 1: Cross-session consistency breaks. Once a session ends, the agent forgets, forcing users to repeat instructions such as “use pnpm, not npm” or “do not touch the generated directory”. The cost of repeated instruction is far higher than the cost of writing it to memory once.

Problem 2: Constraint drift. A technical choice recorded months ago may already have been overturned by refactoring, but the agent still follows the old plan, producing code that is inconsistent with the current architecture. This is often discovered only dozens of turns later.

Problem 3: Information corruption. The memory store may contain both correct and stale information, or two contradictory versions of the same fact. Faced with conflicts, the model may randomly choose one, making behavior unpredictable. This is more dangerous than having no memory at all, because no memory is at least explainable.

2. Layered Memory Architecture: Partition by Change Frequency, Not by Content Type

A mature memory system’s first decision is not “what format to store” but how to partition information by lifecycle and change frequency, then apply different loading, eviction, and write policies to each layer.

Layer	Name	Physical Location	Typical Content	Loading Strategy	Eviction Strategy
L0	System-resident layer	Hard-coded inside the agent	Behavior rules, tool schemas, safety constraints, runtime environment information	Fully resident, not subject to budget trimming	Updated with version releases
L1	Project configuration layer	Memory files in the repository, potentially multi-level nested	Technology stack, coding standards, architecture decisions, directory conventions, prohibitions	Fully loaded at session start, with subdirectory-level activation when needed	Human-maintained and version-controlled
L2	Team-shared layer	Organization-level knowledge base / retrieval index	Shared component usage, historical incident retrospectives, internal conventions	Retrieved by semantic search, injected as Top-K	Periodic review and expiration
L3	Session working layer	Runtime context	Current task goal, modified files, real-time user constraints, pending items	Rolling window, summarized and compressed when limits are exceeded	Discarded when the session ends
L4	Archive offline layer	Object storage / logging service	Complete session traces, tool call records, failure cases	Not included in online context; used only for offline distillation and retrieval	Retention policy

The core design philosophy can be summarized in one sentence: the lower the layer, the slower it changes and the more stable it is, so it should be resident; the higher the layer, the faster it changes and the more easily it decays, so it should be retrieved or discarded.

For L0, one practical consensus is that constraints involving dangerous operations should be hard-coded in locations that cannot be tampered with by user files or external content. This includes prohibiting irreversible deletions and force pushes, prohibiting modifications to generated code, and prohibiting migration scripts in production. This is a safety baseline and should not be carried by editable configuration files.

3. Five Key Mechanisms

3.1 Why Markdown Instead of Structured Configuration

Project-level memory should be written in Markdown with front matter rather than pure YAML/JSON:

• LLMs typically follow natural-language instructions better than they interpret configuration fields;
• Decisions need to carry rationale and cost, which free text expresses well and structured fields struggle to contain;
• Humans must read, review, and integrate it into code review; Markdown has a friendly diff experience;
• The front matter retains a small number of structured fields for programmatic processing, such as scope, version, and expiration.

Example format:

---
scope: project                 # project | team | personal
version: 12
updated_at: 2026-06-18
ttl_days: 90                   # mark for review once expired
supersedes: ["ADR-003"]        # explicitly declare supersession instead of deleting directly
owners: [platform-team]
---
## Tech Stack Constraints
- Use pnpm for dependency management; do not introduce package-lock.json.
- All database access must go through repository interfaces; do not concatenate SQL in the controller layer.
## Architecture Decisions (with rationale)
- ADR-011: Switch the write path to event-driven. Rationale: peak shaving and replayability are required; cost: the pipeline becomes eventually consistent, and the frontend must show a "processing" state.
## Prohibitions
- The generated/ directory is auto-generated output; any modification will be overwritten by the next generation. Modify schema definitions and templates instead.

3.2 Layered Loading: Give Each Layer Its Own Budget Quota

def build_context(session, total_budget=48_000):
    parts = []
    # L0 resident; not subject to trimming
    parts.append(load_system_prompt(session.env))
    # L1 loaded at startup; cascade by directory hierarchy (root -> subdirectory), rules closer to the file take precedence
    parts.append(load_project_memory(session.root, cascade=True))
    # L3 session working layer; summarize when exceeding the limit instead of truncating directly
    parts.append(compact_turns(session.turns, limit=int(total_budget * 0.45)))
    # L2 + retrieval layer use the remaining budget
    remaining = total_budget - sum(len(p) for p in parts)
    queries = extract_queries(session.recent_turns(3), session.active_files)
    parts.append(budgeted_retrieve(queries, remaining=remaining))
    return merge(parts)

Cascading loading is key to solving cross-session consistency breaks: memory files inside a subdirectory apply only to that subtree, parent rules are inherited by default, and for entries with the same name, the nearest one wins. This avoids wasting the context window by putting all repository conventions into one file, and it allows modular coding conventions to evolve with the directory structure.

3.3 Hybrid Recall and Budget Trimming

Memory retrieval should not rely only on vector similarity. A large amount of critical project information is exact-match type, such as symbol names, configuration keys, paths, and error codes, where keyword search performs better. A common engineering approach is hybrid retrieval:

def budgeted_retrieve(queries, remaining, top_k=20):
    if remaining < 4_000:                      # inject less rather than more when budget is low
        return []
    candidates = hybrid_search(queries, top_k=100)   # keyword + vector fusion
    ranked = rerank(candidates, queries)[:top_k]
    # Memory retrieval should not exceed 30%–35% of total budget; leave space for code and tool output
    quota = int(remaining * 0.35)
    kept, used = [], 0
    for item in ranked:
        if item.stale or item.overridden_by_newer():   # drop stale or overridden memory directly
            continue
        cost = len(item.render()) + 64
        if used + cost > quota:
            break
        kept.append(item)
        used += cost
    return render(kept)

3.4 Write-back: What Qualifies as Memory

If memory accepts everything, it will inevitably decay within weeks. Writes must have admission criteria and should be completed in a separate offline distillation step:

Extract information that is worth retaining long-term from this session, and strictly follow these rules:
1. Admission criteria (all must be satisfied):
   a) Still valid in later sessions;
   b) Cannot be directly read from code or configuration (readable information belongs to retrieval; do not write it);
   c) Affects the specific behavior of the next task.
2. Output fields: conclusion / rationale / evidence (file path + symbol name) / confidence / recommended expiration.
3. When conflicting with existing memory, output a supersedes relation instead of directly overwriting the old entry.
4. If no entry satisfies 1a–1c, output an empty list. Do not generate content just to produce something.
5. If the user explicitly says "this is only a temporary exception", mark it with scope=session and do not write it to long-term memory.

These rules govern memory in three categories, matching common memory types:

• Semantic memory: facts and norms, such as technology stack, dependency constraints, and naming conventions
• Procedural memory: operating procedures, such as local startup steps, release processes, and debugging patterns
• Episodic memory: historical decisions and failure cases, such as why an approach was abandoned or the root cause of an incident

Among these, failure cases are the most valuable but also the most perishable. They must carry time and context conditions, such as “unusable under version X and scenario Y”, otherwise they are easily misused.

3.5 Expiration, Conflict Resolution, and Forgetting

Forgetting mechanisms are as important as write mechanisms. They have three levels of handling:

1. Soft expiration: Items that exceed TTL or have not been recalled for N consecutive times are marked stale and downweighted; they are retained but rarely enter context.
2. Merge deduplication: When multiple items express the same conclusion, merge them into one, keep the version with the strongest evidence, and attach a citation count.
3. Hard deletion: Truly remove items that have been superseded, manually marked incorrect, or have exceeded the retention period.

Use an explainable composite score for ranking rather than similarity or time alone:

score = (
    0.45 * relevance          # retrieval relevance
  + 0.20 * normalized_hits    # historical hit rate / citation count
  + 0.15 * recency_decay      # exponential time decay; half-life depends on layer
  + 0.10 * confidence         # confidence at write time
  - 0.30 * staleness_penalty  # expired, questioned, or unreliable source
)

Different layers commonly use different half-lives: project norms can last months, temporary session constraints have a half-life measured in hours, and incident cases should “retain conditions long-term and review conclusions periodically”.

4. Engineering Implementation: Combining with General Cloud Capabilities

Memory systems have both online and offline paths. Engineering implementation is usually a combination of general cloud capabilities:

• Object storage: Store version snapshots of memory files and complete session traces, support replay by version, and make it easier to diagnose why behavior differed in a given run.
• Message queue: Asynchronously receive session-end events, trigger offline distillation, index updates, and deduplication/merging, avoiding blocking interaction.
• Vector retrieval service: Provide semantic recall and form hybrid retrieval with keyword indexes.
• Logging and monitoring/alerting: Record memory items injected into each context and their hit status; set alerts for sudden increases in memory injection volume or rising constraint violation rates.
• Configuration center: Support canary release and rapid rollback of memory policies, such as quota ratios, TTL, and weights.

The team-shared layer also needs permission isolation: personal preferences and team norms must be stored in separate domains; personal memory must not be pushed to the shared layer; writes to the shared layer should go through a review process aligned with existing code review and change management.

5. How to Evaluate a Memory System

The effectiveness of a memory system cannot be judged by whether it “feels like it understands me better”. It needs quantifiable metrics:

Metric	Definition	Optimization Direction
First-attempt accuracy	Proportion of outputs that meet project norms without user correction	Improve recall precision and completeness of norm injection
User corrections per task	Number of times the user corrects the agent within a single task	Reduce constraint drift
Constraint violation rate	Proportion of outputs that hit prohibited items	Strengthen L0/L1 resident constraints
Cross-session continuity	Number of rounds needed to restore information when continuing a previous session task	Optimize session summaries and working-layer compression
Context ratio	Memory tokens / total context tokens	Keep within 25%–35%
Stale hit ratio	Proportion of recalled items that are expired or overridden	Test the effectiveness of the forgetting mechanism
Cost and latency per task	Tokens consumed and response latency for completing one task	Balance budget allocation

For implementation, build a regression evaluation set: select 50–100 real historical tasks, fix evaluation scripts, rerun them after each adjustment of memory policy, such as quota, weights, or TTL, and observe changes in the above metrics. Tuning without an evaluation set is essentially a random walk based on intuition.

6. Common Anti-Patterns

1. All-resident memory: Writing a giant memory file that summarizes the entire codebase and keeping it resident. It may work short-term but quickly consumes space needed for code and tool output, and triggers attention decay.
2. Write-only, no deletion: No expiration or deduplication mechanism; contradictory versions appear after three months.
3. Using memory instead of retrieval: Putting frequently changing facts such as function signatures, line numbers, and configuration values into memory. The correct boundary is: memory carries behavioral priors; retrieval carries the factual source.
4. Treating summaries as memory: Session summaries lose key constraint details and are hard to trace. Summaries should retain “evidence + conditions”, not be compressed into conclusions.
5. Cross-project contamination: Mixing global personal memory with project norms causes conventions from project A to leak into project B.
6. No explainability: It is impossible to trace which memory was injected and why, making problems hard to locate. Every memory injection must carry a source identifier for auditing and revocation.

7. Implementation Checklist

• Is memory layered by change frequency, with independent quotas and eviction policies for each layer?
• Are dangerous-operation constraints placed in a resident layer that cannot be tampered with by external content?
• Are project memories under version control and included in code review?
• Does write admission enforce conditions: still valid long-term, not directly readable from code, and affecting behavior?
• Is an explicit supersession relation, supersedes, implemented instead of deleting history directly?
• Are TTL, soft-expiration marking, merge deduplication, and hard deletion implemented as forgetting mechanisms?
• Does memory retrieval use hybrid recall plus reranking and remain under hard budget constraints?
• Can every injection be traced to a source, audited, and revoked?
• Is a regression evaluation set and a core metrics dashboard established?

The essence of memory engineering is not making an agent remember more, but making it recall the right thing at the right time with the right priority, and forget things that are no longer valid. Agents built on these three actions deserve the positioning of long-term collaborators.