Building a Memory System for AI Coding Assistants: Layered Design and Practical Implementation

The Problem Is Not the Model, but the Session Boundary

Anyone who has used an AI coding assistant for a while has probably experienced this loop: start a new session, spend five to ten minutes explaining the tech stack, directory conventions, where the last change left off, and which approaches were already rejected. By the time the background is established, a significant part of the context window is gone, leaving less room for the actual coding work.

This is not a model capability problem. It is the inevitable result of stateless sessions. Large language model inference does not retain any mutable state across requests. What people call “memory” is really just historical messages being stuffed back into the input. Once the context window is truncated or replaced by a new session, the information goes to zero.

Blaming the model for “bad memory” is not useful. The better approach is to treat this as a design constraint and add a layer of persistent state management around the model.

This can be broken down into three measurable losses:

1. Repeated injection cost: The same project background has to be explained again every day, consuming token budget and crowding out code context that is actually needed.
2. Progress discontinuity: Cross-day task state, unfinished technical debt, and the reasoning behind temporary tradeoffs are easy to lose. These are exactly the kinds of information that cannot be reliably inferred from the code itself.
3. Decision drift: Conclusions reached in a previous session, such as “do not introduce this direction for now” or “focus on stabilizing the core path,” are not automatically inherited. Humans are easily distracted by newly appearing opportunities, and an assistant without memory will not stop you.

So the goal is not simply to build a “larger context window.” It is to build an externalized, retrievable, and evolvable memory system.

Layered Memory Architecture: L0 to L4

Layering memory by time scale and change frequency is often what determines whether such a system is practical. If all memories are mixed together in one place, they eventually turn into a pile of notes that nobody reads and nobody dares to update.

L0 Short-Term Working Memory (Within a Session)

This is the context window of the current session. It handles immediate reasoning for the current task.

Its role is that of a volatile cache. It is not persisted. At the end of a session, it only produces summaries for outer layers.

Do not try to force long-term information into L0. That only dilutes the signal-to-noise ratio of the current task.

L1 Project Fact Layer (Semantic Memory, Weekly Changes)

This layer describes “what this project is.”

It includes:

• Tech stack and versions
• Directory responsibilities
• Build and test commands
• Required engineering conventions
• Confirmed architecture decisions

Its characteristics are that it is objective, verifiable, and slow-changing. It is best stored as plain text and put under version control.

L2 User Profile Layer (Preference Memory, Monthly Changes)

This layer describes “what kind of person I am and how I make decisions.”

It is often overlooked, but it is the dividing line between a “new colleague” and a “long-term partner.”

It should include at least three types of information:

• Collaboration preferences: Whether the user prefers direct conclusions or comparisons of multiple options, whether every change should be explained, and how dense code comments should be.
• Decision criteria: The standards used to judge whether something should be done. For example: “New capabilities must reinforce the current main path; otherwise, do not do them.”
• Risk tolerance: Acceptable temporary solutions, the upper limit of technical debt, and requirements for test coverage.

L3 Event Timeline Layer (Episodic Memory, Appended Daily)

This layer describes “what happened.”

It includes daily progress, options that were discussed, reasons why they were rejected, unfinished items, and blockers.

This layer is an append-only event stream. It supports retrospectives and task continuity.

L4 Retrieval Index Layer

This is the recall layer across L1 through L3.

Its job is to bring the most relevant memory fragments back into the session with as few tokens as possible at the start of each conversation. This is also the performance bottleneck of the whole system, which is discussed separately below.

Storage: Prefer Files Before Introducing a Vector Database

A common overdesign is to set up a vector database immediately. But the real pain point of a memory system in its early stage is that memories need to be readable, editable, and diffable, and file systems naturally provide these qualities.

A recommended directory layout is:

.memory/
├── project.md          # L1 project facts
├── profile.md          # L2 collaboration preferences and decision criteria
├── decisions/
│   ├── 20260801-storage-choice.md
│   └── 20260823-no-new-domain.md
├── journal/
│   ├── 2026-08-20.md
│   └── 2026-08-21.md
└── index.json          # L4 lightweight index: paths + titles + keywords

Each memory file can use frontmatter to declare metadata. This makes it easier to handle time decay and conflict resolution later:

---
type: decision          # fact | preference | decision | event
scope: core-service     # scope for multi-project isolation
confidence: 0.9         # lower for auto-extracted items than human-confirmed ones
created: 2026-08-23
updated: 2026-08-23
supersedes: 20260801-storage-choice
tags: [storage-choice, cost, capacity-planning]
---
# Conclusion
At the current stage, do not introduce a new storage component. Object storage plus local cache is sufficient to cover the read-to-write ratio.
# Background and Criteria
- Criterion: The operational cost of adding a new infrastructure component must be lower than the benefit it brings.
- Review condition: Single-table data volume exceeds the expected threshold, or read amplification significantly degrades P99 latency.

The supersedes field is often missing in homemade memory systems. Without it, old and new conclusions can both be retrieved, and the model may randomly pick one. In that case, the memory system becomes a source of confusion rather than help.

Write Mechanism: What to Remember, When to Remember It, and Who Records It

Most failures of a memory system happen on the write side. Poor retrieval is a user experience problem. Polluted writes are a correctness problem.

There are three recommended triggers for writing:

• Explicit instruction: The user says “remember this” or “do not mention this again.” This has the highest priority, and confidence should be set to 1.0.
• Automatic extraction at the end of a session: Before a session ends, ask the model to output a structured summary. Only four types of content should be kept: new facts, decisions and their criteria, unfinished items, and blockers.
• Periodic consolidation: Aggregate scattered journal events into L1/L2 entries, and archive outdated content.

For deduplication and compression, use an event stream plus periodic folding pattern instead of rewriting records in place:

def consolidate(journal_entries, existing_facts):
    # 1. Semantic clustering: group events by the same topic
    clusters = cluster_by_topic(journal_entries)
    for c in clusters:
        # 2. Extract stable conclusions and discard process details
        claim = extract_stable_claim(c)
        # 3. Compare with existing facts
        hit = find_similar(existing_facts, claim, threshold=0.85)
        if hit and conflicts(hit, claim):
            # New item wins on conflict, but keep a link to the old one
            archive(hit)
            existing_facts.append(with_supersedes(claim, hit))
        elif hit:
            existing_facts[hit].merge(claim)   # merge instead of duplicate append
        else:
            existing_facts.append(claim)

Three hard rules should be followed:

• New information wins, but the old information is not destroyed: Archive it instead of deleting it, and keep the supersedes chain. This makes it possible to answer the question “Why did we change our mind?”
• Distinguish facts from inferences: Auto-extracted entries should have discounted confidence. During retrieval, human-confirmed entries should be trusted first.
• Sensitive information must never enter memory: Credentials, keys, internal addresses, and customer data should be blocked by rules or a classification model before writing. Once memory files are committed to version control, the exposure surface is much larger than expected.

Retrieval Mechanism: Progressive Loading, Not Full Injection

L1, L2, and L3 can quickly grow to tens of thousands of characters. Stuffing all of them into the context is neither economical nor effective. Irrelevant information reduces the model’s sensitivity to critical constraints.

A practical approach is three-level progressive disclosure:

1. Resident layer (about 500 to 800 tokens): Load only the L1 project identity card and L2 decision criteria. These two parts are small and reused frequently, so they must be online in every session.
2. On-demand layer: The model actively reads specific files when the current task requires them. The index.json file only stores paths, titles, and keywords, so its size can be negligible.
3. Retrieval layer: The journal for the current day is loaded by default. Earlier events are recalled through keyword matching or vector similarity to retrieve the top-k results.

For recall strategy, hybrid retrieval is better than single vector retrieval. Engineering memory often contains proper nouns, function names, paths, and error codes. These are exactly where lexical matching is strong and dense vectors are weak. A practical approach is to perform lexical recall first, then rerank candidates semantically:

def recall(query, k=5, budget=2000):
    lex = bm25_search(index, query, topn=20)      # lexical recall, strong on identifiers
    dense = vector_search(index, query, topn=20)  # semantic recall, strong on intent
    merged = rerank(lex + dense, query)[:k]
    selected, used = [], 0
    for doc in sorted(merged, key=lambda d: -d.score * d.recency_weight):
        if used + doc.tokens > budget:            # strict token budget
            break
        selected.append(doc); used += doc.tokens
    return render(selected)

From an engineering perspective, on cloud platforms you can store memory files in object storage for multi-device synchronization, and use logging services to observe the recall hit rate and token usage of each session. This makes it easier to continuously tune recall quality.

Observed Benefits in Three Scenarios

Scenario 1: Cross-Session Task Continuity

At the start of a session, ask directly: “What should I do today?” The system pulls unfinished items from the recent event stream: which draft still needs revision, which path has not been fully tested, and where the previous discussion stopped.

The value is not only saving time. Human short-term memory is unreliable. Missed technical debt is often recovered only through external records.

The same applies to code. Unresolved warnings, half-finished refactors, and test cases that still need to be added can all be tracked as explicit to-do items over time.

Scenario 2: Retrospectives Based on Records

By aggregating daily journals, you can draw conclusions that are hard to reach through intuition alone: how many days a task stalled, the actual completion rate of planned items, the frequency of context switching, and which types of tasks are repeatedly reworked.

The value of this kind of conclusion lies in the fact that it uses data to speak. You cannot refute it by saying “I feel like I have been working hard lately.”

Without records, a retrospective is just self-satisfaction.

Scenario 3: Decision Guardrails

The most valuable memory is not facts, but criteria.

Once a decision criterion is persisted, the assistant can review new proposals in reverse:

User: Everyone seems to be going global recently. Should I start a new project to try it?

Assistant: According to the criterion you established on August 23 — new investment must reinforce the current main path — this project does not meet the condition. It cannot reuse existing user and content assets, and it requires an independent cold start. If you really want to explore it, start with a low-cost validation and do not take up mainline schedule space.

The moment of pause is typical: the rule was set by yourself, but you forgot it. Humans are unreliable memory stores, and a partner with good memory can pull you back from a drifted path.

The key implementation point is to write decision criteria as decidable conditions, not slogans like “stay focused.” Only when a criterion is decidable can the model execute it.

Pitfall Checklist

• Memory pollution amplifies itself: Once an incorrect “fact” is written, it may be cited as a premise in subsequent sessions. Always keep source annotations, and allow one-click tracing and rollback.
• Forgetting is harder than remembering: Set TTLs for event-type memory, such as 90 days. Apply confidence decay to fact-type memory. Archive old entries periodically. A system that never forgets will eventually fail because of retrieval noise.
• Multi-project environments must be isolated: Use the scope field to prevent conventions from one project from polluting another. Personal preferences can be global, but project facts must be local.
• Memory files should be under version control: Memory is a team asset. It should be diffable, reviewable, and rollbackable. One bad merge is enough to make you understand why this is necessary.
• Do not forget human confirmation: The accuracy ceiling of fully automatic writes is not high. For key entries that affect decisions, add a human confirmation step. The benefit is far greater than the cost.

Conclusion

The essence of this system is to compensate for the model’s missing cross-session state with an externalized, layered, retrievable, and forgettable storage structure.

Its value is not “remembering more.” It is two quantifiable benefits:

1. It reduces the cold-start cost of each session from ten minutes to one sentence.
2. It shifts decision consistency from personal willpower to an executable set of criteria.

A practical implementation order is from simple to complex. Start with L1 plus L3, meaning project facts plus daily events, and run it for two weeks. Then add L2 decision criteria to enable guardrails. Only after that should you introduce vector recall and automatic consolidation. Introducing a complex retrieval layer too early usually just means spending time maintaining an index that you yourself do not read.