Why AI Coding Assistants Forget: Designing and Managing Cross-Session Memory

1. Why Do Rules Agreed in One Conversation Disappear in the Next?

Imagine that you are maintaining a legacy system and tell an AI coding assistant:

“For this task, fix only the login issue. Do not refactor the authentication module or change any public interfaces.”

During the current conversation, the assistant understands the instruction correctly and keeps its changes within scope. A few days later, however, you open a new conversation and the assistant proposes redesigning the authentication flow.

This does not necessarily mean that the system has no memory capability. A failure may have occurred at any point in the cross-session memory chain:

• The instruction existed only in the previous conversation and was never written to persistent storage.
• The system saved a session summary but omitted the critical constraint against refactoring.
• The constraint was saved, but the new session did not load records for the relevant project.
• Retrieval found the relevant note, but too much background information pushed the important constraint out of the effective context.
• An old note conflicted with a new instruction, and the system failed to resolve their priorities correctly.

Therefore, determining whether an assistant “remembers” something requires more than checking whether a memory file exists. The complete chain must be examined:

Identify information → decide whether to save it → persist it → retrieve it when needed → inject it into context → execute correctly → update or delete it.

If any step fails, the user may simply see the result as: “It forgot again.”

2. Large Context Windows Solve Capacity; Long-Term Memory Solves Continuity

2.1 A Context Window Is Not Persistent Storage

A context window is the amount of information a model can receive during a single inference. Expanding the window allows the system to include more code, conversation history, and documentation. It does not automatically make that information available in the next session.

Even when a system supports very long inputs, a new session still needs to reload historical content. Deciding whether to load information, what to load, and how much to load is an application-level memory management problem. Context-window size alone cannot guarantee this behavior.

Using a computer analogy, the context is closer to working memory, while long-term memory is closer to persistent storage. The analogy has limits, however. A model does not read context by exact memory address. Even when information has been included, the model may not always use it consistently.

2.2 Replaying the Entire History Is Not a Reliable Solution

Sending the complete conversation history every time may seem safest, but it creates three problems.

First, cost and latency increase with the input size. A small task usually does not require carrying a large amount of old conversation into every request.

Second, historical information can conflict. A technical approach that was acceptable last month may now be deprecated. A temporary workaround used during an incident should not automatically become a permanent project rule.

Third, noise can hide important constraints. In hundreds of discussion messages, a single sentence such as “Do not modify public interfaces” may be less visible than a separately listed project rule.

The goal of long-term memory is therefore not to preserve the maximum amount of text. It is to provide sufficiently accurate and appropriately scoped information at the right time.

3. Distinguish Four Mechanisms Before Building a Memory System

Different approaches are often grouped under the word “memory,” but they solve different problems.

3.1 Manually Maintained Rule Files: Store Stable Constraints

Rule files are appropriate for long-term project requirements that must be followed explicitly, such as:

• Directories that may be modified and modules that must not be touched;
• API compatibility requirements;
• Testing, build, and acceptance procedures;
• Security boundaries and requirements for handling sensitive data;
• Code style and dependency-management principles.

Their main advantages are reviewability and traceability. They can be reviewed as code or project artifacts, helping teams establish shared expectations.

Their limitation is that someone must maintain them. In addition, the existence of a file does not mean that the assistant will automatically read it. A clear loading process is still required.

Rule files should focus on constraints and operational entry points rather than copying the entire project documentation set. Keeping core rules to roughly 200 lines or fewer can be a useful maintenance starting point, and the rules can be split as complexity grows. This is a maintainability guideline, not a guarantee of accuracy.

3.2 Single-Session Compression: Continue the Current Task

As a conversation becomes longer, a system may compress earlier content into a summary that preserves the task objective, completed work, and unresolved issues.

This mainly reduces context pressure during a single long-running task. Unless the summary is persisted and reloaded in later sessions, it does not constitute cross-session memory.

Compression is also a lossy transformation. If a summary retains only “the login issue is being fixed” but drops “do not refactor the authentication module,” later actions may violate the original requirement.

A useful summary should preserve at least:

• Prohibited actions;
• Acceptance criteria;
• Unresolved questions;
• Unverified assumptions;
• Known risks and remaining failures.

3.3 Automatic Notes: Turn Experience into Candidate Memory

Automatic notes are created when a system extracts information from conversations or task results. They may be generated when a user explicitly requests it or when a task reaches a natural stopping point.

This reduces the cost of manual organization, but it introduces a judgment problem: which information is worth preserving?

For example, “Do not run the full test suite this time” may be an emergency decision. It should not automatically become “This project does not require the full test suite.” Likewise, “I want a brief explanation today” does not necessarily mean that the user always prefers short answers.

Automatic notes are therefore better treated as candidate knowledge than as automatically trusted rules.

3.4 External Memory Storage: Support Larger-Scale Management

When a system needs to manage many records across projects, roles, or tasks, memory can be stored in an independent service or repository. Keyword search, structured filtering, and semantic retrieval can then be used to find relevant records.

The value of external storage is not merely that it can store more data. It can also support:

• Separating information by user, project, and permission scope;
• Versioning, validity periods, and deletion states;
• Tracking sources and usage;
• Reusing confirmed knowledge across different task types.

External storage also adds costs, including access control, index maintenance, data governance, and operations. A small project may not need a complex retrieval system.

4. Memory Writes: Separate Facts, Preferences, Constraints, and Temporary State

4.1 Different Information Requires Different Storage Policies

Candidate memories can initially be divided into four categories.

Facts
For example: “The current service uses a layered architecture.” The record should include the applicable version or observation time so that an architectural change does not leave the old description looking like the current state.

Preferences
For example: “The user prefers to see the conclusion first.” These should normally be scoped to the relevant user and remain overridable.

Constraints
For example: “Public interfaces must not be modified.” The record should identify its source, scope, and whether it has been confirmed.

Temporary state
For example: “A particular test is still failing.” This is usually better stored in a task handoff record. It should be reviewed or removed when the task ends.

It is also important to distinguish between an assistant’s inference and a user-confirmed instruction. Repeated behavior may provide evidence for a possible preference, but it should not be promoted directly to a team-wide standard.

4.2 A Single Occurrence Is Not Enough to Justify Long-Term Storage

The following questions can help determine whether information belongs in long-term memory:

• Is it likely to be reused in future tasks?
• Is its scope sufficiently clear?
• Is it a confirmed conclusion or an unverified assumption?
• Does it contain sensitive information?
• If it becomes invalid, can the system detect and revoke it?

For security requirements, architectural decisions, and team-level constraints, a safer approach is often to generate a candidate record automatically and require an owner to confirm it before publication.

4.3 Store the Source, Not Just a Summary

The following is an illustrative memory record. Its fields demonstrate a design approach rather than an existing system interface:

Record ID: Project-Constraint-017
Content: Do not modify public API signatures during this maintenance cycle
Category: Constraint
Scope: Current project, current maintenance cycle
Source: Confirmed maintenance task instructions
Status: Confirmed
Review Condition: End of the maintenance cycle or approval of an API upgrade
Sensitivity: Internal

The source and scope allow the system to explain why the instruction is being applied. The review condition provides a basis for future updates.

5. Memory Retrieval: Finding Similar Text Is Not Enough

5.1 Restrict Access Before Calculating Relevance

Retrieval must satisfy permission requirements before semantic relevance is considered.

If a system first searches all records for the most similar content and filters permissions afterward, unauthorized information may be exposed during intermediate processing. A safer design restricts the search at the entry point by user, project, organization, and visibility scope.

High similarity can never justify crossing a permission boundary.

5.2 Use Separate Paths for Deterministic Rules and Experience Notes

Project rules generally require deterministic loading. When a task starts in a project, the system should load the current valid rule version rather than relying on semantic search to find it.

Historical experience can be retrieved on demand. For example, login incident records may be loaded when investigating a login failure, but they do not need to accompany every task involving page styling.

This produces a practical hierarchy:

• Stable rules: Load when the task starts;
• Task state: Load when work is resumed;
• Historical experience: Retrieve when relevant;
• Expired records: Keep out of the working context by default.

5.3 Conflict Resolution Cannot Depend Only on Which Record Is Newer

A newer record may describe a temporary experiment, while an older record may still represent the official standard. Conflict resolution should consider several factors:

• Authorization level;
• Scope;
• Confirmation status;
• Validity period;
• Source.

When memory materially conflicts with the current task requirements, the system should not silently choose one side. It should identify the conflict and request confirmation. Long-term memory must also never override system-level security requirements or access permissions.

6. Updates and Deletion: Prevent Incorrect Memory from Spreading

6.1 Merge Similar Records and Review Contradictions

Saving the same preference repeatedly can give it disproportionate weight during retrieval. Similar records can be merged while retaining their sources and update times.

Conflicting records should be marked for review rather than overwritten automatically.

The system should also avoid circular self-confirmation. For example, an assistant may generate an unconfirmed note, later treat its own response as new evidence, and eventually conclude that the original claim has been independently verified several times.

6.2 Deletion Must Cover Every Readable Copy

If the original record is deleted but the information remains in a summary, index, cache, or derived note, it may still be retrieved.

A memory system should therefore maintain relationships between source records and derived copies. When a source is updated or deleted, related copies should be handled accordingly.

Backup retention also requires explicit data-governance rules. “The record is no longer visible in the interface” should not be treated as equivalent to complete deletion.

6.3 Convenience Should Not Come at the Cost of Sensitive Data

Credentials, personal information, customer data, and internal keys should not be written automatically into general-purpose memory simply because they might be useful later.

Instead, the system can store the compliant access path and operational requirement without storing the sensitive value itself. For example, the following is safer:

“Before deployment, obtain a temporary credential through the authorization workflow.”

That is preferable to storing the credential itself.

7. Start with a Lightweight Three-Layer Approach

Layer 1: Project Rules

Create a short, reviewable rules document containing:

• Project boundaries and prohibited actions;
• Common validation procedures;
• Change-acceptance requirements;
• Locations of detailed documentation;
• The rule owner and last-updated time.

When loading the rules, the assistant should identify which entries constrain the current task instead of merely claiming that it has “read the rules.”

Layer 2: Task Handoffs

At the end of each task, generate a handoff record:

Task objective:
Completed work:
Verified results:
Unverified assumptions:
Current failures and known risks:
Prohibited actions for this task:
Recommended next steps:
Files or change scope:

“Verified results” and “unverified assumptions” must remain separate. Otherwise, the next session may treat an investigation direction as an established fact.

Layer 3: Confirmed Experience Notes

Reusable experience should be recorded using the structure:

Applicable conditions → Recommended action → Verification method

Before being added to formal memory, the note should be reviewed.

For example, instead of storing:

“Clear the cache when login fails.”

A more reliable note would be:

“When a particular class of symptoms appears after a configuration change, check cache consistency. Before clearing the cache, confirm the impact scope and use the specified tests to verify recovery.”

This record is longer, but it significantly reduces mechanical reuse outside the original conditions.

8. When Building a Custom System, Use Evaluation to Justify Complexity

Once project rules and task handoffs can no longer provide adequate retrieval efficiency, permission isolation, or lifecycle management, introducing an independent memory service becomes more reasonable.

A general processing pipeline could be:

Extract candidate information → identify sensitive content → classify and deduplicate → check for conflicts → request human confirmation when necessary → store → retrieve according to permissions → inject into task context → record usage results.

Evaluation should not focus only on how many records were successfully saved. It should also measure whether:

• Rules that should be remembered are correctly followed in new sessions;
• Temporary instructions are incorrectly made permanent;
• Expired records continue to affect decisions;
• Deleted information can still be retrieved;
• Information is accidentally shared between projects;
• Adding memory reduces task success rates or increases irrelevant context.

A fixed set of scenarios can make these tests concrete. For example, first instruct the assistant not to refactor a module, then start a new session with a similar task and observe whether the constraint is followed. Next, revoke the constraint and test whether the system updates its behavior correctly.

This type of evaluation is more meaningful than simply checking whether several new files have appeared in a memory directory.

Conclusion: Reliable Memory Is Governable, Not Unlimited

Cross-session continuity in an AI coding assistant does not depend on how many chat messages it can retain. It depends on whether the system can correctly distinguish long-term rules, temporary state, and unverified inferences.

For most everyday development work, establishing project rules, task handoffs, and reviewed experience notes is more practical than immediately building a complex database. More advanced storage and retrieval mechanisms should be introduced only when scale, permissions, or retrieval requirements genuinely demand them.

A trustworthy memory system should be able to answer five questions:

1. Where did this information come from?
2. Where does it apply?
3. Why is it being used?
4. When should it be updated?
5. How can it be fully revoked?