Why Large Context Windows Still Forget: Cross-Session Memory for AI Coding Assistants

Many teams have encountered a similar situation. The previous week, someone spent 20 minutes explaining three project rules to an AI coding assistant:

• Do not refactor the authentication module yet because legacy clients still depend on it.
• Start the local cache service before running tests.
• Run the static analysis script before committing.

The assistant followed those instructions correctly, and the task was completed. But when a new session started the following week, its first suggestion was a plan to “refactor the authentication module.”

This does not necessarily mean that the assistant has no memory capability. More often, the issue is that information a user mentions is not automatically information the system decides to retain long term. Whether something is remembered, what gets stored, when it is retrieved, and when it is removed are all controlled by policies.

Expanding the context window to millions of tokens only makes the “memory” larger. It does not turn the system into one with a reliable “hard drive.”

This article focuses on three questions:

1. Why do large context windows still fail to solve cross-session forgetting?
2. How do mainstream approaches store, retrieve, and delete memories?
3. What is the simplest practical approach for day-to-day development and self-hosted AI agents?

1. Why a 1M-Token Context Window Still Does Not Guarantee Memory

A context window answers the question, “How much information can the model see during one inference?” Cross-session memory answers a different question: “Can the system recover stable information during the next inference?”

These are not the same thing.

First, a context window is a session-level resource. When a session ends, the temporary context inside it does not usually become long-term knowledge automatically. Even if a single conversation can hold large amounts of code, logs, and discussion, a new session may still begin with none of that information loaded.

Second, larger windows create a greater need for filtering. Putting the entire history into every prompt introduces noise, conflicts, and cost. An old rule may no longer apply. A one-time decision may be mistaken for a permanent standard. A temporary debugging parameter may contaminate future tasks. The difficult part of a memory system is not how much it can store, but what it should store.

Third, memory requires lifecycle management. From creation to use, a memory item may need to pass through admission rules, confidence assessment, conflict resolution, expiration, cleanup, and audit tracking. Without these steps, remembering more can actually increase the system’s error rate.

In simple terms, a large context window is larger “working memory.” Cross-session memory is a “disk” with indexing. The two should work together rather than replace one another.

2. A Four-Layer Memory Model

Before discussing implementation, it helps to distinguish the major memory layers.

1. Human-authored rule files

These include project instructions, coding standards, prohibited changes, test commands, and release procedures.

Their advantages are stability, auditability, and version control. Their main drawback is that someone must maintain them.

2. Summaries of a single long conversation

A system can summarize the current session to control token usage. This primarily supports the current task and does not guarantee retention across sessions.

3. Assistant-managed notes

The system may extract notes automatically according to a policy, or the model may explicitly call a tool to write them. This layer sits between project rules and temporary context and requires admission and cleanup mechanisms.

4. An external memory database

Long-term memories can be stored in an independent system using vector retrieval, knowledge graphs, event logs, or other mechanisms. Relevant memories are then injected into the context when needed.

This approach supports sharing across applications and more complex retrieval, but it also introduces greater engineering complexity.

Many comparisons become misleading because they mix these layers together. Comparing a project rules file directly with an external memory database is like comparing project documentation with a search engine. Teams should first clarify the goal:

• Do they need stable project standards?
• Do they need task handoff across sessions?
• Do they need knowledge retrieval across projects?

Different goals call for different solutions.

3. Nine Main Approaches: Who Decides What Gets Written?

Based on public information and common engineering practices, memory strategies for AI coding assistants generally fall into nine categories.

1. Human-written rules with no automatic memory

The system reads instruction files in the repository, while people maintain all long-term rules.

Advantages: High control and easy review.
Disadvantages: Strongly dependent on team discipline.

2. Delayed extraction after a session ends

The system does not write memories immediately. After the session has been idle for a period of time, a background process summarizes the conversation into notes.

Advantages: Less noise.
Disadvantages: Lower timeliness and a risk of missing important context.

3. Immediate automatic extraction under policy control

During the conversation, the system determines which information is worth retaining and writes it according to predefined rules.

Advantages: A smoother user experience.
Disadvantages: Requires mechanisms for handling incorrect memories and conflicts.

4. Allowing the model to modify project rule files

The model can write stable agreements back to a project’s rule files.

Advantages: Transparent and version-controlled.
Disadvantages: Human review is essential; otherwise, the model may turn a one-time decision into a permanent standard.

5. Writing memory back to the code repository

Task handoffs, decisions, pending work, and risks are stored in repository documents.

Advantages: Closely connected to the codebase and easy to review.
Disadvantages: Can create documentation noise.

6. Retrieval and injection from an external memory store

Memories are stored independently and injected into the context through vector or structured retrieval when needed.

Advantages: Large capacity and cross-application access.
Disadvantages: Requires indexing, permissions, and cleanup policies.

7. Exposing note-taking as a tool

The model uses explicit tool calls to create, read, update, and delete memories.

Advantages: Memory operations are observable.
Disadvantages: The approach depends on the model calling the tools appropriately.

8. Long-conversation compression only

The system summarizes the current session without promising cross-session memory.

Advantages: Simple to implement.
Disadvantages: A new session may still forget everything.

9. Removing automatic memory

This approach emphasizes explicit rules and human maintenance while minimizing uncontrolled writes.

Advantages: Greater stability.
Disadvantages: Requires more effort from the team.

None of these approaches is universally superior. The central question is who decides what gets written to the notebook:

• A person writes it manually.
• The model records it during the task.
• A background process summarizes it later.

For production teams, the most reliable choice is often a hybrid model:

• People maintain stable project rules.
• The model writes task handoff information.
• Background processes handle summarization and cleanup.

4. Five Memory Architecture Patterns

Common memory architectures can be grouped into five categories: vector retrieval, relationship graphs, hierarchical summarization, event logs, and tool-based management.

Each emphasizes a different capability:

• Vector retrieval is useful for semantic recall, but it may retrieve content that is similar yet outdated.
• Relationship graphs are effective for representing entities and relationships, but they are expensive to build and maintain.
• Hierarchical summarization is useful for compressing long conversations, but summaries can lose details.
• Event logs support auditing and historical reconstruction, but retrieval and aggregation require additional design.
• Tool-based management exposes memory operations to the model, making the system flexible but dependent on the model’s tool-calling strategy.

When selecting an architecture, teams should not ask only, “Can it store information?” They should also ask:

• How are memories admitted?
• How are they retrieved?
• How do they expire?
• How are conflicts resolved?
• How are changes audited?

A memory system without deletion and conflict-merging capabilities will eventually become a warehouse of noise.

5. Misremembering Is Worse Than Forgetting

Memory writes should be governed by admission rules. A practical classification is:

1. Stable rules
2. Task facts
3. Temporary context

Stable rules should require human confirmation. Task facts can be extracted automatically, but their sources should be recorded. Temporary context should not enter long-term memory.

Retrieval should also be scoped. When injecting memory into a new session, the system should load project rules and the most recent handoff notes first, then retrieve external memories based on task relevance. It should not insert the entire history into the context at once.

Deletion requires a lifecycle policy. At minimum, the system should support expiration, conflict detection, and a manual revocation path.

For example, a rule saying “start the cache service before testing” should be able to expire after the test environment is redesigned. A temporary restriction saying “do not modify the authentication module” should be removable once legacy clients have been retired.

A simplified implementation might look like this:

on_session_end(task, transcript):
    stable = extract_stable_rule(transcript)
    if stable and needs_human_review(stable):
        propose_to_project_rules(stable)   # Propose a change; require approval
    handoff = summarize_task_state(transcript)
    write_handoff_note(handoff, ttl=14d)   # Give the handoff note an expiration time
    drop_temp_context(transcript)          # Do not retain temporary context long term

The important part is not the specific API. It is the policy:

• Stable rules require review.
• Task handoffs should have an expiration period.
• Temporary context should be discarded.

6. Most Teams Do Not Need a Vector Database on Day One

For most development teams, the simplest starting point is not a sophisticated memory database. It is two basic practices.

First, create a concise project rules file

A project rules file should ideally stay within roughly 200 lines and contain only stable information, such as:

• Module restrictions
• Test prerequisites
• Pre-commit checks
• Coding conventions
• Release procedures
• Common troubleshooting steps

The file should be stored in version control, and changes should go through review.

Second, create a task handoff note at the end of each task

Each completed task should record:

• What was completed
• Which files were changed
• Unresolved issues
• Recommended next steps
• Known risks

Handoff notes can include expiration times to prevent them from accumulating indefinitely.

Automatic summaries and retrieval can be added later as needed. As the team grows, project rules, handoff notes, and audit logs can be managed through a shared version-control system and logging service. Cross-project retrieval may justify introducing an external memory store later.

There is no need to add uncontrolled automatic writes simply to make the system appear more intelligent.

Practical implementation checklist

1. Put only stable agreements in the project rules file and keep it readable.
2. By default, have automatic memory produce suggestions rather than directly modify rules.
3. Require handoff notes to include task status, risks, and next steps.
4. Attach a source, timestamp, confidence level, and expiration policy to every long-term memory.
5. Periodically remove expired rules and resolve conflicting memories.
6. Require human review for important rule changes, with rollback capability.
7. When a new session starts, load the project rules file and the most recent handoff notes first.

Conclusion

The memory of an AI coding assistant is fundamentally an information governance problem.

• Large context windows address single-session capacity.
• Rule files preserve stable project standards.
• Handoff notes preserve task continuity.
• External memory stores support large-scale retrieval.

Confusing these layers leads to the misconception that a larger context window should automatically remember more across sessions.

A reliable design assigns different responsibilities to different components: people maintain stable rules, models create reversible task records, and background processes handle summarization and cleanup.

Remembering the right information—and forgetting what should no longer matter—is more important than simply remembering more.