AI Coding Assistant Context Memory: Why It Forgets and How to Configure It

1. The problem: It does not have poor memory. It has no persistent memory.

Consider a scenario familiar to almost every developer.

You agree with an AI coding assistant that “all amounts must be represented as integers.” After 30 turns, it generates code that handles prices with floating-point numbers. You correct it, it apologizes and fixes the code, and a few turns later the same mistake appears again.

The usual reaction is “the model has poor memory.” That diagnosis is engineering-wrong: it has no memory, only a field of view.

Mainstream large language models are designed to be stateless. When a session ends, everything it learned about your project structure, coding preferences, and agreed conventions resets to zero. When you start a new session, the model is effectively a stranger to your project again. Even within the same session, what looks like memory is not memory in the traditional sense. It is the context window. Your messages, the model’s replies, and the file contents it has read are all placed as tokens in the same sequence. If something is visible in the window, the model knows it. If something has been pushed out of the window, the model genuinely does not know it; it is not refusing to recall it.

In one sentence: the context window is the model’s field of view, not its long-term storage. Treating a very long prompt as the solution is like expecting someone to remember a book after a single glance.

2. Deconstructing the three causes of apparent amnesia

To solve the problem, first identify the cause. In practice, apparent memory failures usually fall into three categories, each with different mechanisms and remedies.

2.1 Context saturation and mid-context attention decay

A single session’s context budget is often on the order of hundreds of thousands to millions of tokens. That sounds generous, but a real development task can consume most of it: pasted error logs, multiple source files, tool-call results, and several rounds of design discussion. Suppose a refactoring task consumes 80,000 tokens. A rule you stated in turn five will gradually drift toward the middle of the context sequence as the conversation continues.

There is an easily overlooked mechanism: large models do not distribute attention evenly across very long contexts. Multiple research papers and engineering experiences have observed a similar pattern: content near the beginning and end of the sequence receives more stable attention, while the effective weight of content in the middle drops significantly. This is commonly described as a position bias in long contexts.

That creates a chain:

Rule is written → context keeps growing → rule drifts into the middle → attention weight drops → the model effectively ignores the constraint.

The key conclusion is this: this is not a moral problem of the model; it is a physical characteristic of the attention mechanism. Therefore, the solution is not “emphasize it again.” It is to place the constraint in a more stable position, closer to the decision point.

2.2 New sessions are naturally amnesiac

Starting a new session clears the context. This is the most complete form of amnesia: not fuzzy memory, but no memory at all.

A common beginner misconception is: “If I configure a project rules file, I am done.” It is necessary to distinguish two types of knowledge:

• Project specification knowledge: coding style, directory conventions, technology-stack boundaries, and prohibited practices. This kind of knowledge changes slowly, is reusable, and is suitable for fixing into files.
• Decision-history knowledge: why a particular storage solution was chosen, why an interface is designed a certain way, or why an alternative was rejected last time. This kind of knowledge is temporal, context-dependent, and may become outdated as the project evolves.

Rules files solve the first type, but not the second. If you write “use an in-memory cache database for session management” into a rules file, the model in a new session will know the conclusion, but it will not know why. Then, when it encounters a requirement such as “session data must not be lost after restart,” it may confidently overturn the previous decision.

2.3 Rule drift and factual disconnection

The third cause is more subtle: the configuration exists, but it is outdated.

Suppose requirements evolve for three months, but the directory description in the rules file still reflects an old structure. After a module is split, a call-chain constraint in the rules file may point to a layer that no longer exists. The documentation may say “synchronous calls of a certain type are prohibited,” while the codebase already contains 20 exceptions. In this situation, the assistant is reading an incorrect world model. It follows the rules, and therefore produces worse results.

A rule file is not a one-time asset. It is code that must be maintained. It can rot just like code.

3. Modeling: Manage “memory” in four layers

To solve the problem systematically, start by separating memory into layers. Using “frequency of change × reuse scope” as the dividing criteria, project memory can be split into four layers:

Layer	Content	Carrier	Lifecycle
L0 Pretrained knowledge	Language, frameworks, general algorithms, and common engineering knowledge	Model parameters	Changes only when the model generation changes
L1 Project static specifications	Technology stack, directory conventions, coding style, hard prohibitions	Rules files in the version repository	Maintained as the code evolves
L2 Session task context	Current requirement, files read, completed steps	Context window	Disappears when the session ends
L3 Decision and state log	Architecture rationale, rejected options, in-progress plans	Documents in the version repository plus an index	Retained long term

You cannot change L0. The real engineering work is concentrated in L1 and L3: write L1 as executable constraints, move L3 out of the model’s head and into the version repository, and then inject L3 into L2 as needed through retrieval.

4. Six configuration details beginners should prioritize

4.1 Understand the load scope and inheritance of configuration

Different tools load rules files from different locations, such as global directories, project roots, and subdirectories, and their priorities can differ significantly. A general recommendation is to organize them by layer:

• Global rules should contain only cross-project preferences, such as response language, whether tests are written by default, or commit-message style.
• Project rules should contain the project’s technology stack and architecture prohibitions.
• Subdirectory rules, such as those for frontend directories or migration-script directories, should contain local strong constraints.

The most common beginner mistake is writing project-specific conventions into global rules, which then interferes with unrelated projects. The opposite mistake is putting rules in a subdirectory but starting the session at the root directory, so the rules are never loaded. First verify that the rules are actually read, then discuss whether the rules are written well.

4.2 Write only executable constraints, not vague correct statements

Sentences such as “write high-quality code,” “follow best practices,” or “pay attention to security” have almost no binding power. They repeat L0 knowledge while consuming valuable context budget. An effective rule must be decidable: it must be possible to answer whether it is violated with “yes” or “no.”

Comparison:

❌ Amount handling must be rigorous and avoid precision issues.
✅ Amounts must be represented as integers in the smallest currency unit, such as cents. Float or double types must not participate in amount calculations. When serializing externally, use strings.

4.3 Pair each hard constraint with a positive and negative example

Models follow examples more reliably than abstract descriptions. Adding two lines of code examples to a rule is low-cost and high-return:

## Hard constraint: Amount representation
- Amounts must be represented as integers in the smallest currency unit, such as cents. Floating-point numbers must not be used in any amount calculation.
  ✅ amount_cents: int = 1990
  ❌ price: float = 19.90
- When serializing amounts in external APIs, use strings to avoid precision and formatting ambiguity.
  ✅ {"amount": "1990", "currency": "CNY"}
  ❌ {"amount": 19.9}

4.4 Control the length of the rules file itself

The rules file is loaded in every session, so it consumes context budget itself. A rules file with thousands of lines causes two problems. First, it crowds out the space that should be used to understand the code. Second, when constraints conflict with one another, the model cannot determine their priority and is more likely to go astray.

A practical approach is to keep the total length to one or two screens. Order the rules as follows: hard prohibitions → architecture boundaries → change discipline → verification method. The earlier a rule appears, the less likely it is to be affected by mid-context decay. Move detailed rules into separate documents, and keep only a one-line index and trigger conditions in the rules file.

4.5 Externalize decision history into structured logs

The correct form of L3 is not a chat transcript, but a document in the version repository that is readable, searchable, and evolvable. A lightweight solution is the Architecture Decision Record, or ADR: one decision per file.

# Decision 0007: Session state storage choice
- Status: Accepted
- Context: Single-instance in-memory storage cannot support horizontal scaling across multiple instances and loses all sessions on restart.
- Decision: Store session state centrally in an in-memory cache database. Use the `sess:` prefix for all keys and set a TTL of 30 minutes.
- Rejected alternative: Write to a relational database, because the latency and connection-cost overhead is too high for short-lived session data.
- Impact: Requires configuration for connection-pool limits, serialization format, and degradation behavior when the cache is unavailable.
- Review condition: Re-evaluate if the requirement changes to “sessions must survive restarts” or if QPS exceeds the threshold.

The value of this document is that the assistant in a new session can read it for one minute and obtain a conclusion that took you two hours to reach in a previous session. The review condition field also prevents the assistant from continuing to use an old conclusion after requirements have changed.

4.6 Turn verbal agreements into runnable checks

This is the most easily overlooked step with the greatest benefit: if an agreement can be checked by a machine, do not rely on the model’s self-discipline.

The model’s compliance probability is not 100%, but a script’s pass rate is deterministic. Add a check in a commit hook or pipeline, and the rule is upgraded from “prompt text” to “engineering constraint”:

#!/usr/bin/env bash
# Validate that amount-related variables do not use floating-point types
set -euo pipefail
if grep -RnE '\b(price|amount|total|balance)\w*\s*[:=].*\b(float|double)\b' src/; then
  echo "[FAIL] Floating-point amount variables detected. This violates project rules." >&2
  exit 1
fi
echo "[OK] Amount type check passed"

Even better, connect this to a standard command, such as make check, which includes lint, type checking, and unit tests. Then state in the rules file that “the check must pass before submission.” Even if the model forgets the rule, the feedback loop will pull it back on track within seconds.

5. A rules-file skeleton you can copy directly

# Project Rules
## 1. Hard constraints (violations are treated as errors and have the highest priority)
- Amounts must be represented as integers in the smallest currency unit. Float or double types must not participate in calculations, and serialization must use strings.
- Do not concatenate SQL directly in business code. Use parameterized queries.
- Before adding an external dependency, explain the reason and alternatives, and wait for confirmation before introducing it.
## 2. Architecture boundaries
- Call direction: interface layer → service layer → data access layer. Cross-layer calls are prohibited.
- Session state is stored in the cache layer. Keys use the `sess:` prefix and a TTL of 30 minutes.
## 3. Change discipline
- Any database schema change must include a rollback-capable migration script.
- Before making changes, read the relevant decision records under `docs/decisions/`. Do not overturn existing decisions unless new evidence is provided.
- After completing each feature point, update the corresponding entry in `docs/decisions/` or `TASKS.md`.
## 4. Verification method
- `make check` must pass before submission. This includes lint, type checking, and unit tests.
- Changes involving amounts, permissions, or data deletion must include corresponding unit tests.
## 5. Collaboration preferences
- Use English for answers and code comments. Commit messages follow Conventional Commits.
- When multiple viable options exist, list the trade-offs first. Do not immediately perform large-scale rewrites.

Also include a minimal task-list file for cross-session handoff:

# TASKS
- [x] Initialize the data model
- [ ] Migrate session state to the cache layer (in progress: service layer changed; degradation logic pending)
- [ ] Unify amount field types (blocked: waiting for the order module owner to confirm the smallest-unit definition)

6. Four common misconceptions

Misconception 1: Treating the rules file as a universal cure. It solves L1, but not L3. Without a decision log, you will repeatedly explain the same thing in every new session.

Misconception 2: Using repeated reminders instead of structured constraints. When you say “remember to use integers” for the third time, the correct next action is not to say it a fourth time, but to write a validation script.

Misconception 3: Stuffing all the weight into the rules file at once. The more rules you pile up, the more conflicts arise, and the more severe mid-context decay becomes. Rules should be subtractive: keep only the entries that are decidable and likely to be violated.

Misconception 4: Never reviewing configuration. A rules file that has not been updated for three months is a liability, not an asset. Include rules files and decision records in code review, and update rules when the architecture changes.

7. Implementation checklist

1. Confirm that the rules file is actually loaded in the session, for example by asking the assistant to restate the key constraints.
2. Check whether all hard constraints are decidable and whether they have positive and negative examples.
3. Check whether the rules file is kept to one or two screens and ordered by priority.
4. Check whether there are critical decisions that exist only in conversation and are not written into documentation.
5. Check whether all machine-checkable agreements have been converted into hooks or pipeline checks.
6. Check whether updating rules files and decision records has become part of the regular review process.
7. Check whether long tasks have been split into shorter sessions, with relevant decision summaries loaded at the start of each session.

Whether you use terminal-based AI coding tools or general-purpose coding assistance and knowledge-retrieval capabilities provided by platforms such as Baidu AI Cloud, the same principle applies: the model handles reasoning and generation; you are responsible for building its inputs.

8. Conclusion

Return to the original question: why does the assistant repeatedly make the same mistake? Because it starts from zero every time, while you expect it to remember the previous session.

The real solution is not to find a model with “better memory,” but to move memory out of the model’s head and into the engineering structure. Write static specifications as decidable rule files, record decision history as searchable documents, and hand automatically checkable agreements to pipelines and commit hooks. Once you do these three things, the model’s amnesia is no longer the ceiling of your efficiency.

If you can do only one thing, do this: turn the verbal agreement that is most easily violated into a check that automatically fails. The return is greater than rewriting the prompt a hundred times.