LLM Agent Memory Testing: Layered Validation and Contamination Control

1. Why a Wrong Note Is More Dangerous Than a Wrong Memory

The hard part of testing agent memory is not whether it remembers correctly. Humans forget, and models forget. The more important engineering questions are: how long can a wrong thing live, and how far can it spread?

Consider a level-running scenario. The task requires an agent to explore an open-world game map, reach a designated location, and advance the level objective. The agent explores a cave area but writes the wrong direction in its memory note: “the cave is to the east.” After leaving the building, it concludes that the cave is already behind it, so it updates the note to “cave completed.” Later, it jumps down a one-way staircase, tries three times without success, and finally decides to “stop instead of continuing the loop.” The run ends.

This sequence contains at least three wrong decisions, but they share the same premise: an unsupported conclusion is treated as a fact.

The problem becomes even clearer when a new run starts. If this note is preserved and automatically loaded at the beginning of every session, every later run starts from “cave completed.” The agent will never return to the cave and will have no chance to discover the original mistake. A single memory error causes one loss, but a persistent memory error makes every retry inherit the same wrong premise. This is a classic testing pattern: a single test case almost always passes, but production continues to fail.

A related experiment showed the other side of the problem. An offline “consolidation” process, executed during idle time, can compare notes line by line with raw session logs, discover the wrong direction, and correct it. After correction, the agent directly found the cave and reached the next city. The uncorrected group never arrived.

These two cases lead to the core proposition of memory testing:

• Memory testing is not about whether the answer is correct, but about how far a wrong memory can propagate.
• The more automated memory becomes, the faster errors spread. The layer that is unconditionally loaded at the start of every new session is both the fastest-spreading layer and the one most likely to be ignored by tests.

Memory testing also needs a clear boundary. Otherwise, it is easy to mix in issues such as within-session context or summary compaction. Here, memory means knowledge that a model obtains from experience, preserves across sessions, and can later read. It does not include three things: changes to model weights, learning inside a single session, or conversation compaction that replaces earlier turns with summaries. This boundary limits the test object to “memory files read and written by agents.” In other words, memory becomes an ordinary engineering component that can be checked with file validation, data comparison, and pipeline gates, rather than a black box that can only be judged subjectively.

2. Four Layers of Memory: Four Different Failure Modes

2.1 From event logs to persistent conclusions

According to common layer definitions in public research, agent memory can be divided into four layers from bottom to top:

Layer	Content	How it is produced	How it is loaded	Scale and update frequency
Session logs	A retrievable event stream of interactions, actions, and tool returns	Recorded automatically by the system	Retrieved on demand	Largest scale; continuously appended
Memory notes	Observations, rules, or corrections the agent judges worth keeping	Written autonomously by the agent	Retrieved on demand or loaded with tasks	Medium; appended or revised sporadically
Memory representations	Structured knowledge grown from notes, such as interlinked knowledge pages	Induced and organized from notes	Retrieved on demand or loaded by topic	Concise; periodically reorganized
Injected context	A small set of content automatically loaded at the beginning of each session	Selected and pushed from lower layers	Unconditionally loaded in full	Fewest entries; only grows and is not automatically reduced

Between these layers is the “consolidation” process, commonly called memory consolidation. During idle time, it re-reads logs, verifies notes, builds or updates structured representations, and pushes the most reliable content to the top layer.

Consolidation is not an optional operational step. It is the core mechanism that determines how knowledge flows between layers. Each layer has both an entrance and an exit. The consolidation process decides what is promoted, what is rewritten, and what should be discarded. Testing therefore needs to examine both sides of the loop:

• Entrance: Can errors be kept out of the persistent layer?
• Exit: Can already-written errors be detected and undone?

2.2 Four failure modes

The four layers differ not only in granularity but also in how they fail.

Session logs record everything, so the main issue is noise. Irrelevant information dilutes retrieval results, while important information may be hard to recall if it lacks structured markers. Errors in the log layer usually have limited impact: as long as they are not promoted into higher-level conclusions, they can still be retrieved and reinterpreted.

Memory notes are written autonomously by the agent. The most typical failure is “an unverified observation being treated as a fact.” In the level-running example, “the cave is to the east” is exactly this kind of failure: it has no corresponding action record and no observational evidence, yet it is written down as a reliable conclusion.

Memory representations are induced from notes. Their most typical failure is “induction distortion.” Several individually valid notes can lose their qualifiers when compressed or merged into one structured conclusion, turning special cases into general rules. After entries are merged, it is also often difficult to trace which original note caused the problem. Troubleshooting cost is therefore significantly higher than at the note layer.

Injected context has the most serious failure mode because it is loaded unconditionally and in full. Every new session reads every entry in it, while users often do not even know what is inside. This layer does not need to be retrieved to influence model behavior; it is already active.

2.3 Measuring layer risk with an amplification coefficient

The same error can have different consequences depending on which layer stores it. Each layer can be assigned an amplification coefficient: the average number of future sessions or tasks affected by one wrong memory during its lifecycle.

• Written to session logs and never read: impact is approximately 0.
• Written to notes but only retrieved for related tasks: impact is limited and decays with relevance.
• Induced into structured representations: may affect tasks across multiple topics.
• Written to injected context: impact equals all subsequent sessions.

The purpose of layered management is to make write choices explicit: write to a layer and accept that layer’s risk. The consolidation process acts as a continuous router. It decides whether a note is promoted, corrected, or demoted back to the log layer.

3. Why Most Memory Tests Miss Errors

Once the layers are clear, it becomes easier to see why tests fail. Missed defects are not only caused by too few test cases. More often, the testing perspective itself is misaligned. There are nine common patterns.

1. Only verify read hits, not the write and re-read chain. Test cases often pre-populate a “correct” memory store, then ask the model to find the answer. This actually tests retrieval ability. Whether content written by the agent itself can be correctly understood and used in the next session is not covered.
2. Fixtures contain the correct answer. Facts in test data are always correct, so the agent has no chance to write wrong memory. Defects in the write path cannot be exposed.
3. Every case uses a fresh memory store. Cases are isolated from one another. Cross-session and cross-task propagation is naturally cut off, so accumulated errors in the persistent layer cannot be observed.
4. Treat injected content as a test constant. Many test frameworks fix each session’s input to the prompt plus a small amount of background. The injection layer is empty or manually specified. Production behavior—automatically loading all persistent content—is effectively absent from testing.
5. Assertions only check final output. Tests usually check whether the model answer contains a keyword or fact. They do not check the actual state of memory files, which layer produced the conclusion, or whether there is supporting evidence.
6. No time or ordering dimension. The order between wrong writes and corrections determines how many times an error is inherited, but cases rarely simulate a “write wrong first, correct later” process.
7. No consolidation validation. Many tests treat consolidation as an omittable implementation detail: they either disable it or only check that it “runs successfully,” rather than verifying that it “corrects correctly, completely, and without false edits.”
8. No size or duplication constraints. The persistent layer keeps accumulating content during use, but there is no upper-bound check. Tests focus on functional correctness, not on whether the layer is growing out of control.
9. Only test single-agent, single-task behavior. When multiple agents or multiple tasks share the same memory, a wrong note may spread across tasks. Tests that close only within one scenario miss this risk.

In short, the root cause of missed defects is a misplaced testing perspective: memory is treated as “part of the output” to be checked, rather than as a subsystem with write, propagation, consolidation, and recovery stages.

4. Four Test Dimensions: From “Correct or Not” to “How Long It Lives and How Far It Spreads”

Based on the failure modes above, memory testing can be organized into four dimensions.

Dimension 1: Layer correctness. Is each piece of knowledge written into the appropriate layer? Are there clear promotion and demotion rules? Will the agent directly promote a single observation into a persistent conclusion?

Dimension 2: Error survival. How much time, how many sessions, or how many other memories pass between the wrong write and its detection and correction? This metric describes the system’s “self-healing speed.”

Dimension 3: Spread scope. How many sessions, tasks, and downstream memories does a wrong memory ultimately affect? Did it enter the persistent layer?

Dimension 4: Consolidation convergence. Can the consolidation process detect and fix errors? Will it wrongly modify correct content? Are repeated runs consistent? How much pending work does each run reduce?

The main focus and observation targets for each dimension are as follows:

Dimension	Core question	Main observation targets
Layer correctness	Is knowledge written to the correct layer?	Items and sources in memory notes, structured representations, and injected context
Error survival	How long does an error live before discovery?	Sessions, log entries, and memory entries between wrong write and correction
Spread scope	How far does an error eventually spread?	Affected sessions, tasks, downstream memory entries, and whether it enters the persistent layer
Consolidation convergence	Is the correction mechanism reliable?	Consolidation detection rate, false-correction rate, idempotency, and pending-work convergence

The most easily overlooked dimension is dimension 2. A system that can “only remember correct things” and a system that “can discover mistakes after remembering wrong things” may look almost identical in functional test cases, but only the latter is suitable for long-term operation.

5. A Decision Table for Choosing the Right Layer

Since layers determine risk, explicit write rules are needed. The table below maps the source, verifiability, and usage scope of a piece of knowledge to its target layer. It can be used as a design guide for write logic and as a checklist for reviewing agent write behavior.

Content feature	Source	Verifiability	Usage scope	Recommended layer	Persistent-layer admission
Raw interactions, tool returns, execution traces	Recorded automatically by the system	Factual record	Current session and later retrieval	Session logs	Not allowed
A single observation not checked against raw records	Written autonomously by the agent	Pending verification	Current task	Memory notes, marked “unverified”	Not allowed
Observations, rules, or user corrections checked against logs	Written by the agent or consolidation process	Verified	Reusable for similar tasks	Memory notes	Only a few reviewed entries may be promoted
Conclusions induced from multiple verified notes, or relationships between entities	Generated by the consolidation process	Traceable to source notes	Cross-task and topic-based retrieval	Structured representations	Load on demand depending on compressed size
High-frequency, stable, reviewed, and sourced conclusions	Pushed by the consolidation process	Multiple consistent checks	Needed in every new session	Injected context	Required

Before writing, ask three questions in order:

1. Is there support from raw records? If not, keep it in the note layer and mark it “unverified.” Do not promote it.
2. Has it been confirmed by independent evidence, or corrected at least once? If not, it cannot enter the persistent layer.
3. Must every new session know it? If not, let it be retrieved on demand instead of being persistent.

Another often ignored principle is: store conclusions, not reasoning. A note saying “I tried three methods and none succeeded” has almost no value for the next session. What matters is a conclusion such as “this area can only be entered from the west side.” Conclusions are short, verifiable, and reusable. Reasoning processes remain in logs for later inspection.

6. A Runnable Size and Duplication Gate for Injected Context

The persistent layer needs hard constraints that can be placed in a pipeline. The following illustrative script checks item count, total size, single-item length, near duplicates, source, and review status.

# inject_gate.py -- Persistent injected context gate (illustrative implementation)
import hashlib
import json
import re
import sys
from dataclasses import dataclass
from datetime import date
@dataclass(frozen=True)
class Limits:
    max_items: int = 30          # Maximum number of persistent items
    max_total_bytes: int = 12000  # Maximum total bytes of persistent content
    max_item_bytes: int = 400     # Maximum bytes for a single item
    max_stale_days: int = 30      # Items not reviewed within this window are stale
@dataclass(frozen=True)
class Item:
    id: str
    text: str
    source: str           # Source: session or log fragment identifier
    reviewed: bool        # Whether reviewed by the consolidation process
    updated_on: date      # Last confirmed date
    scope: str = "global" # global = always persistent, scoped = loaded by task or topic
def shingles(text: str, k: int = 4) -> set[str]:
    s = re.sub(r"\s+", "", text)
    return {s[i:i + k] for i in range(len(s) - k + 1)} or {s}
def simhash(text: str, bits: int = 64) -> int:
    v = [0] * bits
    for sh in shingles(text):
        h = int(hashlib.md5(sh.encode()).hexdigest(), 16)
        for b in range(bits):
            v[b] += 1 if (h >> b) & 1 else -1
    return sum(1 << b for b in range(bits) if v[b] > 0)
def hamming(a: int, b: int) -> int:
    return bin(a ^ b).count("1")
def check(items: list[Item], today: date,
          limits: Limits = Limits()) -> dict:
    findings = []
    global_items = [it for it in items if it.scope == "global"]
    total = sum(len(it.text.encode()) for it in global_items)
    if len(global_items) > limits.max_items:
        findings.append(
            f"Persistent item count {len(global_items)} exceeds limit {limits.max_items}"
        )
    if total > limits.max_total_bytes:
        findings.append(
            f"Total persistent size {total} bytes exceeds limit {limits.max_total_bytes}"
        )
    hashes = []
    for it in global_items:
        size = len(it.text.encode())
        if size > limits.max_item_bytes:
            findings.append(
                f"[{it.id}] Single item length {size} bytes exceeds limit"
            )
        if not it.source:
            findings.append(f"[{it.id}] Missing source; origin cannot be traced")
        if not it.reviewed:
            findings.append(f"[{it.id}] Entered persistent layer without review")
        if (today - it.updated_on).days > limits.max_stale_days:
            findings.append(
                f"[{it.id}] Not reviewed for {limits.max_stale_days} days; "
                "demote or remove it"
            )
        h = simhash(it.text)
        if any(hamming(h, prev) <= 3 for prev, oid in hashes):
            findings.append(f"[{it.id}] Near-duplicate of an existing persistent item")
        hashes.append((h, it.id))
    return {
        "total_bytes": total,
        "findings": findings,
        "passed": not findings,
    }
if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "memory/inject.json"
    with open(path, encoding="utf-8") as f:
        raw = json.load(f)
    items = [
        Item(**{**it, "updated_on": date.fromisoformat(it["updated_on"])})
        for it in raw
    ]
    report = check(items, date.today())
    print(json.dumps(report, ensure_ascii=False, indent=2))
    sys.exit(0 if report["passed"] else 1)

The rules and design rationale are as follows:

• Limits on item count, total size, and single-item length. Persistent content consumes the context budget, so hard caps are required. The values should be determined through baseline load testing, not intuition.
• Near-duplicate detection. Locality-sensitive hashing such as SimHash is recommended. Each item’s fingerprint is computed only once. New content only needs to be compared with existing fingerprints. The cost is proportional to the number of persistent items, making it suitable for pipelines. To choose a better threshold, first compare historical items pairwise, observe the actual similarity distribution, and then set the threshold.
• Source and review status. Every persistent item must be traceable to a session or log fragment and confirmed by the consolidation process. This is consistent with the decision table rule that “a single unverified observation cannot enter the persistent layer.”
• Stale-item demotion. Items not reviewed within the configured window should be removed from the persistent layer or demoted back to the note layer.

The script output should also be sampled and verified in real environments:

• An item exceeding the single-item length limit often means reasoning has been mixed into a note. It should be split into a conclusion part and a log part.
• Items flagged as “near duplicates” require human confirmation: are they true duplicates, or two different but similarly phrased conclusions?

7. Mapping the Four Dimensions to Test Cases

The gate only constrains the persistent layer. The four test dimensions still need concrete test cases.

7.1 Layer correctness: assert on writes

The test case does not check what the model says. It checks what the memory files become.

def test_unverified_observation_never_reaches_inject(agent, memory):
    memory.reset()
    agent.run(task="Explore the cave", memory=memory)
    # Unverified observations not checked against logs can only stay in the note layer.
    for note in memory.notes(layer="inject"):
        assert note.reviewed and note.source
    unverified = [n for n in memory.notes() if not n.verified]
    assert unverified, "Unverified observations should be kept in notes for later checking."
    assert all(n.layer != "inject" for n in unverified)

7.2 Error survival and spread: fault injection

Manually write a wrong note, run several subsequent sessions, count how many rounds it survives and which tasks it affects, and check that the persistent layer remains clean.

import pytest
def test_wrong_note_is_contained_and_corrected(agent, memory, logs):
    memory.reset()
    memory.add_note(
        text="The cave is to the east",
        source="manual-injection",
        verified=False,
    )
    for round_no in range(1, 6):
        agent.run(task="Reach the designated location", memory=memory)  # Affected task
        agent.run(task="Organize repository documents", memory=memory)  # Unrelated task
        inject = memory.layer("inject")
        assert "east" not in inject, (
            f"Round {round_no}: wrong note was promoted to a persistent conclusion"
        )
        # Consolidation should compare notes against raw logs and correct the error.
        report = memory.consolidate(logs)
        if report.fixed:
            assert all("east" not in n.text for n in memory.notes())
            break
    else:
        pytest.fail("Wrong note was not corrected within 5 sessions")

Metrics to record include:

• Survival rounds: the number of sessions between the wrong write and correction.
• Impact scope: the number of sessions and tasks where the wrong content was injected or retrieved.
• Cross-task isolation: whether unrelated task outputs remain identical to a clean baseline.

7.3 Consolidation convergence: detection, no false edits, idempotency

The consolidation process is also a system under test. Let multiple agents run for a period to generate real logs and notes, then run consolidation and check four points:

• Detection rate: the percentage of injected wrong notes that are corrected.
• False-correction rate: the percentage of correct notes that are wrongly modified.
• Idempotency: when content is unchanged, a second run should not produce new changes.
• Convergence: the number of pending consolidation items should decrease over runs, not accumulate.

8. Integrating into the Engineering Workflow

A single test case can discover problems. Continuous gates prevent them from accumulating. A four-stage rollout is recommended:

1. Connect to the pipeline. Add the persistent-layer check script to continuous integration and run it at commit time. Item limits, missing sources, unreviewed entries, and near duplicates should all cause failures.
2. Establish a baseline. Run a fixed set of tasks in a clean environment and record the size, item count, duplication rate, and change frequency of persistent content. If growth is clearly abnormal, it means consolidation is only appending, not merging or deleting.
3. Run periodic drills. Every month, inject known errors into the test-environment memory store. Verify whether the system can find and correct them within the expected number of rounds and whether they spread to other tasks.
4. Design for rollback. Before each consolidation run, snapshot the memory store. After the run, keep change records. If a false edit is found, roll back to the previous version. Without versioning and rollback, any wrong consolidation rewrite can become new persistent contamination.

Public research on memory use in large-scale code repositories also offers a useful lesson: before writing, decide whether an observation truly has cross-session value. Fewer writes mean fewer errors for gates and consolidation to handle. The same applies to testing. Rather than searching for contamination among massive entries, it is better to reduce unnecessary writes at the entrance.

9. Conclusion

• The core problem of memory testing is not “can the model remember,” but “how long can a wrong memory live and how far can it spread.”
• Layers determine risk. Write rules determine layering. The consolidation process determines how knowledge flows between layers.
• The persistent layer must simultaneously have size gates, duplicate detection, source requirements, and review requirements.
• The four test dimensions—layer correctness, error survival, spread scope, and consolidation convergence—must be implemented through fault injection and cross-session test cases, not functional cases that rely on pre-set correct answers.
• Errors are unavoidable, but their lifecycle and impact range can be designed. A memory system that can confine errors to the log layer, keep them out of the persistent layer, and correct them within a limited number of rounds is a memory system that can run for the long term.