Agent Long-Term Memory: From Full Logging to Selective Retention

For an agent to stay coherent across multiple turns and multiple sessions, it needs some form of memory. But remembering more is not necessarily better. Some recent comparative experiments have shown that agents which save a large volume of notes in full sometimes underperform, on the same tasks, versions that keep only a small number of essential notes. This suggests that the core question for a memory system is not “how much to store,” but “what to store, when to store it, and how to organize it.” This article breaks down a filtering and consolidation approach along these lines and provides parameters and evaluation methods that can be applied directly.

1. Defining the Boundary: What Counts as Agent Memory

Agent memory in this article refers to information that an agent obtains from past experience, can retain across sessions, and can retrieve and use in later reasoning. This information includes user interaction history, records of completed tasks, and preferences and constraints that affect later decisions.

Three common mechanisms are easily confused with agent memory and need to be considered separately:

1. Weight updates. The model adjusts parameters through backpropagation during training, writing knowledge into the model itself. This happens in offline training and is a change to model parameters, not runtime memory.
2. In-context learning. The model reads materials provided within a single conversation window. After the session ends, this information disappears with the window.
3. Context compression. Old conversations are summarized into shorter summaries to free up new context space. The summary keeps the main points, but details are lost. This is a form of lossy compression.

Memory writes and consolidation can happen asynchronously, but reads usually occur in the reasoning loop. This determines the system design: the write side can be slower and more detailed, while the read side requires speed and accuracy.

2. Why Full Logging Fails

The idea behind full logging is “store more rather than miss anything.” This approach may look safe at first, but problems accumulate over time.

First, the signal-to-noise ratio declines. In long-horizon tasks, key decision points—for example, “the user changed the priority of the requirement to performance”—are often drowned out by a large number of low-value events, such as “the user clicked a button.” During retrieval, the top results are occupied by noise, and key information does not rank high enough.

Second, storage and retrieval costs rise. Every record consumes storage and also consumes computation during retrieval. For systems that generate a large number of logs every day, the infrastructure overhead of full retention continues to increase. Retrieval latency may also rise with the number of records; the extent depends on the index structure, data scale, and query complexity. To determine whether it has become a bottleneck, measure p95 latency at different memory sizes on the same batch of queries before drawing a conclusion.

Third, accidental co-occurrence is treated as a pattern. Not all events that appear together have a causal relationship. Full logging causes the agent to repeatedly see meaningless coincidences. If the retrieval mechanism treats these coincidences as evidence, systematic bias can form.

The table below compares the two strategies side by side:

Dimension	Full logging	Selective memory
Storage volume	Grows linearly over time	Controlled by thresholds and windows, grows more slowly
Retrieval noise	High; key information is easily drowned out	Low; every record has been filtered
Retrieval latency	Rises with scale; high index pressure	Controllable scale; more stable latency
Interpretability	Poor; hard to explain why a record was retrieved	Good; each memory has a score and source
Main risk	Accumulation of noise and bias	Missed records, which must be found through evaluation

Selective memory also has a cost. The main one is missed records. It must therefore be used together with evaluation, and continuous checks are still needed after launch.

3. Event Value Scoring: Deciding What Is Worth Remembering

To answer “what should be remembered,” an importance score needs to be assigned when an event occurs. This score can be obtained as a weighted sum of four dimensions:

1. Outcome orientation: Whether the event directly affects the task outcome. “The user confirmed the final plan” scores high; “the user opened the document” scores low.
2. Anomaly and deviation: Whether the event deviates from the default process. “The model refused the instruction” or “the user interrupted the standard process midway” often indicates that the strategy needs adjustment.
3. Instruction explicitness: Whether the event contains an explicit instruction for the future. “Next time, please report in a table” is more worth saving than “this is a bit messy.”
4. Information irreversibility: Whether the information in the event can be obtained again from other channels. A preference stated orally by the user is hard to recover later if it is not recorded.

Score each dimension from 0 to 1 and take the weighted sum. A set of starting weights is: outcome orientation 0.35, anomaly and deviation 0.25, instruction explicitness 0.25, and information irreversibility 0.15. The weights should be adjusted by task type; these values are only a starting point.

There are two ways to set the threshold. One is to set it by capacity budget: if at most 500 memories can be kept per day, use the position of the top 10% of scores as the threshold. The other is to plot a precision-recall curve on a batch of labeled samples and choose the threshold corresponding to an acceptable miss rate. The former is simple and suitable for a quick launch; the latter is more reliable and suitable for scenarios with labeled data.

Only events above the threshold enter the next stage of memory. Events below the threshold can remain in short-term logs and be processed together after the window ends.

4. Interval Batching: Evaluating Events in Context

Pointwise evaluation has an obvious weakness: when an event is viewed in isolation, its value is often hard to read. “The user sent a piece of code” contains little information by itself. But if it appears during a period of “discussing performance optimization,” its meaning is completely different.

Therefore, interval batching can be used instead. Set a time window.