Agent Memory Engineering for Long-Horizon Agents: Layered Memory Structures and Injection Budgets

Conversational AI used to be a short turn: one question, one large model. Today’s agents can run for hours or even days, decomposing tasks autonomously, calling tools, writing and executing code, and advancing the same project across many separate sessions. Once the operating window stretches out, a new bottleneck appears: the model itself has no persistent memory across sessions, and the context window remains a scarce resource that is finite, expensive, and prone to getting “muddy.” As a result, “giving the agent memory” has become a core engineering problem for shipping long-horizon agents.

One vendor has publicly described the components of a hosted agent memory capability. One detail is especially practical: when injecting persistent memory into the agent, there is a clear ceiling, and typically only roughly the first ~200 lines, about 25 KB of a memory file, are brought into context. The vendor also explains “in-session notes, offline consolidation, and injection budget” as separate pieces, rather than vaguely claiming “unlimited memory.” The value of this information for teams building their own agents lies not in any specific number, but in the design philosophy behind it: memory is a tiered system to be managed, and you cannot expect to pour an entire history into the context in one shot.

1. First, Define Terms: What “Memory” Actually Means for a Long-Running Agent

Borrowing the distinction between working memory and long-term memory from cognitive science, and combining it with engineering practice, agent memory can be divided into three layers:

• Short-term working memory: the context window of the current session. It holds recent tool calls, reasoning steps, observations, and intermediate conclusions. Its capacity is limited, and it is cleared when the session ends.
• Persistent memory: structured notes kept across sessions, recording a user’s stable preferences, project constraints, key decisions, and lessons learned. This is usually stored in files, a database, or a retrieval index.
• Raw traces: the complete session records and tool call logs. These are large and information-dense, suitable for looking up on demand rather than entering the context by default.

It is worth separating “memory” from “context” explicitly. Memory is knowledge that must persist across sessions and that you deliberately bring back after filtering; context is everything the model can see during a single inference. The goal of memory engineering is to put exactly enough of the right memory into the context at the moment it is needed.

2. Three Typical Failure Modes of Long-Horizon Agents: Remembering Wrong, Remembering in Fragments, Remembering Too Much

The vendor summarizes the difficulty of memory capability as “remembering wrong, remembering in fragments, and remembering too much.” These three failure modes are equally common in self-built systems and can be defined concretely as follows:

Failure mode	Concrete symptoms	Common causes
Remembering wrong	Stale information is treated as current, for example still calling the old API after a tech stack upgrade; contradictory facts coexist; a user’s temporary preference hardens into a long-term rule; the model “fills in the blanks” during consolidation and invents facts that never happened	No time or source markers; overwriting instead of merging; no fact-checking at the extraction stage
Remembering in fragments	The same fact is scattered across several near-duplicate notes; fragments lack context; tags are messy and retrieval fails; consolidation gets messier the more you do it	Appending to the end of the file every session; no deduplication or structured template
Remembering too much	The memory file keeps growing and injection cost climbs sharply; context is crowded out by stale details and the model “gets lost in the middle”; key information drowns in noise	Writing without deleting, no eviction mechanism; no retrieval capability, so full injection is the only option

These three failure modes amplify one another: remembering wrong creates contradictions, remembering in fragments amplifies noise, and remembering too much simply crushes the context. A complete memory system therefore has to answer three questions at once:

1. What to remember: which information is worth keeping long term;
2. How to manage it: how to consolidate, deduplicate, update, and evict;
3. How to use it: within a limited injection budget, which parts go into the context and in what order.

3. Stage One: In-Session Notes — The Raw Material for High-Quality Memory

The quality of persistent memory depends largely on what gets recorded during a session and how. In-session notes can follow these principles:

1. Atomic: one note expresses one fact or one decision, which makes later merging, retrieval, and individual eviction easier.
2. Structured: use a uniform template with fixed fields, such as topic, content, source, time, applicable scope, and confidence.
3. Carries time and source: every memory should know “when it was written and based on what.” This is the first line of defense against remembering wrong.
4. Distinguishes facts from inferences: what the user explicitly said is a “fact”; inferences the model makes from observation should be labeled by type so later sessions do not treat them as established fact.
5. Distinguishes long-term from temporary: long-term preferences go into persistent memory; temporary decisions inside a task stay in the session and are discarded when the task ends.

3.1 Example of a Structured Memory Entry

[
  {
    "id": "pref-001",
    "category": "preference",
    "content": "All example code uses Python 3.11; Python 2 syntax is not allowed",
    "source": "user_explicit",
    "created_at": "2026-01-05T10:12:00Z",
    "updated_at": "2026-01-05T10:12:00Z",
    "scope": ["code-generation"],
    "confidence": 0.95
  },
  {
    "id": "dec-014",
    "category": "decision",
    "content": "The order service cache uses a two-level structure of Redis plus a local cache",
    "source": "session_summary",
    "created_at": "2026-01-08T15:40:00Z",
    "updated_at": "2026-01-08T15:40:00Z",
    "scope": ["project:order-service"],
    "confidence": 0.8
  }
]

3.2 Extraction Flow at the End of a Session

In long sessions, tool output and reasoning steps routinely occupy tens of thousands of tokens. Rather than stuffing an entire session into memory, it is better to do a single “distillation” when the session ends:

# Illustrative code: post-session memory extraction flow (pseudocode)
async def extract_memories(session, llm, existing_memory):
    # 1. Summarize in layers first to control input volume
    dialogue_digest = await llm.summarize(
        session.dialogue, max_tokens=2000, style="fact list"
    )
    tool_digest = await llm.summarize(
        session.tool_calls, max_tokens=1000, focus="key conclusions and changes"
    )
    # 2. Extract candidate memories against a fixed schema
    schema = "category, content, source, scope, confidence"
    candidates = await llm.extract(
        f"Existing memory:\n{existing_memory}\n"
        f"Session summary:\n{dialogue_digest}\n{tool_digest}",
        schema=schema,
        rules=[
            "Keep only stable information that is reusable across sessions",
            "Do not record one-off task details or temporary data",
            "Inferred content must be marked source=inference",
        ],
    )
    return [c for c in candidates if c["confidence"] >= 0.7]

Extraction results should not be appended directly to the memory file. They should go into a “pending consolidation queue” and be handled together by the offline consolidation stage.

4. Stage Two: Offline Consolidation — Deduplicating, Updating, and Layering Memory

“Offline” does not mean the work must happen in a nightly batch job. Its core meaning is that consolidation is decoupled from the agent’s main task: it does not block the critical path, and it does not compete with content generation for the same context budget. The consolidation cadence can be every N sessions, once a day, or triggered when the memory file exceeds a size threshold.

4.1 The Five Stages of the Consolidation Pipeline

1. Deduplicate and merge: first use a full-text index or vector retrieval to find similar entries, then let the model decide whether they are duplicates, supplements, or conflicts.
2. Resolve conflicts: when contradictory statements appear on the same topic, prefer the more recently written entry; when the conflict cannot be judged, mark it as “pending user confirmation” instead of silently deleting it.
3. Archive and downgrade: move superseded old entries into an archive that is excluded from injection but still retrievable on demand.
4. Summarize and compress: merge scattered single observations into one general conclusion, for example combining ten “the API returns slowly” notes into “the report API has high P99 latency, optimized on a given date.”
5. Control total volume: set a line-count and size budget for the memory file, and proactively compress low-value entries as it approaches the threshold.

4.2 Memory File Structure After Consolidation

# Core profile (always kept lean, used for injection)
## User preferences
- [pref-001] Example code uses Python 3.11 (explicit from user · 2026-01-05)
## Project constraints
- [cons-003] The order service must not introduce new synchronous blocking calls (code review · 2026-01-08)
## Key decisions
- [dec-014] Order cache uses a two-level Redis plus local cache structure (session summary · 2026-01-08)
## Pending confirmation
- [todo-002] Whether the production database has been upgraded to 8.0 (conflicting sources · pending user confirmation)
# Topic index (for retrieval on demand)
- Databases and middleware → db-notes.md
- Historical incidents and fixes → incidents.md
# Archive (excluded from injection, retrieval only)
- [superseded-007] Order cache uses Redis only (superseded by dec-014)

This structure brings three benefits: the core profile stays a manageable size, the topic index supports retrieval on demand, and the archive preserves historical traceability.

5. Stage Three: Injection Budget — Why Full Injection Does Not Work

The “injection budget” is the share of context reserved for persistent memory on each model call. A commonly seen practical ceiling is to inject only roughly the first ~200 lines, about 25 KB of the memory file. Converted to text, that is roughly 5,000 to 7,000 tokens for English; Chinese costs more tokens per character, so the effective allowance is smaller.

5.1 Four Reasons Full Injection Is Not Worth It

• Diminishing marginal returns: for most tasks, only a small number of high-value memories actually influence the output, while the rest barely changes it.
• Diluted attention: the longer the context, the more likely the model misses information in the middle, the so-called “lost in the middle” phenomenon.
• Cost and latency grow linearly: prefill compute consumption is roughly proportional to input length, and the accumulated overhead across a long-running task is substantial.
• Errors resonate: any stale or wrong memory left in the context long term gets amplified on every single call.

5.2 A Tiered Budget: Core, On-Demand, On-Lookup

Tier	Contents	Injection method	Suggested budget share
L1 Core	Stable user preferences, hard project constraints, a few high-value decisions	Injected on every request	~10%–15%
L2 On-demand	Topic notes, historical incidents, coding standards	Retrieved by task label, inject Top-K	~10%–20%
L3 On-lookup	Complete sessions, raw logs, archived records	Agent reads files or queries as needed, no proactive injection	0 (consumed on demand)

Note that these proportions are only a design starting point and should be adjusted through measurement; they should not be treated as a universal standard. The result of this allocation is that the system “remembers a lot” overall while each request “carries only a small portion.”

5.3 Budget Accounting and Retrieval Example

# Illustrative code: budget accounting and retrieval before memory injection (pseudocode)
CHARS_PER_TOKEN_CN = 1.5   # Rough conversion for Chinese; calibrate against the model in use
LINE_LIMIT, SIZE_LIMIT = 200, 25 * 1024
RATIO_CORE = 0.15          # Share of this injection's budget allocated to core memory
def build_memory_prompt(memory_file, task, retriever, token_counter, budget):
    # 1. Physical limits: line count plus size
    core = memory_file.head(max_lines=LINE_LIMIT, max_bytes=SIZE_LIMIT)
    # 2. Logical limit: token budget for a single injection
    core_budget = int(budget * RATIO_CORE)
    core_lines, used = [], 0
    for line in core.lines():
        cost = token_counter.estimate(line, ratio=CHARS_PER_TOKEN_CN)
        if used + cost > core_budget:
            break
        core_lines.append(line)
        used += cost
    # 3. On-demand retrieval: recall L2 memory by task labels and semantics
    retrieved = retriever.search(
        query=task.description,
        tags=task.labels,
        top_k=8,
        max_tokens=budget - used,
    )
    return render_system_block(core_lines, retrieved, budget_used=used)

The retrieval component can be implemented with a general-purpose vector database or retrieval service on a cloud platform. On Baidu AI Cloud, for example, it can also be assembled from general-purpose cloud products such as object storage, vector databases, and logging services.

6. Hands-On: A Minimal Viable Memory Module

The illustrative code below shows how to chain the three stages into a closed loop: extraction at session end, scheduled consolidation, and budget-aware retrieval at injection time.

# Illustrative code: minimal viable memory module
class AgentMemory:
    def __init__(self, llm, store, retriever, token_counter,
                 line_limit=200, size_limit=25 * 1024, budget=4000):
        self.llm, self.store = llm, store
        self.retriever, self.counter = retriever, token_counter
        self.line_limit, self.size_limit, self.budget = (
            line_limit, size_limit, budget)
    async def on_session_end(self, session, user_id):
        # 1. In-session notes → candidate memories (pending consolidation)
        candidates = await extract_memories(session, self.llm,
                                            self.store.core_text(user_id))
        await self.store.enqueue_pending(user_id, candidates)
    async def consolidate(self, user_id):
        # 2. Offline consolidation: dedupe → resolve conflicts → archive → rewrite the core profile
        pending = await self.store.take_pending(user_id)
        merged = await self.llm.merge_and_resolve(pending,
                                                  self.store.core_text(user_id))
        await self.store.commit(user_id, merged, keep_history=True)
    async def build_context(self, task, user_id):
        # 3. Injection: physical limits + token budget + on-demand retrieval
        core = await self.store.core_text(user_id)
        core = core.head(self.line_limit, self.size_limit)
        used = self.counter.estimate(core)
        extra = await self.retriever.search(task.description,
                                            top_k=8,
                                            max_tokens=self.budget - used)
        return core + "\n" + extra

Three points to note:

• Each stage has a single responsibility, which makes it easy to test and replace independently;
• The pending queue and the commit operation are idempotent, so consolidation jobs can be retried safely;
• Physical limits and token budget constrain injection jointly: the physical limits keep behavior predictable when migrating across models, and the token budget keeps cost under control.

7. Governance and Security: A New Risk Surface Created by Memory

A memory system makes the agent “understand the user better,” but it also amplifies privacy, security, and compliance problems, because it stores data long term and reuses it across sessions. Governance priorities include:

• Minimize what is recorded: store only the information required to complete the task; sensitive data such as ID numbers, bank card numbers, and credentials should not be written into memory.
• Redact and encrypt: run sensitive-data detection and redaction before writing to storage; enable encryption at the storage layer and enforce fine-grained access control.
• Traceability and correctability: every memory must be traceable back to its source session; provide view, edit, and delete interfaces and support instant corrections of the “forget this one” kind.
• Retention period and right to deletion: set an explicit retention period, and when a user asks for deletion, clear the core profile, the index, the archive, and any caches at the same time.
• Permission isolation: in multi-tenant or multi-user scenarios, memory must be strictly isolated by user and by project to prevent context overreach and cross-account memory leakage.

8. How to Evaluate Whether a Memory System Is Good

It is worth defining at least four categories of metrics, with task success rate as the north-star metric:

• Memory quality: number of conflicting entries, share of entries pending confirmation, share of stale entries;
• Retrieval effectiveness: the hit rate of injected entries, that is, the proportion the model actually cites or that influences the output;
• Budget efficiency: memory tokens consumed per request, and cost per completed task;
• Task effectiveness: completion rate, first-pass rate, and manual correction rate on the same regression task set.

For the evaluation method itself, maintain a fixed set of regression tasks, and after every change to the extraction prompt, the consolidation strategy, or the injection budget, re-run them and compare results to see whether the change means “remembering more” or “using memory more accurately.”

Conclusion

“Give the agent unlimited memory” sounds appealing but does not work in practice. Once memory is split into three stages — in-session notes, offline consolidation, and injection budget — the problem becomes measurable and optimizable: in-session notes determine the quality ceiling of memory, offline consolidation determines its tidiness and trustworthiness, and the injection budget determines whether memory actually pays off at the critical moment.

As for concrete limits like “roughly 200 lines, 25 KB,” each team can adjust them to fit its own models and tasks. What is really worth borrowing is the trade-off logic behind them: do not chase “everything,” chase “the right thing.” First make the agent remember correctly, then make it remember durably and use memory well.