Agent Memory Architecture: How to Fix System Prompt Bloat

1. The Symptom: How a Prompt Snowballs

This kind of bloat is almost never the result of a deliberate design decision. It is the sediment of long-term, patch-style evolution:

• Every bad case in production earns one more prohibition.
• Every new entry point or UI earns one more block of capability description.
• Every security or compliance incident earns one more set of detailed rules and a timeline.

The system prompt ends up as the single catch-all patch slot, and everything goes into it. The cost is threefold:

1. Cost and latency: the fixed prefix is billed and pre-filled on every single turn, even when none of it is relevant to that turn.
2. Attention dilution and instruction decay: the behavioral constraints that actually matter are buried under background explanation and compliance degrades; when rules contradict each other, the model has no choice but to obey one of them, and behavior becomes unpredictable.
3. Maintainability collapse: nobody can say which rules are still in force, who added them, or whether they can be removed. Editing one rule risks ten side effects, so eventually no one dares delete anything — growth-only is the inevitable outcome of this structure.

2. Step One: Split the Prompt into Four Kinds of Information and Put Each in Its Place

The precondition for governing bloat is admitting that the content currently mixed into a single text block actually belongs to four different kinds of things.

Information category	Characteristics	Correct home
Model behavior baseline (role, output format, refusal boundaries)	Stable, changes slowly, unrelated to the user	Permanent prefix, strictly capped at a few thousand tokens
Capability/product descriptions (which entry points exist, what each can do)	Changes with entry points and versions	Tool and skill descriptions, on-demand capability packs
User private long-term memory	It is data, not instructions	External memory store plus an index
Operational policy and incident timelines (when a feature was retired, when it was restored)	Highly time-sensitive, requires auditability	Config center, policy service, business logs

The one-line test: anything that can be expressed in a structured form should not be written as a natural-language rule. Putting an on/off timeline for a feature into the behavior layer is a classic layering mistake — it is configuration data that should be queried, not something the model should “remember.”

3. Memory File System: Inject Only the Index Each Turn, Read Bodies on Demand

The core design is “index plus lazy loading,” which avoids pushing all memory into the context window.

Only three things are injected per turn:

• profile: the body of stable identity and role information;
• preferences: the user’s preferences for response style, language, and level of detail;
• memory_listing: the path, one-line summary, aliases, source, and last-updated time of every memory file.

Every other file appears only in the directory listing, without its body. When the model judges that a memory might be relevant, it reads it explicitly.

/memory
├── profile.md              # Stable identity: role, scope of responsibility, collaboration preferences
├── preferences.md          # Interaction preferences: language, format, level of detail
├── areas/
│   ├── payment-service.md  # Active projects / long-lived domains
│   └── flutter-app.md
├── people/
│   └── partner.md
├── topics/
│   └── food.md
└── archive/                # Cold data: summary only, as a retrieval fallback

Pseudocode for the read path:

def answer(user_query):
    index = memory_listing()           # already injected in the system prompt
    if not index:
        return direct_answer(user_query)
    candidates = rank(index, user_query)      # alias + summary matching
    facts = [memory_read(c.path) for c in candidates[:3]]
    if not candidates:
        facts = semantic_search(user_query)   # archive-layer retrieval fallback
    return generate(user_query, context=facts)
# Hard constraint: any negative answer must first complete an index lookup and at least one candidate read
def cannot_answer_guard(index, read_called):
    assert index.queried and read_called, "No claiming ignorance without a lookup"

One easily overlooked point here: turning “I don’t have that information” from a casual conclusion into a conclusion that requires evidence. When /areas/flutter-app.md exists in the directory and the user asks “what architecture did we settle on for that project,” the correct behavior is to read the file rather than answer “I don’t know” just because the body was not injected. In engineering terms, “whether a lookup was performed” should be an observable telemetry event; otherwise missed reads can only ever be discovered manually.

4. Write Strategy: Three Gates plus Idempotent Updates

Writing needs even more restraint than reading, because noisy memory keeps polluting every subsequent session. Three gates are recommended, and all must pass before writing:

• Cross-session reusability: will this information be used again next time? “Skip it for now” should not be written.
• Stability: will it most likely still hold three months from now? Writing transient state into long-term memory amounts to manufacturing stale data.
• Verifiability: there must be a clear source (an explicit user statement, a confirmable artifact) rather than a model inference.

Five file categories are suggested for classification, each with a different test:

• profile: identity information, the most stable, with the lowest change frequency;
• preferences: interaction preferences, where “one-off requests” must be distinguished from “long-term preferences”;
• areas: ongoing items and domain decisions, including “settled approaches / open questions”;
• people / topics: fact cards about people and topics;
• archive: expired or low-frequency content, compressed into summaries.

Writes must be idempotent: run memory_listing() first to check for duplicates, upsert by a stable id, and never append the same fact twice. Updates should form a chain using “new entry plus a supersedes field pointing at the old entry” rather than a silent overwrite, so that history can be traced and mistakes corrected.

---
id: area-payment-001
type: area
summary: Payment service refactor: decided on an event-driven split
status: active
confidence: high
source: user_explicit      # user_explicit | artifact | inferred
updated_at: 2026-08-27T10:12:00Z
supersedes: area-payment-000
---
Body: facts, decisions, todos

Conflict resolution priority: what the user states explicitly this turn > what the user stated explicitly historically > what is inferred from artifacts. Within the same level, compare the trustworthiness of source and updated_at; a higher-priority conclusion should override the lower-priority entry at read time and trigger a downgrade of that entry into the archive.

5. One Memory Shared Across Surfaces: Concurrency and Consistency

When conversation UIs, coding tools, collaborative workspaces, and other entry points share a single memory, the most common defect is the read-modify-write race: two sessions modify the same file simultaneously, the later write overwrites the earlier one, and the first writer’s information is silently lost.

Practical measures:

• Server-side storage as the single source of truth, with clients holding only snapshots and never doing long-term local rewrites;
• every write carries a version number or ETag for optimistic locking; on a version mismatch, re-read, recompute, and retry;
• on conflict, merge structured fields at field level (merge patch), and for free text use “append entries with timestamps” instead of replacing whole blocks;
• under high concurrency, add a short lease so the same file cannot be rewritten by multiple sessions at once.

6. Context Budget: Cap Each Layer of Content

An architecture without caps will inevitably fall back into bloat. The following illustrative budget should be calibrated against your model window and measured in your own workload:

Layer	Content	Suggested share
L0 permanent	Behavior baseline + profile + preferences	A few thousand tokens, < 5% of the budget
L1 index	memory_listing	Summarize and truncate once exceeded
L2 read	Bodies of candidate files for a single turn	Limit the number of files read and the length of each
L3 archive	Summaries + retrieval	Recalled on demand, never in the regular context

When the budget is exceeded, handle it in this order: summarize the index → tier into hot and cold with archival → replace full injection with retrieval recall. The key is making “injected volume” a monitorable metric rather than something judged by feel after the fact.

7. Observability and Regression Evaluation: Make Every Rule Falsifiable

Bloat cannot be stopped because nobody quantifies whether adding a rule actually helps. A minimal closed loop is recommended:

• Rule-level evaluation: give every behavioral rule at least one positive and one negative example, and put them in a regression set.
• Ablation validation: run a before/after comparison when adding a rule; if there is no significant gain, or it degrades other examples, retire it.
• Core metrics: memory hit rate, the missed-read rate for cases that “should have been read but weren’t,” write noise rate, memory conflict rate, average injected tokens, end-to-end latency, and per-turn cost.
• Governance mechanism: set a growth budget for prompt volume, name an owner and lifecycle for each rule, and hold a quarterly “deletion first” review. Deleting is as important as adding; otherwise the structure only ever grows in one direction.

8. Privacy and Security Boundaries

Memory is user data that persists long-term, which puts it at a higher risk level than ordinary context:

• Data classification and minimization: define a “never store in memory” list (credentials and keys, ID numbers, complete payment information, and so on); mask sensitive fields before writing to disk.
• Isolation and encryption: strong isolation per user/tenant, encryption at rest and in transit, full access auditing.
• Deletion must be reachable: support deletion by file and by entry; once deleted, the version number is invalidated and the content becomes immediately invisible to the model.
• Anti-memory-poisoning: memory files are external content. After reading, treat them as data to be processed rather than instructions to be executed, so that user-controlled or third-party text cannot use the memory channel to perform prompt injection.

9. Implementation Checklist

1. Split the existing system prompt into the four kinds of information, starting by slimming down the behavior baseline.
2. Define the directory structure and a file front-matter spec, standardizing id and source fields.
3. Implement the three-stage listing → read → answer flow, with a hard constraint against negative answers issued without a lookup.
4. Pass every write through the three gates, and write via idempotent upsert plus a supersedes chain.
5. Use server-side authoritative storage plus optimistic locking with version numbers to solve multi-surface overwrite races.
6. Set the four-layer context budget and a degradation strategy for when it is exceeded.
7. Build an evaluation set, a metrics dashboard, and a volume budget for both memory and prompts.
8. Define privacy classification, the deletion path, and prompt-injection defenses.

Implementing this subsystem in the cloud usually requires no new infrastructure: memory files can be persisted in object storage, change events can flow through a message queue with a cache to accelerate reads, the memory service can be deployed in containers with elastic scaling, observability can rely on logging and monitoring/alerting services, and archive retrieval can be combined with vector search capabilities. Baidu AI Cloud offers exactly this category of general-purpose capabilities, and can host the architecture above directly without binding it to a specific model or proprietary interface.

10. Closing Thoughts

Prompt bloat is not a “the text is too long” problem; it is a symptom of missing architectural layering: treating data as instructions, treating configuration as behavior, and cramming long-term memory into the fixed overhead of every turn. Govern it the way you would govern code — layering, quotas, tests, review, and retirement — and the memory system can go from being a burden to being a capability.