System Prompt Bloat: Consolidating a 200,000-Character Prompt with Context Engineering

System prompts that grow past 200,000 characters are rarely the result of a deliberate design choice. The growth comes from hundreds of small, reasonable-looking edits, each one adding a rule, a usage note, or a definition to address one specific case. The result is a prompt that behaves less like a configuration file and more like an encyclopedia. This article looks at why prompts swell, what that scale actually costs, and how to consolidate one through context engineering. The focus is on a resident core with a lightweight index, on-demand retrieval of memory, and prompt governance run the way code is run.

Why System Prompts Swell into Encyclopedias

Inflation is almost never a decision. It is the cumulative result of a simple path. A bad case shows up in production, so the team adds a rule to close it. A feature is hard to use, so the team adds usage instructions. A group of users complains, so the team adds a set of boundary definitions. Every individual edit is cheap and delivers immediate results. As a result, nobody owns the total size.

Three structural causes are worth calling out separately:

1. Failure-driven additions. Rules are patches, and patches have no removal mechanism. Six months later, the prompt may contain three overlapping or even contradictory instructions, and nobody knows which of them is still necessary.
2. Responsibility drift. The safety, product, interaction design, and localization teams all have requirements to add. The prompt is the one shared file that anyone can edit, so it ends up serving four roles at once: configuration, documentation, specification, and test cases.
3. No measurement. If nobody tracks metrics such as character count, token count, instruction-following rate, and cache hit rate, growth goes unnoticed and unbounded.

The Real Cost of 200,000 Characters

A resident context of that size does not just cost “a few more tokens per request.” Several things degrade at the same time:

• Economic cost. The prompt is billed again on every request, and high-frequency conversations turn that into a multiplier.
• Latency. A longer prefix means longer prefill time, which directly raises time to first token.
• Lost prefix cache hits. A prompt that changes frequently is hard to match against the prefix cache, which raises the effective cost per unit of work. This point is easy to overlook.
• Instruction dilution. As context grows, information in the middle is less likely to be used effectively. The model becomes less likely to follow any one specific rule, producing instructions that were “written but never took effect.”
• Conflicts that can’t be diagnosed. When two rules clash, troubleshooting means reading through 200,000 characters, which is close to unmaintainable.
• Compliance and audit gaps. When safety policies are buried in prose paragraphs, you cannot show which rule governs a given behavior. Change impact analysis is equally difficult.

What the Bloat Actually Contains

Grouping the prompt by responsibility shows a fairly typical composition:

Layer	Content	Change Frequency	Should It Be Resident?
Identity & capability boundaries	Role, scope of capabilities, refusal principles	Very low	Yes
Safety red lines	Protection of minors, mental health crisis intervention, dangerous information, degradation over long conversations	Low	Yes (must be minimal and non-overridable)
Tool contracts	Input parameters, return values, and retry semantics for each tool	Medium	Inject on demand
Product & interface descriptions	Differences across clients, coding assistants, office document and spreadsheet assistants, and browser-integrated assistants	Medium	Retrieve
Memory rules	When to read or write, file format, directory structure, privacy boundaries, conflict handling	Medium	Rules resident, data on demand
Style & format	Language, tone, length, citation style	High	Make structured

The memory system is by far the biggest source of growth. Its specification runs to more than 200 lines and is detailed enough to read like an internal product spec. Its core idea, however, is interesting. It models cross-session working memory as a file system. Writes happen not because the user asked for them, but because “a future version of myself will need this context.”

Key Design: A Resident Index Plus On-Demand Reads

What is worth borrowing is not the length of that specification. It is the loading strategy. Only three things are injected on each turn:

• The user profile itself: stable identity, such as role, responsibilities, and long-term preferences.
• Interaction preferences: answer length, formatting, and language habits.
• A memory listing: the path, summary, aliases, and source of every memory file, but not the file contents.

Once the model sees the listing, it decides which entries might be relevant and reads those files on demand.

def build_context(user_msg, memfs, tools):
    ctx = [IDENTITY, SAFETY_MIN]                 # resident core, target < 3k tokens
    ctx.append(memfs.read("/profile.md"))        # stable identity
    ctx.append(memfs.read("/preferences.md"))    # interaction preferences
    ctx.append(memfs.render_listing())           # path + summary + aliases + source only
    # Retrieve tool and domain specs by intent instead of keeping all of them resident
    for spec in tools.select(user_msg):
        ctx.append(spec.render())
    return ctx
# Key constraint: no denial before retrieval.
# The model must check the listing first, then call memory_read(path), and only then answer.

One hard rule matters as much as the loading strategy. Before answering “I don’t have that information,” the model must check the listing and try to read the candidate files. Without it, users get the worst possible experience. For example, a user settled on a technology choice three months ago. The assistant still claims not to know about it, simply because the file was never injected into the context.

Visible in the index:
/areas/mobile-app.md      Cross-platform client architecture decisions and constraints
/people/reviewer.md       Code review preferences and focus areas
/topics/release.md        Release process and canary rollout strategy
User: "What architecture did we settle on for the client project?"
Expected behavior: read("/areas/mobile-app.md") first, then answer from the file contents.
Forbidden behavior: answer "no matching record found" because the file contents were never injected.

The underlying idea is to turn the prompt from a full manual into a runtime assembly that ships with an index.

Organizing Memory Files

In the leaked specification, memory is stored in directories organized by type, using the stability of the information as the dividing criterion. That method can be reused directly:

• Identity: name, title, role, and long-term responsibilities. The test is simple: will this likely still be true three months from now? If yes, write it to an identity file. If no, don’t.
• Domain: architecture decisions, constraints, and known pitfalls for a specific project or topic.
• People: preferences and focus areas of collaborators.
• Topics: conclusions from temporary discussions, each with explicit expiration conditions.
• Preferences: answer length, format, language, and citation habits.

Four engineering conventions are needed to support this:

1. Metadata. Each file carries a summary, aliases (for fuzzy-match retrieval), a source (user statement, tool output, or model inference), and a last-updated timestamp. The source determines how much the information is trusted.
2. Conflict handling. Store stable facts in the identity file and volatile facts in sub-files. When the two conflict, the side with the more recent update and the more trusted source wins. Confirm the change with the user rather than overwriting silently. Conclusions the model inferred must never be promoted directly into identity facts.
3. Privacy boundaries. Redact data before writing. Secrets such as keys, tokens, government IDs, and exact addresses never enter storage. Set visibility by category, offer view, export, and delete options, and define retention periods per category.
4. Concurrency consistency. Several entry points, such as chat, coding, and office assistants, may share the same files and even modify them at the same time. The system needs optimistic concurrency control or per-file write locks, plus an append-only write log, to stop read-modify-write cycles from overwriting each other. This layer cannot be enforced through the prompt; only the storage layer can guarantee it.

Governing Prompts as Engineering Artifacts

The only sustainable way to contain prompt bloat is to treat the prompt like code.

Layers and priority. Define priorities explicitly so that higher layers override lower ones when they conflict:

[P0] Safety and compliance red lines    — no downstream content may override
[P1] Identity, capability boundaries, and tool contracts
[P2] User profile and memory facts
[P3] Style, format, and tone

A single source of truth. Each rule is allowed to exist in exactly one place. Product capability descriptions should be self-describing through the tool schema, not written out by hand in the prompt. Together, these two practices remove most of the redundancy.

Modularity and versioning. Split the prompt into independent fragments by responsibility. Each fragment gets its own version number and owner, changes go through review and canary rollouts, and past versions are kept so they can be rolled back.

Deletion-driven evaluation. The key to reducing bloat is being willing to delete. Run an ablation experiment for each rule: remove it, run a fixed regression set, and if the pass rate doesn’t drop, delete the rule for good.

rule: memory_before_deny
cases:
  - query: "What architecture did we settle on for the client project?"
    memory: ["/areas/mobile-app.md"]
    expect: [memory_read, answer_from_memory]
  - query: "Which release strategies have we discussed?"
    memory: ["/topics/release.md"]
    expect: [memory_read, cite_source]
forbidden:
  - deny_without_read

Metrics to watch. Track the resident prefix token count and its trend, the prefix cache hit rate, time to first token, and the follow rate for individual rules. Also track the number of near-duplicate semantic entries after clustering, the memory read hit rate and the staleness of memory, and the context cost per task. Without this dashboard, governance efforts tend to slide back within three weeks.

Putting It into Practice

None of this is exotic from an engineering standpoint:

• Configuration assets: prompt fragments and memory files are configuration assets. Store them in versioned form in object storage or a configuration center, and use a secret management service for access control.
• Asynchronous writes: persist memory writes asynchronously through a message queue so they don’t block the conversation.
• Continuous observation: track the call chain, token usage, and shifts in follow rates through logging services and monitoring alerts.
• Safe rollout: roll changes out to a small share of traffic on a container platform, and roll back quickly if anomalies appear.

In platform-based environments such as Baidu AI Cloud, generic capabilities like object storage, configuration management, logging, monitoring and alerting, and container platforms can be reused directly instead of being built from scratch.

Checklist

• Can the resident core be squeezed down to a few thousand tokens? If not, responsibilities have not been separated into layers.
• Are product and tool descriptions retrieved on demand rather than injected in full?
• Is “deny without reading first” explicitly forbidden?
• Does every rule have a single owner, a version number, and a designated owner?
• Does memory carry source labels, expiration conditions, redaction, and a deletion mechanism?
• Does the storage layer protect against concurrent writes from multiple entry points?
• Are there rule-level evaluations and regular deletion experiments?
• Are tokens and follow rates monitored continuously?

The conclusion is straightforward. A system prompt derives its value from clear instructions and explicit priorities, not from the total volume of information. 200,000 characters is not proof of capability; it is technical debt that has not yet been paid off. Rebuilding the prompt from an encyclopedia into a layered, runtime-assembled configuration is currently one of the most worthwhile refactors in context engineering.