1. What Happened: A System Prompt That Grew to 200,000 Characters

A system prompt allegedly belonging to a frontier LLM vendor recently circulated in developer communities. By community counts it runs about 200,000 characters, while the same vendor’s previous flagship model sat at roughly 120,000. One version apart, and the prompt had grown by about two-thirds.

From the leaked content, the growth was not scattered padding but systematic accumulation:

• Operational history inlined into the prompt. The prompt contained a complete timeline of a service being temporarily pulled and then restored in a particular region. Information of this kind belongs in an incident record, yet it was written verbatim into a system prompt that loads on every single conversation.
• Product-surface documentation. Coding assistants, office collaboration tools, browser automation, spreadsheet and presentation generation, design tools — each carried detailed feature and usage descriptions.
• Retrieval and output specifications. Covering search, file handling, artifact generation, long-form writing, citation formatting, and tool invocation.
• Expanded safety policy. Minors protection, mental-health crisis detection, weapons and dangerous substances, the boundaries of offensive and defensive cybersecurity, and management of very long conversations were all refined further.
• Style and interaction rules. Large volumes of very specific tone, wording, and interaction conventions.
• Cross-session memory specification. One specification named <memory_filesystem> runs more than 200 lines and reads like an internal product spec. It is the most visible part of the bloat.

The essence of the bloat is patch-style evolution. Every category of problem earns one new rule, every new requirement earns one new block of instructions, and there is no mechanism for consolidating or pruning. So the prompt gradually shifts from “manual” to “encyclopedia.” Even a temporary compliance-policy adjustment and its reversal in one region ended up in the resident prompt. That looks less like guidance for the model’s current behavior and more like background knowledge added for its own sake.

2. Where the Bloat Comes From: Four Structural Causes

1. Patch-style fixes. When a model produces a wrong answer or crosses a boundary in some scenario, the most direct remedy is to append a rule. The problem is that a rule can only constrain the specific cases someone thought of, not the variants of that case. Each time a new variant shows up in testing, another rule goes in, and the rule count grows monotonically.

2. Product-line reuse. Once coding, office collaboration, browser automation, and multimodal generation are integrated into a single assistant, every product line pushes its own UI documentation into the shared prompt. Even when a user is only using one of those features, the rest still consume context.

3. Safety rules only ever grow. Safety items usually carry review sign-off in the process, so the psychological barrier to deleting or merging them is far higher than the barrier to adding. The safety section becomes a one-way region of the prompt: nothing leaves.

4. The “free resources” illusion created by larger context windows. When the window grows from tens of thousands of tokens to hundreds of thousands or a million, “one more paragraph” is nearly imperceptible within a single turn. But in production the system prompt is copied into every request. Taking 200,000 characters as the example:

• At a rough 4 characters per token for English, that is around 50,000 tokens;
• If a large share of the content is Chinese characters, the real token count is typically higher;
• The exact conversion depends on the tokenizer in use.

There is a further hidden cost: every edit to the system prompt invalidates the prefix cache, and the more often it changes, the lower the hit rate.

3. The Cost of Bloat: From Token Spend to Behavioral Consistency

1. Cost and latency scale linearly. The system prompt sits at the very front of the context and is part of the prefix that every request must reprocess, so compute grows linearly with its length. A long prompt raises time-to-first-token and squeezes the budget left for conversation history, retrieval results, and tool return values. In long sessions, once total length hits the window limit, what gets truncated is often the user’s own context.

2. Attention dilution. Research indicates that models use information in the middle of a context markedly less effectively than information at the start or end — commonly called the “lost in the middle” effect. The longer the prompt, the lower the probability that any single rule is followed reliably, and the easier it becomes for rules to interfere with and contradict one another. A prompt does not hard-constrain model behavior; it is a set of probabilistic nudges. Past a certain number of rules, marginal returns fall off quickly and can turn negative.

3. Maintenance cost and regression risk. A 200,000-character prompt cannot be reviewed by reading it end to end. When one rule is changed, it is hard to tell whether it conflicts with any of the dozens of older rules, and hard to tell what the change does to overall behavior. Without an accompanying evaluation suite, every edit is effectively a blind edit.

4. Spillover security risk. A system prompt contains product architecture, safety policy, and boundary definitions. If it leaks, the defensive design is effectively published to attackers, who can then craft bypasses that target it specifically. Concentrating a large volume of sensitive policy as plaintext in a resident prompt amplifies the blast radius of a leak.

5. An implicit model split. The leaked content also contained a rule that a separate model takes over when a high-risk request arrives. From an engineering standpoint that is routing and orchestration logic, which belongs in the application layer rather than in the prompt.

4. What’s Worth Borrowing: Organizing Cross-Session Memory as a File System

Even though the overall scale is out of control, the memory design inside this prompt offers a pattern worth studying: instead of stuffing all memory into context, organize memory as a file system and read it on demand.

4.1 Inject only an index each turn, not the full text

Each turn the system injects three kinds of content:

• <profile>: the body of /profile.md, the user profile;
• <preferences>: the user’s preferences for how answers are delivered;
• <memory_listing>: the path, summary, alias, and source of every memory file.

All other memory files appear only in the directory listing. Once the model decides a file might be relevant, it calls a memory_read-style interface to fetch the body. This is effectively two-tier storage for memory:

• Resident cache: identity profile and preferences — small, high value;
• Secondary index: the memory directory;
• On-demand loading: the body of each individual memory.

4.2 Layered writes: decide whether something is worth remembering first

Memory is organized into five kinds of files, with /profile.md holding stable identity information such as name, role title, role, organization, and long-term areas of responsibility. The write criterion is: will this information still be true in three months?

That criterion is really a way of assigning memory a lifecycle. Stable information moves into the profile layer and is retained long term, while volatile information stays at the session layer or is filed under a topic file, so stale information stops polluting the context over and over.

4.3 Search first, then answer: eliminating “false forgetting”

The prompt requires explicitly that before answering “I don’t have that information,” the model must check the directory listing and read any files that could be relevant. If the listing contains /areas/mobile-app.md and the user asks about an architecture decision made earlier in that project, the model should read that file first rather than claiming ignorance simply because the body was not directly injected. Sketch of the logic:

import re
def answer(question: str, listing: list[dict]) -> str:
    """
    Retrieve relevant memory from the directory listing only, and read the
    body after a hit, to avoid "body not injected => answer 'I don't know'"
    false forgetting.
    """
    candidates = [m for m in listing
                  if re.search(m["query_pattern"], question, re.IGNORECASE)]
    if not candidates:
        return "(no memory hit; treat as no relevant memory)"
    contents = {m["path"]: memory_read(m["path"]) for m in candidates}
    return generate_answer(question, contents)

4.4 Lessons and limitations

This mechanism solves “can’t remember enough,” but it does not solve “writes too much.” A memory specification of more than 200 lines is still part of the prompt, and the more detailed it is, the higher the runtime cost. Three further risks exist:

• Concurrent writes. Multiple surfaces share the same memory files and may modify them at the same time, so version and conflict-handling policies must be defined clearly.
• Privacy boundaries. It must be clear which content may be written and which should not be.
• Retrieval quality. The summaries and aliases in the directory listing determine recall; if they are written poorly you end up with files that exist but cannot be read.

5. How to Govern It: Make System Prompts Leaner and More Accurate

5.1 Layered architecture and token budgets

Layer the prompt by stability and set a budget ceiling for each layer. When a layer exceeds its ceiling, trim that layer rather than expanding the whole:

prompt_budget:
  total_tokens: 6000          # total system prompt budget (illustrative; tune per scenario)
  layers:
    - name: identity           # L0 identity and boundaries: role, goals, hard red lines
      budget: 300
    - name: contract           # L1 capability contract: tool list, parameter constraints, output format
      budget: 1200
    - name: policy             # L2 safety and compliance: short, tiered, verifiable
      budget: 800
    - name: style              # L3 style and interaction: keep only universal rules, defer volatile ones
      budget: 300
    - name: dynamic            # L4 dynamic injection: user profile, preferences, retrieved snippets, memory listing
      budget: 3400

Put token counts into monitoring and alert on them. A budget is not a suggestion but a hard constraint. Governance only gets traction once the cost becomes visible.

5.2 Separate a stable prefix from a dynamic tail

For inference optimization, mainstream model services commonly support prefix caching, reusing the computed result of an identical context prefix. To maximize hit rate, keep stable rules byte-for-byte unchanged and push anything variable to the end of the prompt:

┌─────────────────────── stable prefix (long-lived, cacheable) ───────────────────────┐
│ L0 identity & boundaries │ L1 capability contract │ L2 safety policy │ L3 general style │
├────────────────────────────── dynamic tail (changes per turn, not cached) ──────────┤
│ L4a user profile / prefs │ L4b retrieved policy snippets │ L4c memory listing │ L4d current task │
└────────────────────────────────────────────────────────────────────────────────────┘

Also control the cadence of changes tightly. One prompt edit invalidates the entire prefix cache, and the more frequent and earlier in the prompt a change lands, the greater the cache loss.

5.3 Externalize rules: don’t write down what can be looked up, don’t rely on prose for what can be validated

First, turn “documentation” into “searchable reference material.” UI feature descriptions, product-line overviews, and step-by-step operational guides belong in a knowledge base and should be injected on demand via retrieval. A reasonable routing table by “rate of change × share of relevant requests × text length”:

Content type	Change rate	Share of relevant requests	Text length	Recommended placement
Core identity and red lines	Low	100%	Short	Resident system prompt
Output format conventions	Medium	High	Short	Resident, plus structured-output validation
Tool invocation specs	Medium	Varies by tool	Medium	Inject only the compact definition of tools currently available
UI feature documentation	Medium	Low	Long	External knowledge base, retrieved on demand
Long-form writing rules, citation specs	Medium	Low	Long	Inject only after the matching intent is detected
Historical events and temporary operational notes	One-off	Very low	Medium	In principle, never in the prompt

Second, if a tool can solve it, don’t write it as prose. Rather than describing citation behavior in long paragraphs, expose a structured citation interface and let the system render it uniformly.

Third, replace verbal constraints with structured output. Constrain format with a JSON Schema or function calling, so the model only has to satisfy a machine-checkable spec instead of reading lengthy instructions.

Fourth, push safety policy down into executable modules. Grade requests in the application layer with a lightweight classifier, then let a separate safety module block them. Do not put policy details and incident timelines into the resident prompt.

5.4 A unified prompt base

Extract the shared specification into a common base so each product surface keeps only its differentiating delta:

final_prompt = compose(
    base_prompt,                       # identity, tool contract, safety policy (one version, one eval suite)
    product_delta[current_surface],    # deltas for the coding assistant / collaboration suite / browser automation, etc.
    retrieved_docs[query],             # retrieved policy documents
    user_memory[listing, profile],     # memory listing and user profile
)

This avoids the same rule being written repeatedly by multiple surfaces and makes unified evaluation and staged rollout much easier.

5.5 Build the memory system as a subsystem

For assistants that need cross-session memory, design and manage the memory system as an independent subsystem:

• Read/write protocol. Define when to write, which class of file to write to, and the deduplication, update, and expiry policies.
• Conflict handling. For concurrent edits across endpoints, use version numbers or timestamps with a last-write-wins rule, and keep a revision history.
• Privacy controls. Inform the user before writing, let users view, edit, delete, and export memory, and cap what may be written for sensitive categories.
• Quality assurance. Periodically tidy, merge, and compress low-value memories so the directory does not grow without bound and flood retrieval with noise.
• Specification slimming. Reduce the memory spec to bullet-point essentials and move details into the knowledge base. Shorter specs are followed more often and consume less context.

5.6 Build an evaluation and regression suite

Map each rule to “rule — test case — metric” so implicit rules become measurable behavior:

# Each rule has at least one positive and one negative example, included in regression tests
EVAL_CASES = [
    {
        "rule_id": "MEM-001",  # must retrieve memory before answering "I don't know"
        "setup": {"listing": ["/areas/mobile-app.md"], "body": "layered architecture"},
        "input": "What architecture did we pick for my mobile app project earlier?",
        "assert": lambda out: "layered architecture" in out and "i don't know" not in out.lower(),
    },
    {
        "rule_id": "MEM-002",  # should not force an association on an irrelevant keyword hit
        "setup": {"listing": ["/topics/food.md"]},
        "input": "Help me debug why a production API keeps timing out",
        "assert": lambda out: "food" not in out.lower(),
    },
    # ... the same approach applies to safety tiering, output format, tool selection, and so on
]

Run the regression cases before every prompt change, then track these metrics after release:

• average context length per request;
• rule adherence rate per rule;
• retrieval recall;
• time-to-first-token;
• user negative-feedback rate.

For platform-level deployment, general-purpose capabilities from cloud platforms such as Baidu AI Cloud — model services, logging services, and monitoring and alerting — can support call-chain tracing, token usage statistics, and observation of prompt effectiveness. On that foundation you can close the loop of “change — evaluate — staged rollout — rollback,” so every prompt edit rests on evidence.

6. Conclusion: Prompt Engineering Has Entered the Context Budget Era

From 120,000 to 200,000 characters, the issue is not how much the prompt says but how it grew: solving problems by appending rules and covering requirements by stacking descriptions. The system prompt is turning from a “manual” into a core configuration artifact of an LLM product. Its quality directly determines cost, latency, safety, and consistency, so it deserves the same treatment as code — version control, code review, automated testing, and continuous refactoring.

Of everything in that leak, the memory rule “check the directory first, then read the file” is especially relevant to teams building assistants and coding tools. The user’s information is usually already there; what the model lacks is often not knowledge but the habit of retrieving it at the right moment.

The takeaway for the industry fits in one line: spend the budget where it matters, and put rules where they can be verified.