LLM Context Window Memory Structure: A Layered Breakdown and Practical Optimization

1. Key takeaway first: the context window is not an “input box,” it is shared memory contested by multiple parties

When many teams evaluate “how long a document a model can read,” they tend to equate the context window with the capacity of a user input box. But in agent-based applications, this window behaves much more like a block of memory shared by multiple tenants: system instructions, tool definitions, conversation history, tool return values, and intermediate reasoning results all request space inside it. When any one party expands without restraint, what gets squeezed out is the last tenant in line — the thing the user actually wants to say in their current turn.

To take a rough, industry-published order-of-magnitude estimate as an example, the window allocation of a certain class of AI coding tool looks roughly like this: system prompts take about 1%~2%, system tools (including externally connected tools) take about 8%, while skill descriptions, tool descriptions, conversation history, and other agent-side context can easily consume 60%~70%, leaving only a very small slice for the user’s actual input. These ratios are not constants; they shift dramatically with the number of tools, the number of conversation turns, and the volume of per-turn tool output. But the structural fact they reveal is stable: system-level instructions take up a very small share, and the real source of bloat is the intermediate-state data produced by the conversation and the tools.

Once you understand this, the optimization direction becomes clear: don’t just fixate on “writing the system prompt shorter” — govern the token-production mechanism across the entire chain.

2. A four-layer budget model: where a request’s tokens are spent

Layer the input of a single agent request by source, and you get a clear budget table (the ratios below are illustrative magnitudes used to explain the structural relationship):

Layer	Main content	Typical share (illustrative)	Growth characteristics
L1 System instruction layer	Role setup, safety constraints, output format, behavior guidelines	1%~3%	Static, changes rarely, cacheable
L2 Capability and tool layer	Built-in tools, parameter definitions and descriptions of externally connected tools, skill manuals, call examples	5%~15%	Grows roughly linearly with the number of tools
L3 Agent working layer	Multi-turn conversation history, file and code echoes, command output, retrieval snippets, subtask results, intermediate reasoning	50%~75%	Accumulates with duration and tool-call count, hardest to control
L4 User current input layer	The real request of this turn, supplementary material	Remainder	Passively bears the squeeze from the first three layers

The key judgment is this: L1 and L2 are budgetable fixed costs, L3 is a variable cost that accumulates over time, and L4 is the only layer that directly determines task quality yet is the most easily compressed. Any governance approach is fundamentally doing three things: turning L1/L2 into stable, cacheable constants, slowing down the growth rate of L3, and thereby giving the window “back” to L4.

3. Layer-by-layer attribution: where the bloat actually comes from

3.1 System instruction layer: many constraints but slow growth

This layer usually contains role definitions, refusal and compliance boundaries, output format requirements, code style conventions, and so on. Its characteristics are many entries, short per entry, and low update frequency. The mistake many teams make here is “additive stacking”: every time they hit a production issue, they append a rule to the system prompt, and half a year later the system instructions have grown from 500 tokens to 4000. Although the absolute value is still not large, it breaks prefix stability — as soon as any change occurs in this layer, all subsequent cache hits are invalidated, and cost and time-to-first-token rise together.

3.2 Capability and tool layer: a quantity-driven cost

This is the most easily underestimated layer. Each tool entering the context brings at least three overheads: the structured definition of parameters, the natural-language description aimed at the model, and several call examples. A single tool may be only a few hundred tokens, but when internal tools and externally connected tools accumulate to dozens or even hundreds, this layer’s volume will exceed the system instructions by an order of magnitude.

More troublesome is the marginal effect: the more tools there are, the higher the model’s selection difficulty, the higher the probability of misselection and repeated calls, and thus the more invalid tool returns are produced — the tool layer not only takes up space itself but also indirectly amplifies L3.

3.3 Agent working layer: the biggest black box

The bloat of this layer comes from several compounding factors:

• Linear accumulation of conversation history: under multi-turn tasks, each turn’s questions and answers are fully retained; doubling the turns doubles the history.
• File and code echoes: reading a medium-sized source file can be thousands of tokens, and reading ten files in one task approaches a third of the window.
• Verbatim backfilling of command and interface output: build logs, test reports, and list-query results often contain many redundant lines irrelevant to decisions.
• Retrieval and subtask results: vector-retrieval snippets and subagent report text are usually merged in by “stuffing the whole segment back into the main context.”
• Intermediate reasoning and tool-call traces: each step’s call parameters and return values take up space, and even if later proven useless, they are not automatically reclaimed.

3.4 User input layer: the squeezed party

When L1~L3 already occupy most of the window, the user’s usable input is compressed. The direct consequence is that the task description is truncated or diluted, the model starts “guessing” the user’s intent, and this manifests as missed requirements, repeated follow-up questions, and a cliff-like drop in overall quality once the context tops out.

4. Make the budget explicit: build a token ledger first, then talk about optimization

You must be able to measure before you optimize. It is recommended to run layer-by-layer statistics on the application side for every request, forming a traceable token ledger:

def build_request(system_prompt, tools, history, user_input, counter):
    """Assemble a request by layer and output a layered token ledger. `counter` is a generic token-counting capability."""
    layers = {
        "L1_system": system_prompt,
        "L2_tools": render_tool_specs(tools),          # tool parameter definitions + descriptions + examples
        "L3_agent": render_history(history),           # conversation history + tool returns + intermediate results
        "L4_user": user_input,
    }
    ledger = {name: counter(text) for name, text in layers.items()}
    total = sum(ledger.values())
    ledger_ratio = {k: round(v / total, 4) for k, v in ledger.items()}
    ledger_ratio["L4_available"] = round(1 - sum(
        ledger_ratio[k] for k in ("L1_system", "L2_tools", "L3_agent")
    ), 4)
    # Budget guardrail: when the user layer drops below the threshold, trigger compaction or a prompt,
    # rather than letting the request silently top out
    if ledger_ratio["L4_available"] < 0.20:
        compact_agent_context(history, target_ratio=0.45)
    return layers, ledger, ledger_ratio

Pair this with a budget declaration that writes “who can take up how much at most” as an explicit contract:

context_budget:
  window_tokens: 200000
  hard_limit_ratio: 0.85        # reserve a safety margin before hitting the ceiling
  quotas:
    L1_system: 0.03
    L2_tools: 0.10
    L3_agent: 0.55
    L4_user_min: 0.20           # minimum guarantee for user input
  on_quota_exceeded: [compact, drop_oldest, summarize_tool_output]

With a ledger and quotas, optimization turns from “the context feels a bit long” into an engineering problem with clear goals.

5. Six practical optimization techniques

5.1 Prompt layering and a static prefix

Split the prompt into three segments — “an extremely stable prefix + a changing middle + the user’s tail” — and move everything that changes frequently (timestamps, current file path, temporary rules) out of the prefix. Once the prefix is stable, the hit rate of the model-side prefix cache (KV cache reuse) rises significantly, which both reduces per-token cost and compresses time-to-first-token. In practice you can set a convention: the system instruction layer changes no more than once a month, and any rule additions go into a “task-level prompt” first rather than the global system prompt.

5.2 Tool governance: from “mount everything” to “load on demand”

The tool layer is the only layer that can be substantially compressed through architectural means:

• Grouped mounting: divide tool sets by scenario (code search, build, test, deploy, doc lookup), and load only the group relevant to the current task in a single session.
• Semantic tool discovery: maintain a tool index, first have the model retrieve candidate tools with a lightweight “find tool” action, then load the full definitions, reducing the O(N) resident cost to an O(1) retrieval cost.
• Description slimming: remove duplicate examples and marketing-style descriptions; a tool’s description is enough with three parts — “when to use + parameter constraints + failure signals.”
• Consolidation and retirement: periodically measure the actual call rate of tools, and retire those with zero calls for a long time. Fewer tools mean fewer misselections by the model, and less invalid bloat in L3.

5.3 Context compression and external memory

The core strategy for L3 is “don’t keep raw data resident long-term; only keep a retrievable reference”:

• Rolling summary: every N turns, or when tokens reach a threshold, compress early conversation into a structured summary (goal, completed actions, key conclusions, open questions).
• Offloading tool output to disk: write large results to external storage (such as object storage or a file system), and keep only the path, a summary, and key fields in the context, reading back on demand.
• Externalizing structured memory: write facts that must be retained across sessions (architecture conventions, user preferences, task progress) into an independent memory store, and retrieve a few relevant entries at session start by relevance, instead of injecting everything.
• Reclaiming intermediate results: explicitly mark which tool returns are “one-time information,” and remove them from history once the decision in the same turn is complete.

5.4 Output-side constraints: reduce backfill volume at the source

Governance cannot be done only on the input side. Apply uniform output constraints to tool calls: limit command output to a maximum number of lines and bytes while keeping the first and last few lines; make list queries paginated by default and return the total count rather than the full set; when a build or test fails, return only the error fragment and a stack summary; filter log-type data through rules (strip timestamps, deduplicate, drop noisy fields) before it enters the context. These constraints are usually the highest return-on-investment part.

5.5 Use retrieval instead of full injection

“Stuff the whole directory into the context and let the model find it itself” is a common anti-pattern. A more robust approach is to first locate candidate snippets with keyword or vector retrieval, inject only the Top-K with source annotations, and make it explicit in the prompt that “when information is insufficient, you should initiate a retrieval rather than guess.” Combined with reranking and deduplication, you can compress the injected volume by an order of magnitude without losing key clues.

5.6 Sub-context isolation

Hand parallelizable exploratory work (large-scale code search, multi-option validation, documentation combing) to subtasks, complete it in an independent context, and return only a conclusion summary to the main context. The main chain thus stays lean, and the bloat of the sub-chain does not affect the usable window of the main task. This is the most effective structural means of controlling L3 growth in long tasks.

6. Engineering it: turning context governance into an observable loop

For optimization techniques to stay effective long-term, they must land on platform capabilities:

• Metrics layer: turn layered token share, ceiling-hit rate, cache hit rate, compaction trigger count, and average turns per task into standard metrics, and drill down by business and scenario dimensions.
• Alerting and guardrails: set alerts for “user-usable space share below threshold,” “abnormally large per-turn tool output,” and “tool layer share steadily rising,” and automatically trigger compaction or degradation strategies when quotas are exceeded.
• Data pipeline: deliver session traces and tool-call records to a message queue and log service, and store them asynchronously in a database for offline analysis; bring prompt templates and tool definitions under version management and canary release, to avoid a single change affecting all traffic.
• Elastic capacity: deploy the inference and tool-execution parts of the agent service on a container platform with on-demand scaling, and combine monitoring, alerting, and tracing to locate slow calls and long-context requests.
• Compliance and isolation: in multi-tenant scenarios, isolate session storage and memory stores by tenant, desensitize sensitive fields in logs, and keep retention periods explicit and controllable.

On platforms like Baidu AI Cloud, these steps can typically be carried by combinations of generic cloud product categories such as object storage, message queues, container platforms, log services, and monitoring and alerting, without building the full infrastructure yourself; the focus is on implementing the token ledger and quota strategy as a first-class citizen on the application side, rather than relying on the implicit behavior of any one platform layer.

7. Three common misconceptions

1. Thinking a big window means no governance is needed. Doubling the window only pushes back the ceiling-hit time; long tasks will still hit the wall, and you pay for the big window every turn, with cost and latency worsening together.
2. Only compressing without attribution. Compressing without knowing which layer the tokens are spent in often ends up compressing L1’s safety constraints or L4’s user requirements, in exchange for unpredictable output drift.
3. Ignoring cache-friendliness. Frequently changing the system prompt prefix or injecting timestamps at the head will make the cache almost never hit, and the actual cost ends up far higher than the difference in per-token unit price.

8. Summary

The essence of the context window is a scarce piece of memory shared by multiple parties. The stable fact is: system-level instructions usually take up only a single-digit percentage, the real bloat concentrates in conversation history and the intermediate-state data produced by tools, and the user’s usable input space is a passive remainder. The actionable governance path is also clear: first make the budget explicit with a layered ledger and set quotas, then compress fixed costs through on-demand tool mounting, suppress cumulative costs through summarization and external storage, reduce injected volume through output constraints and retrieval, and control main-chain growth through sub-context isolation — and finally solidify this mechanism into a long-term effective engineering capability with metrics, alerting, and versioning. Once all this is done, improvements in model performance often do not come from a “bigger window,” but from a “cleaner, more targeted window.”