Understanding LLM Context Budget: Give the Context Window Back to Reasoning

In engineering practice for intelligent coding tools and agent frameworks, a common misconception is that a larger context window means the model can handle more complex tasks. The reality is often the opposite: the window is an upper limit, not a usable budget. Once system prompts, tool declarations, skill descriptions, conversation history, and tool outputs are all placed in the same window, the space left for the user’s actual question and for the model’s reasoning is often much smaller than expected.

1. The starting problem: window size is not usable reasoning space

All tokens a model sees in one invocation form its working memory. This memory is not partitioned by purpose; it shares one budget. Public technical analyses of certain coding assistants have estimated the following approximate context breakdown:

Context component	Typical share (illustrative)	Notes
System prompt: role, rules, style constraints	1%–2%	Relatively fixed and carried every turn
System tools, including externally integrated tools exposed through tool protocols	5%–10%	Declarations grow linearly as integrations expand
Agent context: skills, tool descriptions, conversation history, tool results	60%–75%	Accumulates dynamically as the task progresses
User input and actual reasoning headroom	Remaining portion	The part that is often compressed

These numbers should be treated with caution. They come from a specific implementation at a specific snapshot. Different products and different task stages can vary significantly, so they should not be cited as a performance benchmark. However, the conclusion they reveal is general: the absolute token count of system-level instructions is usually controllable and stable, but their share is continuously squeezed by the other three parts.

One detail is often overlooked: a low percentage does not mean a low cost. System instructions are resent every turn, so after N turns their cumulative overhead is N times the single-turn cost. The same applies to tool declarations. Therefore, evaluating context health should focus on absolute token usage and growth curves, not on the share in a single snapshot.

2. Three structural causes of context expansion

2.1 Linear stacking of tools and capabilities

Every additional tool or skill adds a JSON Schema description, usage examples, and constraints. A single tool declaration can easily consume hundreds of tokens. After integrating dozens of tools, the declarations alone can take up a considerable part of the budget. More seriously, similar functionality described across tools can interfere with each other and increase the probability that the model selects the wrong tool. This is both a space problem and an accuracy problem.

2.2 Monotonically accumulating history and tool outputs

Many agent loops are implemented by appending every intermediate thought, command, and result to the message list. Reading a log file, returning a directory tree, or executing a build can produce thousands of tokens, and that output does not disappear automatically. In long-running tasks, history grows roughly linearly and eventually triggers truncation or compression, often at the most critical point.

2.3 State is not externalized

When the intermediate state of a task—completed steps, pending items, key conclusions—exists only in the conversation history, the history becomes the single source of truth. Once compression happens, state can be lost. Once replay happens, cost doubles. This is a common root cause of long-running agents “suddenly forgetting what they were doing” in the later stages of a task.

3. Quantitative modeling: measure before optimizing

Context optimization requires measurability. A lightweight budget reporter should be added to the agent framework to report token usage by category before each invocation:

from dataclasses import dataclass
@dataclass
class ContextBudget:
    system_prompt: int
    tool_declarations: int
    skills_and_descriptions: int
    history: int
    tool_outputs: int
    user_input: int
    @property
    def total(self) -> int:
        return sum(self.__dict__.values())
    def report(self, window_limit: int) -> str:
        rows = [(k, v) for k, v in self.__dict__.items()]
        lines = [f"{k:<22}{v:>8} tokens  {v / self.total:>6.1%}" for k, v in rows]
        lines.append(f"{'Total':<20}{self.total:>8} tokens  "
                     f"window utilization {self.total / window_limit:.1%}")
        return "\n".join(lines)

Because different models use different tokenizers, engineers can approximate usage with character counts—roughly 1 token per Chinese character and 4 characters per token in English—or calibrate with a tokenizer interface at the integration layer. The key is to use a stable measurement method for each category so that usage can be compared across tasks and versions.

Three core metrics are recommended:

• Share of usable reasoning space: user input plus reserved reasoning headroom divided by the window limit. The goal is to leave enough room for output and reasoning, avoiding a state where the window is full immediately after reading the context.
• Fixed cost per turn: system instructions plus tool declarations. This determines the baseline cost of long-running tasks.
• History growth rate: average and peak token growth per turn, used to decide whether compression should be triggered inside the loop.

4. Five governance strategies

Layer 1: Tool and capability pruning

The core principle of tool governance is to expose tools by task rather than keep all tools resident at all times. Practical approaches include:

• Semantic grouping and lazy loading: group tools by domain, such as files, build, test, deployment, and retrieval. Inject the full declaration only when the task involves that domain; otherwise, use a one-line index placeholder.
• Concise descriptions: remove lengthy examples and keep only what the tool does, when to use it, parameter constraints, and failure signals. Move detailed documentation to external documents that can be loaded on demand.
• Deduplication and merging: merge overlapping tools into one tool with a mode parameter. This reduces both declaration count and selection ambiguity.

Layer 2: Structure and compress tool output

Tool output is the fastest-growing source of context expansion. The principle is: return summaries and location information, not raw text.

def summarize_output(raw: str, max_chars: int = 2000,
                     keep_head: int = 500, keep_tail: int = 500) -> str:
    """Keep the head and tail for oversized outputs, fold the middle, and return a traceable external path."""
    if len(raw) <= max_chars:
        return raw
    omitted = len(raw) - keep_head - keep_tail
    return (f"{raw[:keep_head]}\n"
            f"... [omitted {omitted} characters; full content available at artifacts://last_output] ...\n"
            f"{raw[-keep_tail:]}")

For logs, build errors, and code search results, a better approach is a structured summary plus an original file handle. Write the full result to external storage and return the path. When the model needs details, it can read specific line ranges. This controls the window while preserving traceability.

Layer 3: History management and prefix stability

Conversation history needs active governance, not passive accumulation:

• Segmentation and rolling summaries: divide history into a stable prefix—system instructions plus tool declarations—and an unstable tail—recent interactions. Keeping the prefix stable helps prefix caching hit, reducing cost and latency. When the tail exceeds a threshold, compress earlier turns into a structured summary: goal, completed actions, key conclusions, and open questions.
• Structured state blocks: maintain a short task status card at the beginning of each turn, using a fixed format to record progress and constraints. This is cheaper in tokens than long history and more resistant to compression.
• Explicit discard policy: define which messages can be safely removed, such as intermediate tool calls already covered by summaries, and which must be retained, such as the user’s original requirement and key decision evidence.

Layer 4: On-demand retrieval instead of preloading

Skill descriptions, domain documents, and specification clauses should not be fully injected at once. A more reasonable approach is to maintain a lightweight index: the context contains only titles and one-line summaries, and the model loads the full text only when it explicitly requests it. This decouples “knowing what exists” from “knowing the details.” The former is very cheap, while the latter is paid for by usage.

Layer 5: Memory externalization and persistence

Separating working memory from long-term memory is key to stability in long-running tasks:

• Working memory: the current window, containing only content strongly related to the current step.
• External memory: files, object storage, vector retrieval libraries, or caches that hold full history, artifacts, and knowledge entries. They are referenced in the context by path or ID.
• Archive layer: after task completion, write logs to logging services and monitoring systems for offline analysis, replay, and cost accounting. This layer does not occupy the reasoning window.

5. Four common pitfalls

1. Assuming a larger window makes everything easier: a larger window hides expansion problems but increases per-turn cost, latency, and attention dilution. The problems tend to explode in long-running tasks.
2. Compressing without structuring first: blunt truncation can lose critical constraints. The correct order is: structure first, compress second, and consider truncation last.
3. Ignoring the hidden cost of tool descriptions: tool declarations are a fixed cost every turn. Improving their governance produces multiplicative benefits.
4. Lacking regression metrics: context budget changes should be monitored by automated metrics, not by a manual feeling that “it seems slower.”

6. Practical implementation path

A pragmatic order is: measure → prune → compress → externalize.

The first step is to integrate the budget reporter and establish a baseline for typical tasks. The second step is to prune and merge tools and descriptions. The third step is to implement output summarization and rolling history compression. The fourth step is to build external memory and artifact storage. At each step, use the share of usable reasoning space and task completion rate as acceptance metrics, avoiding quality loss merely to save tokens.

At the implementation level, common cloud capabilities such as model inference services, object storage, message queues, logging services, and monitoring alerts can be combined into a complete context governance pipeline. The inference service handles model invocation and prompt assembly. Storage carries artifacts and external memory. Queues and caches support retrieval and state transfer. Logs and monitoring provide continuous observation of cost, latency, and truncation events. On Baidu AI Cloud, model services and these general capabilities can be combined in this way to form a closed loop from measurement to governance to observation.

The essence of context engineering is to allocate a limited attention budget to the most relevant information. The window size is determined by the model, but budget allocation is determined by the system designer—and that is the truly controllable variable in agent engineering.