The Long Context Era: Controlling Token Costs Through Context Engineering
As context windows expand from 4K to millions of tokens, capacity and affordability have become two separate concerns. This article explains why long-context tasks can drain budgets through repeated token billing, why a higher cache hit rate does not always mean lower cost, and how context engineering techniques such as sliding windows, summarization, retrieval-augmented generation, external memory, and tool result pruning keep spending under control. It includes practical Agent-level strategies for setting conservative context budgets, automatic compression triggers, and a checklist of best practices for making the most of a limited token budget.
1. When “Million-Token Context” Became the Default Selling Point
Over the past two years, the arms race over context windows has never stopped. We have moved from 4K to 32K to 128K, and now to million-token scale. The implicit promise is that if you stuff enough history into the model, an Agent will “remember everything” and complete longer chains of tasks. Vendors have followed suit, shifting their pitch from “smarter” to “can hold more.”
But in engineering practice, window capacity and usable cost are two entirely different dimensions. A model that can hold a million tokens does not mean you can afford to run inference on a million tokens continuously. The window is an upper bound on capability; cost is a real-world constraint. Conflating the two is the root cause of many long-horizon tasks blowing through their budget.
2. Why Long Context Is “Unaffordable”
To understand the cost problem, start with the basic logic of token billing. The vast majority of large model APIs bill per token, typically distinguishing input tokens from output tokens, and offering discount mechanisms such as “prompt caching” for long prefixes that recur across calls. This works well for single-turn, short interactions, but it changes character in long-horizon tasks.
Long-horizon tasks (for example, multi-turn Agents, automated coding, and continuous chat assistants) share a key trait: every round of inference must feed the accumulated context back into the model. Suppose a task has already accumulated N tokens of history. Then every subsequent call must pay for those N tokens. The longer the task runs, the larger N grows, the higher the per-round cost, and the cumulative cost grows roughly super-linearly. A poorly designed long-horizon Agent can easily burn through its entire quota within a few dozen rounds.
Round 1: input 2,000 tokensRound 10: input 25,000 tokensRound 50: input 120,000 tokens <- paying for the full history every roundRound 100: input 240,000 tokens <- the quota is long gone
If the billing model offers only a limited discount for cache hits, then the intuitive instinct of “just stuff as much as possible into the context” becomes a cost disaster.
3. The Cognitive Trap of Cache Hit Rate
Here is a counterintuitive conclusion: under certain billing models, a higher cache hit rate does not necessarily save money.
We usually assume a high cache hit rate is a good thing: the cached portion is billed at a discount, so it seems like you save a fair amount. But the key is the size of that discount. If the discounted price is not much lower than the original price, then the cost of retaining large amounts of redundant context to maintain a high hit rate may far exceed the savings from that discount.
Put differently, rather than going to great lengths to nudge the cache hit rate up a little, it is often better to simply drop what should be dropped. Dropping something costs nothing, while keeping a low-discount cache is still a net expense. A rational strategy is to stop fixating on the cache hit rate as a metric and instead watch “the total token spend required to complete the same task.” In practice, deliberately pushing the cache hit rate down from near-saturation to a moderate level, combined with aggressive context pruning, can often significantly slow the rate at which the quota is consumed.
This leads to a general principle: the optimization target should be total task cost, not any single intermediate metric. Cache hit rate and context length are only means to an end. When they conflict with the cost goal, they should yield to it.
4. Context Engineering: Compression, Retrieval, and External Memory
Since we cannot blindly fill the window, we need a “context engineering” methodology whose core is to maximize the density of useful information within a limited token budget. Common techniques fall into a few categories:
1. Sliding window / truncation
The most straightforward approach: keep only the most recent K turns of conversation or the most recent N tokens, and discard the rest. It is cheap to implement but loses early critical information, making it suitable for scenarios with weak dependence on history.
2. Summarization
When history exceeds a threshold, call the model to compress earlier content into a summary, then continue with “summary + recent verbatim content.” Recursive summarization (summarizing a summary) can handle extremely long tasks. The tradeoff is that the summary itself loses information, and the summarization process consumes additional tokens.
3. Retrieval-augmented generation (RAG) instead of full injection
Rather than stuffing all material into the context, place the material in an external knowledge base and retrieve the most relevant fragments on demand for injection. This shifts “memory” from the context window to an infinitely scalable vector or document store, and is the preferred approach for handling large amounts of background knowledge.
4. External memory and offloaded state
Write task state, completed steps, and intermediate artifacts to external storage (databases, files, object storage, and so on), leaving only a pointer or index in the context. Retrieve the content by ID when needed. This is essentially building a “tiered memory” for the Agent.
5. Tool result pruning
When an Agent calls tools, they often return large volumes of raw data (logs, retrieval results, file contents). Filtering, truncating, or structurally distilling this before injecting it into the context, keeping only the fields needed for decisions, can save a great deal of tokens.
5. Implementation Strategies at the Agent Level
Methodology ultimately has to land in the Agent’s runtime logic. The core idea is: proactively set a conservative context budget and configure automatic compression triggers, rather than passively waiting for the window to fill up.
The following is illustrative pseudocode for a context manager, showing the basic skeleton of “budget threshold + compression trigger”:
class ContextManager:def __init__(self, soft_limit=32_000, hard_limit=64_000):self.soft_limit = soft_limit # soft threshold that triggers compressionself.hard_limit = hard_limit # absolute ceiling, never exceedself.history = [] # message historydef add_message(self, msg):self.history.append(msg)self._maybe_compress()def _token_count(self):# in practice, use the corresponding model's tokenizer for exact countingreturn sum(len(m["content"]) for m in self.history) // 3def _maybe_compress(self):count = self._token_count()if count <= self.soft_limit:return# keep the most recent turns, fold earlier content into a summarykeep = self.history[-6:]to_summarize = self.history[:-6]summary = self.summarize(to_summarize)self.history = [{"role": "system", "content": f"[History Summary] {summary}"}] + keepdef summarize(self, messages):# call the model to generate a summary; this itself has a cost,# so trigger it only when necessary...
A few key engineering points:
- Separate soft and hard thresholds: the soft threshold triggers compression, while the hard threshold is a red line that must never be crossed. The soft threshold should leave enough headroom to reserve token space for the compression operation itself.
- Layer the compression: keep recent, high-value content verbatim as much as possible, and compress older history more aggressively, forming a structure similar to “memory decay.”
- Count with an exact tokenizer: do not estimate roughly by character count; token density varies widely across languages, so plug in the corresponding model’s tokenizer for accurate measurement.
- Wire cost metrics into monitoring: instrument every round’s input/output tokens, cumulative consumption, and cache-hit status, and observe trends through logging and monitoring/alerting systems so you can catch signs of budget burn early.
6. A Checklist of Best Practices
Overall, controlling cost in long-context scenarios can follow these guidelines:
- Treat total task cost as the single north star metric, and do not get led astray by intermediate metrics such as cache hit rate.
- Drop whatever you can: resolutely remove redundant, stale, and low-value content from the context; zero cost beats a low discount.
- Prefer retrieval over injection: route large-scale background knowledge through RAG, and keep only the currently relevant fragments in the context.
- Offload state: write task state to external storage and keep only an index in the context.
- Compress proactively instead of passively filling up: set a conservative budget and configure automatic triggers, keeping control in your own hands.
- Make the whole pipeline observable: continuously instrument and alert on token consumption so that cost overruns are detected before they happen.
It is worth emphasizing that these cost and context management problems are not model capability deficiencies, but an inevitable challenge in operationalizing long context at scale. The model provides a large window, but how to use it well and spend the limited token budget on genuinely valuable inference is the responsibility of the application-layer designer. Model services from platforms such as Baidu AI Cloud generally follow the same common per-token billing logic, so the context engineering methodology discussed here applies equally to the various long-horizon task scenarios built on them.
In the era of long context, the competition is no longer about “who can hold more,” but about “who can hold it more intelligently within a limited budget.” Only by treating context engineering as a first-class concern can you enjoy the capabilities of a long window while holding the line on the cost that keeps everything alive.
