Million-Token Context and High-Throughput Agent Inference in Practice
A practical guide to running long-horizon AI agents on models with million-token context windows. It explains why agents consume context far more aggressively than single-turn chat, how the prefill and decode phases differ in their bottlenecks, and which metrics actually matter for throughput (TTFT, OTPS/TPOT, P99 latency, cache hit rate, and per-task cost). It then covers explicit prompt caching, context budgeting, and cross-region inference, and how each one shifts cost, latency, and compliance. Finally, it addresses security and governance for hosted open-weight models and provides an evaluation and tuning checklist for keeping long-context behavior stable and observable.
1. Why Million-Token Context Becomes Agent Infrastructure
Agents and traditional single-turn Q&A differ fundamentally in how they consume context. Traditional dialogue grows mainly through the linear accumulation of message history, whereas an agent writes everything back into context on every turn: tool definitions, tool-call parameters, tool return values, failure and retry records, and reasoning traces. A single repository-level task — where even the “retrieve–read–rewrite–verify” loop alone can trigger dozens of tool calls — will often see a single returned log, diff, or compilation error run to thousands of tokens. Context swelling to the hundred-thousand-token range within a few steps is the norm, not an exception.
This leads to three direct consequences:
- Truncation is distortion. Once the context limit is hit, the system can only truncate bluntly, and what gets cut is precisely the earliest-discovered constraints and the original goal. The model then falls into self-contradictory loops.
- Attention dilution. Key instructions buried in the long tail of hundreds of thousands of tokens see a sharp drop in adherence, showing up as behavioral regressions like “forgot I’m not allowed to modify test files.”
- Cost spirals out of control. Every replay of a long context re-bills the input tokens, and multi-turn agent tasks often reach an input-to-output ratio of tens to one.
The significance of million-level context is not just “fitting more in,” but shifting the engineering focus from “how to compress” to “how to organize.” When capacity is no longer a hard constraint, lossy techniques such as retrieval-based recall, crude summarization, and forced truncation can retreat to being optimization measures rather than fallback mechanisms, allowing the model to reason over a complete evidence chain. This capability is now available in both open-weight models and the model hosting services of mainstream cloud platforms; Baidu AI Cloud’s model service platform likewise supports long context, open-weight model hosting, and caching capabilities — for specific specifications, refer to the official documentation.
2. The Engineering Meaning of High Throughput: What OTPS Actually Measures
The OTPS metric (output tokens per second) repeatedly emphasized in the source material is easily misread as a single indicator of “how fast the model is.” Strictly speaking, a single inference runs in two phases whose performance bottlenecks are entirely different:
| Phase | What it does | Compute characteristics | Main bottleneck | User perception |
|---|---|---|---|---|
| Prefill | Processes all input tokens at once and builds the KV Cache | Matrix-multiply heavy, highly parallelizable | Compute (compute-bound) | Time to first token (TTFT) |
| Decode | Generates output autoregressively, token by token | Each step only reads weights and KV Cache; low arithmetic intensity | Memory bandwidth and capacity (memory-bound) | Output smoothness (TPOT / OTPS) |
This table explains two counterintuitive phenomena. First, long context degrades both phases simultaneously: prefill’s compute cost grows roughly linearly with input length, and the KV Cache’s memory footprint also grows linearly, the latter squeezing the concurrency batch size you can fit and thereby pushing down OTPS. In other words, “longer context” and “faster output” are inherently in conflict. Second, OTPS decays under high concurrency: a rising token count per batch improves compute utilization, but every step must read the entire KV Cache, so bandwidth pressure rises in step, and the ceiling ends up being set by bandwidth rather than compute.
Therefore, a reasonable set of metrics for evaluating a high-throughput agent model is a group rather than a single number:
- TTFT: determines the length of each agent “thinking pause” and directly shapes the interactive feel.
- OTPS / TPOT: determines the wait for long answers and long code generation. Human reading speed is on the order of tens of characters per second; when output throughput falls below this, users clearly perceive stutter.
- P99 latency: an agent’s serial chain amplifies tail latency, so a single step’s P99 regression compounds into minute-level waits over a ten-step task.
- Cache hit rate: the single variable with the largest impact on cost and latency in long-context scenarios.
- Per-task cost: rather than per-token price, which fails to reflect the agent’s amplification effect.
import timedef measure_streaming(client, model, messages, cache_point=None):"""Illustrative implementation for measuring TTFT and decode-phase throughput"""t0 = time.perf_counter()ttft, out_tokens = None, 0stream = client.stream_chat(model=model,messages=messages,cache=cache_point, # explicit prompt cache breakpointstream=True,)for chunk in stream:if ttft is None and chunk.text:ttft = time.perf_counter() - t0out_tokens += chunk.usage.delta_tokenstotal = time.perf_counter() - t0decode_window = max(total - ttft, 1e-6)return {"ttft_s": round(ttft, 3),"otps": round(out_tokens / decode_window, 1),"out_tokens": out_tokens,"total_s": round(total, 3),}
Engineering-side throughput optimizations, roughly ordered by cost-effectiveness, are: request coalescing and continuous batching; chunked prefill (to avoid long requests blocking short ones); prefix cache reuse; speculative decoding (parallelly verifying multiple tokens with a small draft model); weight and KV Cache quantization; expert-parallelism / tensor-parallelism scaling; and turning parallelizable tool calls into concurrent launches rather than serial waits. The first two are typically handled on the platform side; the rest are where the application layer can genuinely make a difference.
3. Explicit Prompt Caching: The Watershed of Long-Context Cost
The core idea of prefix caching is not complicated: during Transformer inference, the Key/Value tensors of the earlier part of the input depend only on themselves and on what precedes them, so a prefix that is byte-for-byte identical across requests can reuse its KV Cache across requests. Explicit prompt caching turns this from “the platform’s best effort” into an “application-declarable, observable” capability.
Hit rate depends on the prefix’s byte-level stability, which imposes several hard design constraints:
- The prefix must be static and placed on top. Role settings, tool schemas, output-format constraints, few-shot examples, and domain glossaries should all move forward. Any single character’s change invalidates everything after the breakpoint.
- Volatile content goes last. Timestamps, request IDs, user identifiers, and dynamic fields other than the sampling temperature must never appear before the cache breakpoint. A common bug is inserting
Current time: 2026-08-26 11:00:32into the system prompt, which pins the hit rate at zero. - Template rendering must be deterministic. Inconsistencies in dict ordering, float formatting, or whitespace collapsing all cause “logically identical, byte-different” output. It is advisable to add prefix-hash checks to CI and fix the prefix fingerprint as a build artifact.
- The cache has a lifecycle. Cache entries have a time-to-live window, and low-traffic periods produce “cold starts” where the first request’s TTFT is noticeably higher. Capacity planning should estimate resident cost by peak, not average.
- Multi-tier combinations. Stable prefix + semi-stable middle layer (e.g., a same-day knowledge-base snapshot) + dynamic tail, labeled with multiple breakpoints, typically yields a significantly higher hit rate than a single breakpoint.
An illustrative layered construction:
SYSTEM_PREFIX = """You are a repository-level coding assistant.Tools: [read_file, edit_file, run_tests, search_repo]Constraints: do not modify test files; run tests after every change; output a diff summary.""".strip()def build_messages(task, index, history, token_budget):static = [{"role": "system", "content": SYSTEM_PREFIX}] # breakpoint 1: highly stablemid = [{"role": "system", "content": index.snapshot()}] # breakpoint 2: invalidated by version / time windowdynamic_budget = token_budget - count(static) - count(mid) - count(task.query)recent = trim_by_token(history, dynamic_budget * 0.6) # keep the most recent few roundsreturn static + mid + recent + [{"role": "user", "content": task.query}]
An often-overlooked benefit is latency rather than cost: a cache-hit request skips most of the prefill phase’s compute, and the curve of TTFT growing with input length is significantly flattened. For agents, this means “longer context” and “faster response” are no longer necessarily at odds.
4. Context Budget Management: Sufficient Capacity Does Not Mean Free Reign to Fill
Even when the window reaches a million tokens, explicit budget management is still worthwhile. The reason is that effective attention is not uniformly distributed across the window; information nearer the end is generally utilized better, and placing truly critical constraints in the middle reduces adherence.
It is recommended to maintain a budget table per request:
| Zone | Suggested share | Content | Invalidation strategy |
|---|---|---|---|
| Static prefix | 5%–15% | Role, tool definitions, format constraints | Long-term resident, kept in cache |
| Semi-static knowledge | 10%–25% | Glossary, code-structure index, domain snapshot | Refreshed by version / time window |
| Retrieved evidence | 20%–40% | On-demand documents, code snippets | Recomputed each round, keeping only highly relevant fragments |
| Conversation trace | 10%–30% | Tool calls and results | Layered summarization, compressing early rounds |
| Current task | <10% | The user’s instruction this turn | Changes every turn |
| Generation reserve | ≥20% | Reserved for output | Hard reservation, cannot be reallocated |
Tool return values are the biggest consumers of budget and need hard constraints at the integration layer rather than waiting for model-side truncation:
- For large files and long logs, use a three-part “paginate + summarize + locate” return: by default provide only the structure and the first few lines, letting the model pull more on demand.
- Log outputs should first undergo pattern clustering, folding repeated lines into “pattern + count + sample.”
- Test failure information should by default return only the first failure and its stack, avoiding full-blown noise.
- Retrieval should truncate after reranking, rather than filling up by raw vector similarity.
5. Cross-Region Inference: The Triangle of Capacity, Latency, and Compliance
The “cross-region inference” mentioned in the source material means, at its technical core, pooling a single model’s inference capacity across multiple regions so that requests can be scheduled by real-time available capacity rather than bound to a single region’s quota. Its value is clear: avoiding rate-limiting or queuing caused by insufficient peak capacity in one region, and improving success rate and elasticity during long-tail periods.
The cost is equally clear and must be actively weighed in the architecture:
- Increased latency. Cross-geography network round trips add tens to hundreds of milliseconds on top of TTFT. The impact on interactive dialogue is limited, but high-throughput services handling high-frequency short requests need to evaluate it.
- Data residency and compliance. This is the most important thing to confirm in advance. If the business is subject to constraints on data leaving the country or a specific region, region-anchoring policies must be enabled, trading some capacity elasticity for compliance certainty. It is advisable to do policy routing at the gateway layer, using compliance labels as the primary routing dimension and capacity as the secondary one.
- Increased observability complexity. Instrumentation is needed across multiple dimensions — “region + model + cache hit rate + rate-limit state” — otherwise fault localization degrades. It is recommended to integrate a unified logging service and monitoring/alerting, with tiered thresholds for TTFT, OTPS, rate-limit rate, and cache hit rate.
def route(request, policy, capacity):# Tier 1: compliance anchoring, cannot be overridden by capacity factorsallowed = [r for r in capacity if policy.data_residency(r.region)]# Tier 2: within compliant regions, weight by available capacity and measured latencyallowed.sort(key=lambda r: (r.queue_depth, # capacity firstpolicy.latency_weight * r.p99_ttft, # latency second-r.cache_hit_rate, # prefer cache hits within the same region))return allowed[0]
6. Security and Governance: Open-Weight Models Should Not Lower the Bar
When open-weight models are hosted on a cloud platform, they should run within the same security boundary as proprietary models. Engineering recommendations should cover the following layers, which on platforms like Baidu AI Cloud can be implemented through general categories: private-network access and egress control, encryption in transit and at rest, key management services, fine-grained authorization for identity and permissions, audit log retention, and content safety filtering.
Several easily overlooked governance points:
- Least privilege down to the tool level. An agent’s file writes, command execution, and external calls should be authorized per tool, defaulting to deny, with mandatory human confirmation for high-risk operations.
- Data masking up front. Perform PII and secret-pattern detection and masking on prompts and tool return content before they leave the local network, then layer additional masking on the logging side to avoid sensitive information spreading a second time in the logging service.
- Treat untrusted content as data, not instructions. If instruction-style text appears in retrieved web pages, issues, or comments, it should be explicitly labeled as data-to-be-analyzed in the prompt, reducing prompt-injection risk.
- Retain a traceable chain. Record each request’s model version, prompt prefix hash, cache-hit status, and tool-call sequence, making it easier to reproduce and cost accounts after the fact.
- Cost and quota guardrails. Set per-task token limits, maximum step counts, and timeouts to prevent an agent from looping and consuming resources indefinitely.
7. Evaluation and Tuning Checklist
Long-context capability must be measured, not inferred from a nominal window size. It is recommended to build three categories of evaluation:
- Information localization: plant several unrelated “needles” in a long document, test multi-point retrieval and cross-paragraph association, and observe how much hit accuracy degrades when the match sits in the middle.
- Long-range consistency: inject constraints early in a multi-round task (do not modify a certain directory, keep an interface signature), then verify tens of rounds later whether they are still honored.
- Throughput and cost: run load tests using tool-return lengths drawn from a real distribution, recording TTFT, OTPS, P99, and per-task cost for both cache-hit and cache-miss scenarios.
The recommended tuning priority follows a “structure first, parameters second” order: first fix prefix stability to secure cache hits, then do retrieval and trimming to reduce input volume, then adjust concurrency and routing, and only then consider sampling parameters and model specifications. In practice, the payoff from prefix design usually far exceeds any single parameter adjustment.
8. Conclusion
Million-level context, high output throughput, explicit caching, and cross-region scheduling together form the production prerequisites for long-horizon agent applications. But the availability of technical capability is only the starting point: what truly determines deployment quality is how context is organized and budgeted, whether the cache prefix is strictly stable, whether compliance boundaries are hard-coded into the routing policy, and whether tool returns are strongly constrained. Once this engineering discipline is in place, “it runs” becomes “stable and operable.” To deploy on Baidu AI Cloud, you can start with the model service platform’s managed inference, pair it with object storage for cold-data and trace archiving, use a logging service for full-chain tracing, and rely on monitoring and alerting to guard the latency and cost red lines.
