Million-Token Context: High-Throughput LLM Inference in Practice
As context windows expand to a million tokens, developers can move from chunking and summarization to agent applications that work directly with large codebases, long documents, and multimodal material. This article explains what a million-token context window changes, which engineering challenges remain, and which techniques matter most: output throughput (OTPS) for smooth streaming, explicit prompt caching for lower latency and input cost, incremental context updates in agent loops, observability, security and governance, and cross-region deployment. It includes Python examples for prompt caching and a long-context document Q&A agent.
Introduction: How Long Context Opens Up a New Paradigm for Agent Applications
For a long time, LLM applications have been constrained by a hard limit: the context window. Long documents had to be split into chunks, codebases had to be trimmed down, and multi-turn conversations had to be compressed through summarization.
As the context window expands from tens of thousands of tokens to a million, a fundamental shift occurs. The model can now fit an entire technical spec, a mid-sized codebase, a full quarter of earnings reports, or a complete piece of multimodal material into a single inference. This has given rise to a new class of applications: agent-driven applications.
Agent applications go beyond passive question answering. The model can break a goal into subtasks, call tools, read long context, iterate on its reasoning, and produce results. This article focuses on three threads that underpin such applications: million-token context, high-throughput inference, and cloud-based security and governance.
1. What a Million-Token Context Window Really Changes
1.1 A New Set of Capability Boundaries
The context window determines how much information the model can take in during a single inference. Roughly speaking, a million-token window can hold millions of characters of text.
The point is not to show off. The point is to move the following tasks from “engineering workarounds” to natural expression:
- Coding: feed in multiple source files from a project at once, so the model understands cross-file call relationships instead of looking at individual functions in isolation;
- Document processing: long contracts, annual reports, and research papers no longer need to be mechanically chunked, so context stays coherent;
- Financial research: analyze large volumes of earnings reports, announcements, and market data together;
- 3D and multimodal creation: native multimodal capabilities allow text, images, and structured data to be reasoned about together in the same context.
1.2 Engineering Challenges You Can’t Ignore
A long context window is not a free lunch. As sequence length grows, the cost of attention computation rises nonlinearly, memory and compute usage increase significantly, and time to first token gets longer.
So making the most of a million-token context depends less on how large the window is and more on whether inference is efficient enough and cost is under control. This is exactly the problem that high-throughput inference and prompt caching are meant to solve.
2. High-Throughput Inference: Why OTPS Is Central to the Experience
2.1 Two Metrics: Throughput and Latency
Inference services are typically evaluated along two dimensions:
- Time to first token (TTFT): the time from sending a request to receiving the first token. It determines how long users have to wait.
- Output throughput (OTPS, tokens output per second): how many tokens the model generates per second during the generation phase. It determines how smooth the output reads.
For agent applications, a single task often involves multiple rounds of generation and large amounts of structured output, such as code or JSON. Low OTPS makes the experience feel sluggish. So the value of high-throughput inference goes beyond speed: it is about keeping a stable, streaming response while long content is generated.
2.2 Common Techniques for Improving Throughput
The industry widely uses the following techniques to balance throughput and cost. All of them are general inference optimization approaches:
- Continuous batching: dynamically combine different requests to improve hardware utilization;
- KV cache reuse: cache intermediate attention states to avoid recomputing them;
- Quantization and operator optimization: reduce the compute cost of each pass while keeping accuracy within acceptable limits;
- Sensible streaming output: return tokens as they are generated to improve perceived latency.
When choosing a deployment platform, look at whether it can deliver stable, high-concurrency inference capacity. On cloud platforms such as Baidu AI Cloud, you can typically combine elastic compute with monitoring and alerting. This lets you continuously track and tune key metrics such as OTPS, error rate, and queueing time.
3. Prompt Caching: A Key Lever for Cutting Cost and Improving Efficiency
In agent scenarios, a clear pattern emerges: many requests share the same long prefix. The same system instructions and reference documents are sent to the model again and again. Computing them from scratch every time wastes compute and pushes up both latency and input costs.
3.1 How Prompt Caching Works
The core idea behind explicit prompt caching is to treat stable content, such as system prompts, long documents, and knowledge base excerpts, as a cacheable prefix.
The first request computes this prefix and builds a cache. Later requests that share the same prefix reuse it directly, and only the new content is processed. This lowers both latency and input-side costs.
3.2 Illustrative Code
# Illustrative: explicit prompt caching to cut the cost of recomputing repeated prefixes# Put the stable system prompt and the long document in the "cacheable prefix"stable_prefix = "You are a senior coding assistant. Answer strictly according to the specs below.\n" \+ load_document("spec.md") # reference material at the million-token scale# First request: builds the cached prefixrequest_1 = {"messages": [{"role": "system", "content": stable_prefix},{"role": "user", "content": "Summarize the core interface design of this document"}]}# Later requests: keep the prefix unchanged and swap in a new question to hit the cache# On a cache hit, only the new content is processed, which significantly lowers latency and input costsdef ask(question, cached_prefix=stable_prefix):return model.generate([{"role": "system", "content": cached_prefix},{"role": "user", "content": question}], stream=True)
The key design principle is to put stable content first and variable content last, so the cacheable prefix gets as many cache hits as possible. This approach pays off especially well in scenarios such as long-document Q&A and codebase understanding.
4. Agent Applications in Practice: Long Context + Tool Use
The real power of a million-token context is only unleashed inside an agent loop. A typical agent application follows a cycle of “understand the goal → call a tool → read the result → iterate on reasoning.” Because the context is large enough, the model can keep accumulating intermediate results without frequent summarization.
4.1 Append Incrementally Instead of Rewriting
A key cache-friendly practice in multi-turn tool use is to append to the context incrementally rather than rewriting it wholesale. This lets you reuse existing prefix caches while keeping the reasoning chain intact.
# Illustrative: a document Q&A agent loop built on long contextdef run_agent(document, goal, max_turns=5):# Inject the long context once, then append to it incrementallycontext = [{"role": "system", "content": "You are a document analysis assistant. You can call retrieval and calculation tools."},{"role": "user", "content": document}, # material at the million-token scale]for _ in range(max_turns):resp = model.generate(context)if resp.needs_tool: # external tool requiredresult = call_tool(resp.tool, resp.args)context.append({"role": "tool", "content": result}) # append so the prefix can be reusedelse:return resp.text # task completereturn "Maximum number of iterations reached"
4.2 Engineering Considerations
- Context budget: even with a million-token window, monitor token usage and avoid letting context grow without bound, or costs can spiral out of control;
- Retries and idempotency: tool calls need timeout and retry policies to prevent the agent from getting stuck in an infinite loop;
- Observability: use logging services and monitoring and alerting to record the inputs and outputs of every step, which makes runs easy to trace back and optimize.
5. Security and Governance: The Baseline for Enterprise Applications
Once a model is connected to real business data, security and governance stop being optional. Taking cloud-hosted model services as an example, enterprises should typically look at these general capability areas:
- Data isolation and encryption: encryption for data in transit and at rest, along with resource isolation between tenants;
- Permissions and access control: call authorization based on least privilege, with audit trails for sensitive operations;
- Content compliance: compliance filtering on inputs and outputs to prevent sensitive information from leaking;
- Consistency with existing security boundaries: model services should run under the same security and compliance framework as business systems, rather than sitting outside governance.
Mature cloud platforms generally turn these capabilities into general product categories, such as logging services, monitoring and alerting, key management, and access control. Developers can build on these foundations to create enterprise-grade agent applications that meet their own compliance requirements.
6. Cross-Region Deployment and High Availability
For services with users spread across wide geographic areas, availability and latency are just as critical.
The idea behind cross-region inference is to route requests to a suitable regional node close to the user. This lowers response latency overall, improves availability, and helps meet compliance requirements such as data residency.
In practice:
- Plan deployment regions based on where your users are located;
- Use monitoring metrics (latency, error rate, and saturation) to evaluate how each region performs;
- Design degradation and rate-limiting strategies to keep core request paths running during traffic peaks or outages.
Conclusion
A million-token context is not an isolated parameter race. It is a systems engineering effort that brings together long context, high-throughput inference, prompt caching, security and governance, and cross-region deployment. What ultimately determines whether an agent application succeeds is how well it turns powerful model capabilities into stable, reliable business value under low-latency, low-cost, and governed conditions.
For developers, this means understanding these general technical fundamentals and making good use of the compute, monitoring, and security foundations provided by platforms such as Baidu AI Cloud. That is what it takes to build truly useful intelligent applications in the era of ultra-long context.
