Million-Token Context LLMs: Capability Evolution and Engineering Practice

Over the past two years, the context length of large language models has grown from tens of thousands of tokens to the million-token scale. This is not merely a parameter race; it is the inevitable result of upgraded application patterns: repository-level code understanding, cross-document analysis of research reports, and multi-turn tool-calling agent workflows all demand that the model stay consistent and accurate within a much longer “working memory.” Based on public information, flagship models recently released by leading vendors have generally reached trillion-parameter scale and natively support million-token context. On the leaderboards of several authoritative third-party evaluation platforms, models from Chinese AI labs continue to narrow the gap with the strongest closed-source models and are already competitive on a number of code and agentic benchmarks.

This article does not focus on any specific product. Instead, it uses million-token context as an entry point to break down the capability composition of such models, evaluation methodology, and safety and cost engineering, before closing with a practical checklist for enterprise selection and deployment.

What Million-Token Context Actually Means

1. Capacity conversion. Roughly speaking, a million tokens corresponds to several hundred thousand Chinese characters, or tens of thousands to over a hundred thousand lines of code—enough to hold the core source code and accompanying documentation of a medium-to-large project. “Putting an entire repository into the model” and “feeding a full set of materials into the model” have thus moved from the demo stage to engineering-feasible reality.

2. Changing application patterns. The value of retrieval-augmented generation (RAG) lies in “on-demand recall at controlled cost,” while the value of long context lies in “global understanding and cross-document reasoning.” In practice, the two are not substitutes: long context handles near-end complete context and cross-section reasoning, while retrieval handles low-cost recall of massive distant knowledge. Hybrid use is the most mainstream architecture today.

3. Technical enablers. Positional encoding extrapolation (e.g., rotary position encoding improvements such as position interpolation and spectrum-aware scaling), attention memory and compute optimizations (e.g., paged KV cache, IO-aware attention implementations), and sparse or hybrid architectures have together turned long context from a paper metric into a commercially viable service capability. The server-side “context caching” mechanism that accompanies them is the focus of this article’s cost analysis.

Capability Dimension 1: Code and Long-Horizon Agents

Code and agents are the most direct beneficiaries of long context—and the most fiercely contested areas in current evaluations.

1. Why agents depend on long context. An agent workflow is a loop of “perceive–plan–call tools–reflect,” and every round appends system prompts, tool definitions, historical trajectories, and tool return results. A moderately complex engineering task often approaches the million-token scale after dozens of rounds. Long context is the agent’s “working memory,” and its length and stability directly determine how complex a task the agent can carry.

2. Evaluation methodology. The industry mainly uses three types of benchmarks here. First, sandboxed terminal task benchmarks: the model is connected to a sandboxed terminal and must autonomously complete multi-step, real-world engineering tasks such as installing dependencies, modifying configurations, running tests, and fixing errors, with end-to-end completion rate as the metric. Second, long-horizon task benchmarks spanning hundreds of steps and hours, which measure how success rate decays over steps rather than a single-point score. Third, pass@k multi-sampling metrics, which reflect engineering usability better than single-pass rates. Public evaluation trends show that top models’ completion rates on sandboxed terminal benchmarks have generally entered the 60%+ range, with the gap to the strongest closed-source models narrowed to single-digit percentages; on long-horizon tasks, the new generation shows similarly significant improvement over its predecessor.

3. Engineering essentials. Deploying agents cannot rely on leaderboards alone. First, context management: summarizing and compressing historical trajectories, and writing large tool outputs to disk while backfilling only references, are standard ways to control context bloat. Second, call reliability: prefer structurally constrained tool calls from the model, with automatic retry and fallback for failed calls. Third, error recovery: explicitly encoding strategies such as “read the logs before acting on errors” in the system prompt can significantly improve long-horizon stability.

Capability Dimension 2: Tool Ecosystems and Scripted Execution

Code is becoming the “universal interface” through which models drive the external world, which gives rise to two directions worth watching.

1. Scripting vertical toolchains. In game development, mainstream game engines all expose scripting interfaces, so models can write component logic, material control, and level scripts directly against the engine’s scripting spec. In 3D design, models can drive CAD and 3D modeling software through parametric scripts, closing the loop of “given constraints → generate model → iterate on modifications.” Evaluation data shows that scores with “tools allowed” are significantly higher than with “text-only answers,” indicating that toolchain integration is a genuine differentiator.

2. End-to-end delivery of industry workflows. In finance, top models already achieve leading results on office-document QA benchmarks and support end-to-end financial modeling, investment analysis, and industry research: from data extraction, model building, and sensitivity analysis to report writing, completed in one pass. For enterprises, the evaluation focus should shift from “single-point QA accuracy” to “workflow completion quality”—any hallucination in an intermediate step propagates and amplifies downstream.

Factual Accuracy and Hallucination Suppression

Long context also enlarges the risk surface for hallucination: the longer the input, the more anchors there are to “fabricate” from.

1. Stress-testing methodology. Responsible vendors typically build adversarial internal test sets that deliberately focus on error-prone cases (confusable entities, outdated facts, locally contradictory descriptions), measured with two metrics: the per-statement factual error rate, and the proportion of responses containing at least one error. In a recent public generational comparison, one model family’s per-statement error rate dropped from about 23% to about 8%, and the proportion of responses with errors fell to less than half of the previous generation. An important caveat: these tests deliberately select error-prone samples, so their error rates do not represent overall everyday usage. When evaluating, focus on generational trends and relative changes rather than absolute values.

2. Application-layer suppression. Beyond training-side improvements, engineering-side practices are mature: retrieval augmentation with mandatory citation and source tracing, multi-model self-consistency checks for critical numbers, post-generation fact-checking pipelines, and constraints such as “must provide evidence, otherwise refuse to answer” in high-risk domains like finance and law.

Safety Alignment and Misuse Risk Control

The stronger the capability, the higher the misuse risk; safety governance must be moved upstream to the training stage.

1. Data-layer governance. A common recent practice among leading vendors is to identify and remove highly sensitive data—such as network attack/defense content—during both mid-training and reinforcement learning stages, reducing the probability of the model being misused for attacks at the source.

2. Quantified capability–safety trade-off. Take one vendor’s published internal evaluation: the unfiltered version scored about 90% pass@3 on high-difficulty attack/defense benchmarks, while the official release scored about 80%—roughly a 10-percentage-point capability sacrifice in exchange for a significant reduction in misuse risk. This trade-off approach is worth borrowing for enterprise-built evaluations: treat safety red lines as an independent scoring dimension rather than patching after the fact.

3. Enterprise-side protection. When letting models execute code or call tools, default to sandbox isolation, least privilege, and network allowlists, and keep audit logs of outputs.

Cost Model and Context Caching

If million-token contexts were billed at full price on every turn, multi-turn scenarios would quickly become unaffordable. Context caching is therefore the key companion to long-context commercialization.

1. Typical industry pricing structure. Illustrative values only (not representative of any specific vendor; amounts in RMB):

Tier	Input	Output	Cache Hit	Cache Write (short TTL)	Cache Write (long TTL)
Standard	¥10/MTok	¥50/MTok	¥1/MTok	¥10/MTok	¥20/MTok
High-Speed	¥20/MTok	¥100/MTok	¥1/MTok	¥20/MTok	¥40/MTok

Four patterns emerge: output prices are significantly higher than input prices; cache hit prices are typically on the order of one-tenth of the input price; cache write prices are tiered by time-to-live, with the short TTL tier comparable to the input price and the long TTL tier roughly twice that; high-speed, low-latency tiers typically cost 1.5–2× the standard tier, while cache hit prices stay the same.

2. A quantified example. Suppose an agent task requires 10 calls, each carrying a 1M-token context and generating 50K tokens. Without caching, the input cost is 10 × 1M × ¥10 = ¥100 and the output cost is about ¥25, for a total of about ¥125. With caching enabled (the first round written at the input price, the following 9 rounds hitting the cache), the input side drops to about ¥19, for a total of about ¥44—a saving of roughly 65%. The more rounds and the more stable the prefix, the greater the caching benefit.

3. Cache-oriented prompt engineering. Cache hits require strictly identical prefixes, so place stable content—system prompts, tool definitions, reference documents—at the front of the message stream and dynamic content after it. In multi-turn scenarios, avoid inserting volatile fields such as timestamps into the prefix.

# Pseudocode: cache-friendly call structure for a multi-turn agent (API is illustrative)
stable_prefix = [system_prompt, tool_specs, project_docs]   # stable prefix, improves cache hit rate
while not done:
    dynamic = compact(history, keep_recent=8)               # compress history; dynamic content goes last
    resp = llm.chat(
        messages = stable_prefix + dynamic + [current_step],
        cache   = {"policy": "auto", "ttl": "5m"},
    )
    for call in resp.tool_calls:
        result = sandbox.execute(call)                      # sandboxed execution with least privilege
        history.append(summarize(result))                   # summarize large outputs before appending
    done = resp.finished

Enterprise Selection and Deployment Checklist

1. Evaluation. Use public benchmarks only for initial screening; you must build your own evaluation set from real business samples. Focus on pass@k, long-horizon success-rate decay curves, and compatibility with your own toolchain—not single-point scores.

2. Architecture. Combine long context with RAG; design memory in layers (cache–summary–external storage); route by latency tolerance—batch tasks with high latency tolerance go to the standard tier, interactive tasks to the high-speed tier.

3. Cost. Make cache hit rate a core monitoring metric; it is the first lever on long-context application cost. Batch tasks can exploit off-peak windows for further savings.

4. Safety. Include safety evaluation in selection scoring; sandbox all execution environments; enable mandatory citation and refusal mechanisms in highly sensitive scenarios.

5. Infrastructure. Managed model services from cloud providers such as Baidu AI Cloud can be combined with commodity capabilities—object storage, message queues, logging, monitoring and alerting—to build a complete pipeline from evaluation and staged rollout to production monitoring, avoiding reinventing the wheel.

Conclusion

Million-token context is not a marketing number; it is infrastructure for the agent era. The right way to evaluate it is to look past the leaderboards at three things: long-horizon task stability, the generational trend of factual error rates, and end-to-end cost including caching. Get evaluation, safety, and cost engineering done up front, and long context can evolve from “fitting an entire repository” to “delivering an entire workflow.”