Million-Token LLMs: Capability Evaluation and Engineering Practices

1. Why Long Context is the New Competitive Focus

The competitive focus of Large Language Models (LLMs) is shifting from “single-turn Q&A quality” to “long-context capacity + long-horizon task execution.” As context windows expand from tens of thousands to the million-token level, the model’s role fundamentally changes. It is no longer just a retrieval-augmented Q&A tool; it becomes a foundational system capable of loading entire code repositories, comprehensive business documents, and complete task histories.

This shift means three things for developers:

• Global understanding becomes possible: Cross-file dependency analysis and long-cycle requirement tracing no longer rely on fragmented slice-based retrieval.
• Interaction paradigms change: A single request can carry the complete background, reducing information loss caused by multi-turn context patching.
• Cost structures change: Token throughput scales up by an order of magnitude, making caching and billing strategies a first-class concern in architectural design.

2. Technical Value and Engineering Challenges of Million-Token Windows

2.1 Application Value

1. Repository-level code understanding: In traditional modes, models only see the current file or fragment, missing cross-file dependencies and historical evolution. This often results in generated code that is “locally correct but globally mismatched.” A million-token window allows small-to-medium repositories to fit entirely into the context, enabling the model to perform architecture analysis, impact analysis, and batch refactoring planning in a single request.
2. Long-document and multi-document analysis: Financial reports, research papers, legal volumes, and compliance files often reach hundreds of thousands of words. Combined with citation tracing, long contexts allow “reading the whole document” instead of “summarizing segments and then patching them together.”
3. Memory capacity for long-horizon Agents: Multi-stage tasks require continuously recording intermediate states, tool call results, and failure lessons. A long window serves as the most direct memory carrier.

2.2 Engineering Challenges

• Attention overhead: The computational cost of standard attention grows quadratically with sequence length. The industry commonly controls this overhead through sparse/sliding window attention, IO-aware operator optimization, and Grouped-Query Attention (GQA).
• Position extrapolation: Training length is limited, while inference length expands significantly. This requires extrapolation strategies like Rotary Position Embedding (RoPE) combined with length scaling to ensure semantic relationships at distant positions.
• KV Cache management: Cache memory usage grows linearly with length. Mechanisms like paged management are needed to improve utilization.
• “Long but not forgetting”: Research widely observes a phenomenon where information utilization drops in the middle of long texts. This requires injecting a sufficient proportion of long-sequence data during training, and validating performance using “needle in a haystack” retrieval tests and real long-document Q&A.

In short: the nominal window length is “capacity,” but the effective context length is “capability.” Selection should focus on the latter.

3. Code and Long-Horizon Agents: Evolution of Evaluation Dimensions

3.1 From Single-Point Generation to End-to-End Delivery

Classic function-level code generation benchmarks are nearing saturation, and their ability to differentiate models has dropped sharply. The new generation of evaluations focuses on constrained tasks in real environments: completing a closed loop of environment configuration, dependency installation, code writing, runtime debugging, and error fixing within a sandbox terminal. Based on recent public industry evaluations, top models’ success rates on these “real terminal tasks” have generally entered the 60%–70% range. The bottleneck is no longer single-point code correctness, but the closed-loop stability of “task planning — tool calling — error recovery.”

3.2 Key Metrics for Long-Horizon Tasks

When evaluating long-horizon Agent capabilities, it is recommended to focus on a combination of metrics rather than a single score:

• Success rate (pass@k) over multiple attempts and its stability;
• Constraint maintenance capability across dozens of tool calls: whether the model still remembers the initial task requirements;
• Failure recovery capability: whether the model can autonomously adjust its strategy after a sub-task fails, rather than falling into a loop of retries;
• Cost per unit task: total Token consumption and expense required to complete the task.

3.3 Tool Calling and Engineering Ecosystem

The capability differences in this generation of models are more reflected in “driving external tools”: proficiently operating mainstream game engines to build scenes, driving parametric 3D modeling tools via scripts to complete a “modeling — rendering — exporting” pipeline, and calling data and documentation tools to generate standardized deliverables. Benchmarks evaluating these capabilities generally adopt a joint “model + toolchain” scoring system. From industry practices, scores for parametric modeling tasks can reach over 90% after introducing toolchains. However, the gap between models often lies in their semantic understanding of tool error messages and the speed of iterative correction—which is also a primary direction for future iteration.

3.4 Vertical Domain Workflows

Taking financial scenarios as an example, financial modeling, investment analysis, and industry research reports require an end-to-end pipeline of “data extraction — calculation modeling — document generation.” The evaluation focus for these tasks has shifted from “generated text quality” to “delivery completeness”: table consistency, chart correctness, and citation traceability are all included in the scoring. Enterprises should prioritize examining the model’s end-to-end performance on their own business samples rather than general leaderboard rankings.

4. Factual Accuracy and Hallucination Mitigation

4.1 Design Principles for Stress Testing

Serious factual evaluations should follow three principles:

• Deliberately focus on error-prone scenarios: vague references, time-sensitive information, numerical calculations, long-tail entities, and plausible knowledge confusions, rather than uniformly sampling everyday questions;
• Distinguish statistical metrics: The per-fact error rate is different from the “proportion of answers containing at least one error.” The latter directly determines usability, and the two must not be conflated;
• Declare applicable scope: The error rate from stress tests cannot be extrapolated to everyday usage levels.

According to public industry information, the per-fact error rate of new-generation models in high-difficulty factual stress tests has dropped from the 20% level of the previous generation to under 10%. The proportion of answers containing errors has been compressed to about 40% of the previous generation. This is primarily attributed to training data cleaning and deduplication, reinforcement learning for factual preferences, and long context itself—placing authoritative materials directly into the window reduces reliance on parametric memory.

4.2 Runtime Hallucination Mitigation

In addition to model-level improvements, the engineering side can stack three layers of guardrails: Retrieval-Augmented Generation (RAG) to provide sourced evidence, structured output constraints to reduce free-form hallucinations, and rule validation for critical numerical values. For high-risk scenarios like finance and law, a three-line defense of “model improvement + runtime guardrails + manual spot-checking” is recommended.

5. Safety Alignment and Data Governance

Greater capability brings a wider potential for misuse, especially in areas like network offense/defense and automated penetration testing. The prevailing governance path in the industry involves phased interventions:

• Mid-training phase: Identify and remove high-risk data such as network offense/defense materials, reducing risk from the knowledge supply level;
• Reinforcement learning phase: Shape behavioral boundaries through safety reward signals;
• Pre-release red teaming: Quantify the “capability-safety” trade-off and disclose it publicly.

An engineering practice worth noting is trade-off transparency: some teams simultaneously publish the gap between the unhardened internal version and the final released version on high-difficulty safety evaluations (e.g., pass rate converging from ~90% to ~80%), letting users clearly understand the capability cost of safety hardening. Enterprise procurement should include “whether safety testing methods and trade-offs are disclosed” as an evaluation criterion.

6. Cost Engineering: Context Caching and Tiered Billing

Long context amplifies cost issues by an order of magnitude. For an Agent application that repeatedly loads system prompts, tool definitions, and complete history every turn, the unit task cost will be unbearable without a caching mechanism. Therefore, the industry generally introduces prefix caching (Context Caching). Its pricing structure typically follows these patterns (the table below is a typical structural example, not an actual quote from any vendor):

Billing Item	Typical Structure
Standard Input	Base unit price
Output	4–6 times the input price (decoding overhead is significantly higher than pre-fill)
Cache Hit	About 1/10 of the input price
Cache Write	Tiered by storage duration: short-term roughly equals input price, long-term is about 2 times
High-Speed Tier	Prices for all items are about 2 times the standard tier

Corresponding engineering optimization key points:

1. Keep the prefix stable: Place high-reuse content like system prompts and tool definitions at the head of the message sequence, and variable content at the tail, to maximize cache hits;
2. Choose speed tiers by scenario: Interactive coding assistants and online Agents prioritize the high-speed tier (latency dictates experience); offline batch processing and nightly pipelines use the standard tier with caching (cost-optimal);
3. Control output length: Output Token pricing is multiples of input. Controlling output scale through structured constraints and concise instructions yields benefits that are often underestimated.

A schematic prompt assembly order is as follows:

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},    # High-reuse prefix → cacheable
    {"role": "system", "content": TOOL_DEFINITIONS}, # High-reuse prefix → cacheable
    *history_messages,                                # Appended growth → partial hit
    {"role": "user", "content": user_query},         # Variable content placed at the tail
]

7. Deployment Guidelines

1. Build a private evaluation set: Use public leaderboards only for initial screening. Stratify and sample from real business tasks to build a private evaluation set covering typical, boundary, and high-risk tasks;
2. Joint capability evaluation: Evaluate effective context length, Agent closed-loop success rate, hallucination rate, and unit task cost jointly to avoid being misled by a single score;
3. Cache-friendly engineering structure: Write cost optimization into architecture reviews rather than treating it as an afterthought;
4. Proactive safety compliance: For high-risk scenarios, require model service providers to provide data governance and safety alignment documentation;
5. Progressive deployment: Move from single-point assistance (completion, Q&A) to workflow automation (end-to-end document delivery), and finally to long-horizon autonomous Agents. Set manual checkpoints and rollback plans at each stage.

At the infrastructure level, teams can also leverage model services and supporting general capabilities (inference acceleration, caching, logging, and monitoring alerts) provided by cloud platforms like Baidu AI Cloud. This allows teams to focus on business-side evaluation and process orchestration rather than building inference and governance systems from scratch.

Conclusion

The combination of million-token context and long-horizon Agent capabilities is pushing LLMs from “conversational tools” to “delivery systems.” For engineering teams, the real threshold is no longer successfully calling an API, but establishing a complete evaluation system: capability should be measured by effective context rather than nominal window size; reliability should be measured by the combination of stress testing and runtime guardrails; cost should be measured by unit task overhead under a caching structure. Solidifying these three aspects is what allows long-context models to truly step out of demos and into production environments.