Million-Token Context LLMs: Capability Evaluation and Engineering Practices

1. From 128K to One Million Tokens: Long Context as the New Flagship Divide

The competitive landscape for large language models has shifted in two notable directions. On one side, flagship models are scaling toward trillion-scale parameter counts. On the other, context windows have expanded from tens of thousands of tokens to the million-token range. For developers, a million-token context means more than just reading longer documents. It changes the default assumptions behind system architecture: entire code repositories, complete contract sets, full conversation histories, and hours of meeting transcripts can now fit into a model’s working memory without aggressive chunking and summarization at the application layer.

Long context is not a free lunch, however. The core engineering challenges include:

• Attention computation complexity: Standard self-attention scales quadratically with sequence length. Million-token contexts require sparse attention, local-global hybrid attention, or other optimized structures.
• KV cache memory pressure: Longer contexts demand larger key-value caches. This calls for paged management, quantization-aware compression, and multi-tier caching.
• Long-range consistency decay: Even when needle-in-a-haystack retrieval tests approach perfect scores, cross-chapter reasoning consistency and instruction adherence can still degrade with distance. This requires dedicated evaluation.

In practice, long-context reading and retrieval-augmented generation are not opposing approaches. For high-frequency, structurally stable content, reading the full context directly yields more complete semantics. For massive dynamic knowledge bases, retrieval remains the more cost-efficient option. Mature systems typically combine both.

2. Evaluation Metrics Are Shifting: From Answer Accuracy to Real Task Completion

Third-party composite intelligence index leaderboards can serve as an initial screening tool. They aggregate reasoning, coding, and knowledge dimensions into a weighted summary, making them suitable for quickly narrowing down candidate models. A notable trend in recent years is that Chinese models have consistently entered the global top tier, with gaps between leading models often measuring only a few percentage points. This means model selection must go deeper into task-specific evaluations aligned with actual business needs.

Task-specific benchmarks generally fall into three categories: knowledge (factual coverage and QA stability), code (extending from function-level generation to repository-level understanding and bug fixing), and agentic tasks (tool calling, multi-step planning, and environment interaction).

The most telling signal of the current capability leap is real-world terminal-constrained task evaluation. Within a sandboxed terminal with restricted network access and limited tools, models must complete environment setup, troubleshooting, and data processing tasks within dozens of steps. This measures end-to-end task completion rates rather than single-point answer correctness. Two points are worth noting when interpreting such scores:

• Leading models typically score in the 50%–70% range, with gaps of only a few percentage points between them. Each percentage point gain represents meaningful progress in long-horizon planning and self-correction.
• Version-over-version comparisons (for example, long-range coding evaluation improving from around 55% in the previous generation to 61%) carry more weight than absolute scores, since they reflect real gains under the same evaluation methodology.

3. Code and Agents: From Completion Tools to Digital Workers

Model capability evolution can be summarized in three stages: single-turn completion → function/module-level generation → end-to-end workflow delivery. The third stage is the current battleground. Representative capabilities include:

• Driving professional software through scripting: For example, calling game engine scripting APIs to complete scene construction and parameter tuning, or using CAD/3D modeling scripting interfaces for parametric modeling. Evaluations are typically designed as “natural language requirements + tool documentation → verify whether the output satisfies geometric and functional constraints.” Leading models with toolchain support can exceed 90% accuracy on such evaluations.
• Vertical workflow delivery: In financial scenarios, a model can complete financial modeling, investment analysis, and industry research end-to-end, producing structured tables, formulas, and conclusions. The key failure points for these tasks are numerical reasoning consistency and correct formula citations, which should be the focus of evaluation.

For enterprise adoption, a practical recommendation is to formalize agent task descriptions into a three-part structure: objective—constraints—acceptance criteria. Pair this with automated acceptance scripts so that model outputs can be machine-verified. This is the critical step in converting benchmark scores into actual productivity.

4. Factual Accuracy: Hallucination Suppression as a Systems Problem

Two common metrics are used to measure hallucination:

• Single-statement error rate: The proportion of erroneous statements among all factual statements.
• Response-with-error ratio: The proportion of responses containing at least one factual error. This metric is closer to user experience.

It is important to understand the construction logic of stress test sets: these tests deliberately focus on error-prone cases (long-tail knowledge, easily confused entities, time-sensitive information), so their numbers are significantly higher than day-to-day usage and should not be directly equated with production error rates. Looking at version iterations, the industry’s leading edge has roughly compressed the single-statement error rate from above 20% down to single-digit percentages, with the response-with-error ratio dropping to about 40% of the previous generation. This indicates that training-side investments in data cleaning and factual consistency reinforcement have produced generational differences.

Even so, high-risk scenarios (healthcare, finance, legal) should retain a fallback design of retrieval augmentation + citation tracing + human review, treating the model as a high-efficiency draft generator rather than the final arbiter.

5. Safety Governance: Data Filtering and the Capability-Safety Trade-off

For cyberattack-related misuse risks, the mainstream approach is to move filtering upstream into the training data pipeline: identifying and removing relevant data during both mid-training and reinforcement learning stages, reducing risk at the source rather than relying solely on inference-time refusal rules.

This filtering has a quantifiable cost, observed through pass@k (the proportion of k samples where at least one passes). On a self-developed high-difficulty safety evaluation set, an unfiltered version achieves roughly 90% pass@3, while the released version reaches about 81%. The roughly 10-percentage-point gap represents the cost of safety alignment. Two engineering lessons follow: safety filtering should be pushed as far upstream into the data stage as possible, with post-hoc rules serving only as a backstop; and model evaluation must cover both capability benchmarks and safety benchmarks. Reporting either side in isolation leads to flawed decision-making.

6. The Cost Model: Token Pricing, Context Caching, and Speed Tiers

Flagship model pricing structures typically include four components:

1. Differentiated input/output pricing: Output unit prices are generally 3–5x input prices, reflecting higher decoding-stage overhead.
2. Context caching: Cache hit pricing can drop to roughly one-tenth of the base input price. Cache writes are tiered by TTL (commonly 5-minute and 1-hour tiers). Longer TTLs cost more per write: the 1-hour tier is typically 2x the 5-minute tier.
3. Speed tiers: High-speed versions trade roughly 2x unit price for higher throughput and lower latency.
4. Generational price reductions: As inference efficiency improves, same-tier model pricing typically trends downward with each generation.

A practical optimization pattern:

# Pseudocode: cache reuse strategy for multi-turn QA over long documents
prefix = build_prefix(system_prompt, knowledge_docs)   # Stable prefix; dynamic content goes after
cache  = ctx_cache.write(prefix, ttl="1h")             # Write once, hit multiple times
for q in questions:                                    # Questions in the same batch share the prefix cache
    ans = llm.chat([cache.ref(), user(q)], tier="standard")  # Use standard tier for offline batch processing

Key practices include: keeping the prefix stable (system prompts and static knowledge base content go first to maximize cache hits); matching TTL to call patterns (short TTL for dense interactions, long TTL for periodic tasks); matching tiers to scenarios (high-speed tier for real-time conversations, standard tier for offline batch processing); and reusing long-document caches (one cache write plus multiple question reads can reduce input costs by an order of magnitude).

7. Model Selection and Practice: Building Your Own Evaluation and Integration Pipeline

Operational recommendations for development teams:

1. Leaderboard for screening, business evaluation for confirmation: Use composite intelligence indices to shortlist candidates. Whether a model enters pilot testing depends on internal evaluations aligned with your business.
2. Build an internal evaluation set: Sample from real business scenarios, covering long-context understanding, tool calling, and factuality. Fix prompts and scoring scripts to create benchmarks that can be regression-tested across model versions.
3. Adopt a staged integration approach: Start with retrieval augmentation and general capability validation to prove feasibility. Then introduce long-context direct reading and cache optimization. Finally deploy agent workflows with automated acceptance checks, monitoring alerts, and gradual rollout.
4. Infrastructure support: Whether running self-hosted inference clusters or using managed model services from cloud platforms such as Baidu AI Cloud, integrate logging and monitoring alerting to continuously track token consumption, error rates, and latency percentiles.
5. Version regression: After a model upgrade, rerun the internal evaluation set to prevent situations where leaderboard scores improve but business performance regresses.

8. Conclusion

Flagship model competition has moved beyond single benchmark scores into a multi-dimensional engineering contest spanning long context × agents × factuality × safety × cost. For developers, understanding evaluation methodology matters more than remembering leaderboard rankings. Understanding caching and pricing structures affects project profitability more directly than chasing benchmark numbers. Building internal evaluation sets, automated acceptance checks, and cost observability is what ultimately converts generational advances in model capability into measurable business value.