Million-Token Context and Long-Horizon Agents: Capabilities, Evaluation, and Production Practices

1. Why “Long Context + Long-Horizon Agents” Has Become the New Focus

The competitive frontier for large models is shifting from “single-turn answer quality” to “long-horizon task execution.” Long-horizon work means the model must sustain effort over hours or longer: understanding a codebase with hundreds of thousands of lines, executing hundreds of tool calls, and maintaining task state across turns. This creates two direct technical requirements: first, the context window must be long enough to fit the full task background; second, the model must “stay on task”—maintaining planning, error correction, and self-recovery across multiple execution steps.

From recent public information on flagship models from multiple vendors, several consensus configurations have emerged: parameter counts reaching the trillion level, context windows pushed to the million-token range, and code and agent capabilities as core upgrade areas. On third-party evaluation platforms’ composite intelligence index leaderboards, domestic models have also secured top positions. “Million tokens + long-horizon agents” is no longer a bonus feature—it is the entry threshold for a new generation of models.

2. The Technical Core of Million-Token Context

2.1 Three Layers of Challenge: From Algorithm to System

Standard Transformer self-attention computation grows quadratically with sequence length. Expanding the window from 100K to one million tokens requires engineering breakthroughs at at least three levels:

• Position encoding extrapolation: Common approaches include adjusting the base of rotary position encodings or adopting NTK-style scaling strategies so the model retains position discrimination beyond its training length.
• Attention and memory optimization: Sliding windows, sparse attention, KV cache quantization, and cross-layer sharing are widely used to flatten the memory curve for long sequences.
• System scheduling: The prefill phase for very long inputs often becomes a latency bottleneck, requiring optimization through compute-communication overlap, chunked prefill, and similar techniques.

2.2 “Nominal Support” Does Not Equal “Actually Usable”

Window length is only an upper bound. What truly determines usability is the actual efficiency of long-range information utilization. The industry commonly uses “needle-in-a-haystack” style tests to examine information retrieval capability at different depths. The approach can be summarized in the following pseudocode:

def needle_test(model, context_lengths, depths, needle="The meeting key is XK-42"):
    results = {}
    for n in context_lengths:            # e.g., 32K / 128K / 512K / 1M
        for d in depths:                 # relative position: head / middle / tail
            haystack = build_docs(n)     # fill background with long documents
            prompt = insert(haystack, needle, at=d)
            answer = model.chat(prompt + "What is the meeting key?")
            results[(n, d)] = (needle in answer)
    return heatmap(results)              # output "length x depth" recall heatmap

When selecting a model, ask vendors to provide this kind of heatmap, or reproduce the test with your own data. Pay special attention to two commonly weak scenarios: the “middle-of-context forgetting zone” in very long contexts, and multi-needle reasoning that requires cross-referencing multiple pieces of information.

2.3 Context Caching: The Cost Lifeline for Long Sessions

In million-token scenarios, one frequently overlooked mechanism is context caching: for prefixes that appear repeatedly across requests—system prompts, tool descriptions, conversation history—the computation only needs to be done once and written to cache. Subsequent requests that hit the cache can skip most of the prefill computation. Typical designs distinguish two write tiers: minute-level short-term caching (suitable for high-frequency consecutive turns) and hour-level long-duration caching (suitable for agent sessions that run throughout the day).

Consider a concrete scenario to estimate the benefit: suppose an agent carries approximately 200K tokens of shared prefix per request, a session issues 100 requests in total, and the interval between requests does not exceed the cache validity period. Without caching, the cumulative prefix cost is 100 × 200K = 20 million tokens of input fees. With caching enabled, only one write plus 99 hits are needed, costing approximately 200K × 1 + 1980K × 0.1 ≈ 2.18 million token-equivalents—an overall savings of nearly 90%. Estimating the break-even condition at typical multipliers: minute-level write pricing equals standard input pricing, so a single cache hit recovers the cost; hour-level writes cost roughly double the write fee, so two hits cover the difference. The longer the session and the more stable the prefix, the higher the cache benefit. This is why cache hit rate should be listed as a core operational metric.

3. Long-Horizon Agent Capabilities: How to Define and Evaluate Them

3.1 The Essence of Long-Horizon Tasks Is “State Maintenance + Error Recovery”

Unlike single-turn Q&A, long-horizon tasks examine the model’s sustained operational capability in constrained environments: whether task decomposition is reasonable, whether intermediate states remain consistent, and whether the model can autonomously locate and roll back from execution errors. Real terminal-based constrained-environment evaluations—tasks completed in a sandboxed terminal that require multiple steps to verify—are currently recognized as the closest assessment to production difficulty.

From recent public evaluation data, top models’ completion rates on these tasks have just crossed the 60% mark, and the gap within the first tier has narrowed to single-digit percentage points. The improvement from one generation to the next on long-horizon code benchmarks is also around 5 percentage points for the same vendor. Two conclusions emerge from these numbers together: first, long-horizon agents remain an industry攻坚 area, far from “good enough”; second, the marginal magnitude of generational improvement is narrowing, with gains beginning to shift toward engineering-side orchestration and toolchains.

3.2 Using the Right Metrics: pass@k and “Stress Metrics ≠ Everyday Metrics”

Evaluating long-horizon tasks typically reports pass@k: the model is allowed k attempts, and any single pass counts as a pass. k=1 examines deterministic capability; k=3 and k=5 examine whether “more attempts can solve it.” Enterprises should set weights based on their scenarios: automated pipelines often have only one chance, making pass@1 the hard metric; interactive assistance can tolerate retries, making pass@3 more informative.

More importantly, be wary of the gap between benchmarks and real-world distributions. A recent public factual stress test from a vendor explicitly stated that the test set intentionally focused on error-prone cases, and the error rate does not represent overall everyday usage levels. Such disclosures remind evaluators to distinguish “stress metrics” from “everyday metrics”: the former is for horizontal comparison of model upper bounds, while the latter is the basis for experience and cost estimation.

3.3 Recommended Enterprise Evaluation Card

• Scenario sampling: Draw 50–200 auto-scorable tasks from real business operations, covering short/medium/long-horizon tasks.
• Dual metric recording: Track both pass@1 and pass@3, with average step count, tool call count, and failure attribution.
• Stability: Repeat the same task 3 times to observe variance—random fluctuations in long-horizon tasks often expose problems better than averages.
• Control baseline: Keep results from the previous generation model or current solution to form a generational comparison.

4. Domain Depth: Code, Engineering Tools, and Financial Scenarios

4.1 Code and Long-Horizon Development Tasks

Code is a natural touchstone for long-horizon capabilities. The new generation of models has reached or approached the level of the strongest closed-source models on public code benchmarks, meaning enterprises can seriously consider incorporating them into development pipelines: automation rates for code review, cross-repository refactoring, and defect localization have substantial room for improvement. For deployment, start with “read-only analysis” (generating review comments, impact analysis), then gradually open up “write operations” (submitting patches) with mandatory human confirmation.

4.2 From Code to Engineering Tools: Game Engines and Parametric Modeling

A noteworthy new trend is the model’s deep mastery of professional engineering tools. Recent public evaluations of flagship models show: the model can proficiently call scripting interfaces of mainstream game engines, directly write and run parametric scripts for 3D modeling software to complete modeling tasks, and has scored above 92% in tool-assisted parametric modeling evaluations, with the gap to the industry’s strongest level narrowing to within 4 percentage points.

The engineering implication is that large models are evolving from “generating text/code” to “operating productivity toolchains.” Evaluation focus should shift from “can it write code” to “can it use tools”—including tool selection accuracy, parameter assembly precision, and self-checking capability on execution artifacts (rendered results, model files).

4.3 Financial Scenarios: From Point Generation to End-to-End Workflows

Public evaluations in the financial domain have also shown landmark progress: the model achieved approximately 64 points on office document benchmarks—the current best—and is used for end-to-end delivery of financial modeling, investment analysis, and industry research report workflows. “End-to-end” includes at least four stages: data ingestion and cleaning, modeling and computation, chart and document generation, and consistency verification of conclusions.

Financial scenarios demand far higher factuality and traceability than general scenarios. Engineering recommendations: all critical data should come from direct structured data source connections rather than relying on model memory; all conclusions must be accompanied by citations and calculation processes; high-risk conclusions should retain human review checkpoints.

5. Factual Accuracy: Stress Testing and Hallucination Governance

Hallucination remains the number one obstacle to large model deployment. A recent vendor’s publicly disclosed internal stress test data is particularly informative: through targeted training, the single-item factual statement error rate dropped from approximately 23% in the previous generation to approximately 8%, and the proportion of responses containing at least one factual error fell to about 40% of the previous generation. This data offers three insights for engineering teams:

1. Factuality can be improved through targeted training. When selecting models, look for whether the vendor has dedicated factuality evaluation and improvement mechanisms.
2. Ask how the test set was constructed. Stress test results focused on error-prone cases are typically several times higher than real everyday error rates; using them directly to estimate online experience would be severely misleading.
3. Model-side improvements cannot replace system-side safeguards. Retrieval augmentation provides factual anchors, citation tracing supports post-hoc auditing, and critical outputs retain rule-based validation and human review as fallback. Layered defense remains standard practice.

6. Safety Alignment: The Engineering Trade-off Between Capability and Risk

One issue that must be confronted directly: safety filtering has a cost. A recent vendor disclosed rare quantitative data—to reduce cyber attack and defense abuse risks, related data was identified and removed during both mid-training and reinforcement learning stages. The final release version’s pass@3 on a proprietary high-difficulty attack-defense benchmark dropped from approximately 90% in the unfiltered version to approximately 81%, representing a safety cost of about 10 percentage points.

The value of this data lies in turning the “safety-capability trade-off” from a slogan into a measurable engineering decision. Enterprise security teams can adopt a three-layer approach:

• Training side: Removing high-risk corpora is the most direct approach, but the collateral impact on adjacent legitimate scenarios (operations troubleshooting, compliance security testing) must be assessed.
• Inference side: Constrain model behavior boundaries through system prompts and tool whitelists.
• Execution side: Any command produced by the model should go through sandboxed execution, least-privilege access, and operation auditing. Do not relax defense-in-depth just because “the model is smart.”

7. Cost Modeling and Selection Practices

7.1 Understanding the Multiplier Relationships

Using a recent new-generation flagship’s public pricing structure as an example (with standard-tier input unit price P as the baseline), a set of typical multipliers can be distilled:

Billing Item	Typical Multiplier
Input	P
Output	≈ 5P
Cache hit	≈ 0.1P
Cache write (minute-level TTL)	≈ 1P
Cache write (hour-level TTL)	≈ 2P
High-speed tier input/output	≈ 2× standard tier

Two generational characteristics are worth noting: some vendors keep cache-hit pricing consistent between high-speed and standard tiers, significantly improving the overall cost-effectiveness of the high-speed tier in high-frequency long-session scenarios; meanwhile, the new generation shows an overall price reduction relative to the previous generation, presenting a scissors gap of “rising capability, falling unit price.”

7.2 Tier Selection and Caching Strategy

• Standard tier: Suitable for offline batch processing, overnight evaluation, and latency-insensitive document understanding tasks.
• High-speed tier: Suitable for interactive coding assistants and high-frequency multi-turn agent workflows—latency is amplified by the number of turns. For a task with 50 tool calls, saving 1 second per turn yields nearly 1 minute of end-to-end improvement.
• Caching strategy: Consolidate stable system prompts, tool descriptions, and domain knowledge into long prefixes and enable long-duration caching. Growing content like conversation history should be covered by short-term caching; idle periods exceeding the TTL require re-writing.

7.3 Enterprise Selection Checklist

• Retrieval and multi-needle reasoning tested under million-token windows (self-built needle-in-a-haystack tests).
• Real terminal/sandbox task completion rates (dual metrics pass@1 and pass@3).
• Clarity on the distinction between factual stress test methodology and everyday error rates.
• Disclosure of safety filtering approach and its capability cost.
• Caching mechanism: whether supported, TTL tiers, hit pricing.
• Billing transparency: itemized unit prices and multiplier relationships for input/output/cache writes.
• Compliance capabilities: private deployment, data residency, audit logs.

8. Recommended Deployment Path

A three-phase approach of “evaluation first, gray-scale rollout, monitoring loop” is recommended:

1. Evaluation phase (1–2 weeks): Build an internal evaluation card following the methodology in Section 3, covering long-horizon tasks in your domain, and establish a reproducible baseline.
2. Pilot phase (2–4 weeks): Select 1–2 low-risk scenarios for gray-scale rollout, with manual spot-checking, focusing on failure attribution and cost curves.
3. Expansion phase: Establish an online monitoring dashboard with core metrics including task completion rate, cache hit rate, per-task cost, factual error rate, and security incident count.

At the infrastructure level, general-purpose capabilities available on cloud platforms—GPU compute resources, model service hosting, object storage, and log monitoring—can be used to solidify the evaluation card and monitoring system, avoiding the need to rebuild everything with each model generation.

9. Conclusion

Million-token context and long-horizon agents are essentially two sides of the same coin: the former solves “remembering,” the latter solves “finishing.” From public evaluation data, real terminal task completion rates have just crossed 60%, indicating the industry is still on the climbing slope of long-horizon capability. Factual error rates dropping to single digits, safety filtering imposing a cost of about 10 percentage points, and cache-hit pricing falling to one-tenth of input—these numbers together paint a realistic picture of the new generation of models: more capable, but with more costs; faster, but requiring more careful planning.

For technical teams, rather than waiting for a “perfect model,” it is better to build a solid evaluation system, caching strategy, and safety boundary—these three things are the truly reusable assets in a rapidly iterating model landscape.