A Practical Guide to Million-Token Contexts and Agentic Programming

1. Background: Entering the Million-Token Era

Over the past two years, the context window lengths of large language models (LLMs) have grown rapidly from 8K and 32K to 128K and 200K. Today, certain models have pushed this boundary to the 1 million token mark. This evolution is not merely a numerical increase; it fundamentally shifts application architecture paradigms:

• Past: Developers had to split long documents into chunks, perform vector retrieval (RAG), and stitch the Top-K chunks into the prompt.
• Present: Developers can feed an entire code repository, a complete set of API documentation, or a full research report into the context at once, allowing the model to reason with complete information.

For developers, this implies two key takeaways: First, the complexity of the “retrieval-and-stitching” pipeline can be significantly reduced. Second, the new bottlenecks have shifted to three dimensions: context caching costs, long-horizon task stability, and factual accuracy. This article outlines engineering practices for deploying recent models with trillion-level parameters and million-token context capabilities in code and agent scenarios.

2. Core Capabilities: Code Generation and Long-Horizon Agents

Based on public benchmarks, the capability upgrades of this generation of models focus primarily on code generation and long-horizon agentic tasks, achieving knowledge benchmark scores on par with or approaching mainstream closed-source models. Understanding the applicability of these metrics is more critical than memorizing the specific numbers.

2.1 Terminal-Bound Tasks: Real-World Execution Capabilities

In benchmarks evaluating real-world terminal-bound tasks (such as Terminal-Bench-style evaluations), this class of model scores around 65.7%, showing a marginal gap compared to top-tier closed-source models at 66.4%, and outperforming another mainstream model at 58.2%. Key characteristics of these evaluations include:

• Tasks are executed in a real command-line environment, rather than pure text-based Q&A;
• The model must autonomously plan multi-step operations, handle errors, and perform rollbacks/retries;
• The focus is heavily on “whether the model can actually complete the task” rather than just “whether it sounds correct.”

On internal long-horizon code benchmarks, the score has increased from approximately 55.95% to 61.2%, indicating substantial progress in state maintenance and error correction during multi-step tasks.

2.2 3D Modeling and Script Generation

These models can proficiently interact with mainstream game engines, directly writing and executing scripts to complete parametric 3D modeling. In tool-assisted CAD-style benchmarks, it scores 92.7%, closely trailing top-tier closed-source models at 96.2% and 95.9%.

The key takeaway here is not simply “writing scripts,” but rather the semantic understanding of parametric modeling: The model must translate natural language geometric constraints into an executable sequence of modeling commands, and reversely correct parameters based on error messages if execution fails.

2.3 Financial and Office Workflows

For workflows involving financial modeling, investment analysis, and industry research reports, these models can deliver end-to-end results. In external office document benchmarks (OfficeQA-style), it achieves a score of 64.4. The challenges of such tasks include:

• Dispersed data sources (spreadsheets, body text, attachments);
• Strict calculation pipelines, where a single mid-step error invalidates the entire output;
• High formatting requirements for outputs (structured tables + narrative conclusions).

3. Factual Accuracy: Quantification and Optimization

Hallucination is one of the most prominent issues in long-context models. The performance of this model class in internal stress tests (Hallu Bench Hard-style) is noteworthy:

Metric	Previous Gen	Current Version
Single factual statement error rate	23.4%	8.3%
Ratio of answers containing at least one factual error	Baseline	Approx. 42.4% of the previous gen

Two boundary conditions must be noted when interpreting these results:

1. This test deliberately focuses on error-prone scenarios; the error rate does not represent the overall level in daily usage.
2. The absolute decrease is more informative than the relative ratio—a drop in single-statement error rate from 23.4% to 8.3% implies that the cost of manual review in high-risk scenarios can be significantly reduced.

Engineering Recommendation: Even with reduced error rates, in fact-sensitive scenarios (legal, medical, financial disclosures), a dual-insurance mechanism of “citation tracing + manual spot-checking” should be maintained. Do not eliminate validation steps simply because the metrics look favorable.

4. Safety and Alignment: Data-Level Governance

Safety capabilities are built during the mid-training and reinforcement learning phases. The officially disclosed approach involves identifying and removing cyber-offensive data during these phases to mitigate abuse risks.

Comparative data on an internal safety benchmark (xdev-bench-hard-style) is as follows:

• Unfiltered version: pass@3 resolves 90.5% of problems;
• Final released version: 81.0%.

This data indicates that safety alignment indeed results in a capability “loss” of about 9.5 percentage points. This is a reality that all teams engaged in safety governance must face—there is an inherent trade-off between safety and capability. The key is keeping the loss within an acceptable range rather than pursuing zero risk.

5. Cost Structure: Caching as the Key Variable for Long Contexts

The cost model for long-context applications differs entirely from traditional short-context models. Below is the pricing structure for this class of model (Unit: CNY / Million Tokens):

Model	Input	Output	Cache Hit	Cache Write (5min)	Cache Write (1h)
Standard	13.00	68.00	1.30	13.00	26.00
High-Speed	26.00	136.00	1.30	26.00	52.00

Key observations:

1. The output price is roughly 5 times the input price. Therefore, models should be constrained to “avoid fluff,” reducing invalid tokens via structured outputs.
2. The cache hit price is only 1/10th of the input price, serving as the biggest cost lever in long-context scenarios.
3. The High-Speed version is twice the price of the Standard version, yet the cache hit price remains identical, making it ideal for latency-sensitive scenarios with highly repetitive contexts.

5.1 Caching Strategy Practices

The standard approach for long-context applications is a “stable prefix + dynamic suffix“:

# Pseudo-code: Place system prompts and fixed docs in the stable prefix, and user input in the dynamic part
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},      # Stable
    {"role": "user", "content": LONG_STATIC_DOCS},     # Stable, hits cache
    {"role": "user", "content": user_query},           # Dynamic, changes per request
]

Best practices:

• Place content that does not change per request at the very beginning to maximize cache hits;
• Cache writing offers 5-minute and 1-hour tiers; the 5-minute tier is more cost-effective for short sessions;
• Avoid inserting volatile fields like timestamps or random IDs into the stable prefix, as this will invalidate the entire cache.

5.2 Cost Estimation Example

Assume a single request: 200K input tokens (190K hitting cache), 2K output tokens:

Input cost = 10K × 13/1e6 + 190K × 1.3/1e6 ≈ 0.13 + 0.247 ≈ 0.377 CNY
Output cost = 2K × 68/1e6 ≈ 0.136 CNY
Total ≈ 0.51 CNY / request

Without caching, the input cost alone would approach 2.6 CNY, a difference of roughly 5x. This is why, in long-context applications, cache design is not an optimization, but a mandatory requirement.

6. Implementation Recommendations: Scenario Selection Checklist

Based on the capability characteristics outlined above, the following selection reference is provided:

Scenario	Recommended Strategy
Full-repo code understanding and refactoring	Million-token context + stable prefix caching; avoid chunk retrieval
Multi-step Agentic tasks	Prioritize long-horizon benchmark performance; configure retries and state checkpoints
3D Parametric modeling	Script generation + execution sandbox; feed errors back for correction
Financial reports / Financial modeling	End-to-end workflow + manual review of key figures
High-frequency short requests	Standard version + 5-minute cache; control output length

7. Conclusion

The combination of million-token contexts and long-horizon agentic capabilities is transforming “whether a model can finish a task” into a solvable engineering problem. When selecting and deploying models, developers should focus on four key points:

1. Long-horizon stability takes precedence over single-step accuracy;
2. Factual accuracy should be benchmarked against stress test data, but always retain a manual fallback;
3. Safety alignment incurs a capability loss that must be weighed against business risk tolerance;
4. Cache design is the core mechanism for controlling costs in long-context applications.

By clarifying these dimensions, ultra-long context becomes more than just “a longer prompt”—it evolves into a productivity tool that genuinely reshapes architectural design.