Scaling Context, Extending Tasks: Capability Evolution and Engineering Practices for Ultra-Long-Context Models

1. Background: From Long Context to Long-Horizon Tasks

In recent years, large language models have continued to evolve in context length, reasoning capability, and tool use. When the context window expands to millions of tokens, the model is no longer just “able to read longer documents.” Instead, it begins to handle long-horizon tasks, cross-file engineering, multi-step planning, and terminal execution.

This shift has also changed how models are evaluated. Traditional question-answering benchmarks are gradually giving way to evaluation systems that more closely resemble real workflows, such as terminal-constrained tasks, long-range code modifications, parametric modeling, and financial workflows. These scenarios share several common characteristics:

• Large input scale, potentially containing multiple files, conversation history, tool outputs, or code repository fragments.
• Long task chains, requiring the model to maintain goal consistency across multiple rounds of interaction.
• Strong verifiability, with results that can often be objectively judged through execution output, numerical computation, or automated tests.
• High requirements for factual accuracy and safety.

Therefore, understanding an ultra-long-context model should not rely on a single leaderboard score. Instead, it requires a holistic assessment that combines training strategy, inference cost, tool-use capability, and performance across different task types.

2. Model Capability Positioning: Code, Agents, and Long-Horizon Reasoning

Taking a recently released 3-trillion-parameter model as an example, its design focuses on three areas: code capability, agentic behavior, and knowledge benchmarks. According to official information, the model supports a 1-million-token context window and approaches or matches the stronger level of current closed-source models on multiple internal and external evaluations.

2.1 Code and Long-Horizon Agents

Coding is one of the most direct application areas for long-context capability. A typical long-range coding task may involve the following steps:

1. Inspect the project structure and understand module dependencies.
2. Locate the code fragments that need modification.
3. Generate a patch and explain the rationale for the change.
4. Run tests or static checks.
5. Iterate based on feedback.

In this workflow, the model must simultaneously handle code understanding, tool invocation, error diagnosis, and long-horizon planning. Evaluations often use terminal-constrained benchmarks to measure performance in an environment closer to real development. For example, on one terminal task benchmark, the model achieved 65.7%, only slightly behind the 66.4% of a leading closed-source model.

Long-range code benchmarks also show that the model improves significantly over its previous generation on tasks requiring cross-file, multi-turn modifications. The difficulties in such tasks include:

• Context information can easily be lost.
• Errors in intermediate steps can accumulate.
• The model must continuously track what has been changed and what remains unfinished.

As a result, the model needs to be trained with more long-horizon trajectory data and optimized for multi-step decision-making through reinforcement learning.

2.2 Tool Use and 3D Modeling

Beyond text-to-code generation, the model is also used to call game engine interfaces, write modeling scripts, and generate parametric 3D objects. For instance, by invoking engines such as Unity, Unreal Engine, or Godot, the model can perform scene setup, asset importing, and script generation. By writing Blender or CAD scripts, it can achieve parametric modeling.

The value of these tasks lies in turning the model’s output from “textual suggestions” into “executable results.” In tool-assisted evaluations such as BenchCAD, the model scored 92.7%, close to the level of leading closed-source models. This indicates that the model can not only understand modeling requirements but also select the right tools, generate valid scripts, and process execution feedback.

For developers, this means the model can be integrated into existing design, simulation, or content production pipelines to automatically generate editable engineering files, rather than merely providing textual instructions.

2.3 Financial and Office Workflows

In the financial domain, the model is used for end-to-end delivery of financial modeling, investment analysis, and industry research reports. Such tasks typically require:

• Reading large volumes of structured and unstructured data.
• Performing numerical calculations and scenario analysis.
• Producing documents that conform to industry standards.
• Maintaining consistent data definitions across a long context.

On the Pro version of an external office Q&A benchmark, the model achieved 64.4 points, which the vendor describes as the best result to date. While this score alone does not fully represent real business capability, it suggests the model performs strongly on complex office tasks.

3. Factual Accuracy and Safety Control

As long-context models expand their capabilities, they also face challenges in factual accuracy and safety. The longer the context, the more likely the model is to introduce inconsistencies or incorrect information during generation.

3.1 Factual Consistency Optimization

According to internal stress-test results disclosed by the vendor, the model reduced its single factual statement error rate from 23.4% in the previous generation to 8.3% in an evaluation that deliberately focused on error-prone cases. The proportion of answers containing at least one factual error was approximately 42.4% of the previous generation.

This improvement comes primarily from two sources:

• Cleaning and denoising training data to reduce the propagation of incorrect facts.
• Introducing a factual consistency reward during reinforcement learning to encourage the model to express itself more conservatively when uncertain.

It is worth noting that the vendor emphasizes the test deliberately amplified error-prone scenarios, so the error rate should not be directly equated with overall performance in everyday use. For developers, however, this metric remains a useful reference, especially in knowledge-intensive applications such as medical consultation, legal assistance, and financial analysis, where factual accuracy should be an important dimension in model selection.

3.2 Safety Capability Trade-offs

On the safety side, the vendor mentions identifying and removing cyberattack and defense-related data during both mid-training and reinforcement learning to reduce misuse risk. In a self-developed benchmark called xdev-bench-hard, the unfiltered version could solve 90.5% of the problems under pass@3, while the final released version solved 81.0%.

This gap illustrates the trade-off between safety and capability. The model underwent capability pruning before release, sacrificing some performance on sensitive tasks in exchange for lower misuse risk. For enterprise users, this strategy can help reduce compliance pressure, but it may also affect usability in certain security research scenarios.

4. Inference Cost and Pricing Structure

Inference costs for long-context models are typically significantly higher than for ordinary models because:

• Attention computation grows superlinearly with sequence length.
• Long sequences require more KV cache space.
• Long-horizon tasks often require multiple rounds of inference and tool calls.

To control costs, the model is offered in standard and high-speed versions, with a context caching mechanism. The pricing structure is as follows:

• Standard version: RMB 13 per million input tokens, RMB 68 per million output tokens.
• High-speed version: RMB 26 per million input tokens, RMB 136 per million output tokens.
• Cache hits: RMB 1.3 per million tokens.
• Cache writes are billed at two tiers: 5-minute and 1-hour.

The significance of context caching is that when multiple requests share the same prefix or system prompt, redundant computation can be avoided. For example, in agent scenarios, system prompts, tool descriptions, and historical context are often highly repetitive, so cache hits can significantly reduce input costs.

Developers may consider the following optimization strategies:

• Place stable, unchanging content at the beginning of the context.
• Reuse session prefixes where possible and avoid frequently switching system prompts.
• Choose between standard and high-speed versions based on latency requirements.
• Monitor cache hit rates to evaluate actual costs.

5. Engineering Recommendations for Production Deployment

For teams planning to introduce such models into their business, the following points deserve attention.

5.1 Task Decomposition and Context Management

Even with a million-token context window, it does not mean content can be filled in without restraint. It is advisable to break tasks into verifiable subtasks and maintain a concise context snapshot for each subtask. This reduces inference cost and lowers the risk of the model losing sight of the overall goal in a long context.

5.2 Toolchain Integration

In scenarios such as code generation, data analysis, and office automation, the model’s value depends heavily on the completeness of the toolchain. It is recommended to provide the model with clear, stable tool interfaces and to design output truncation and compression strategies to prevent irrelevant information from consuming context.

5.3 Evaluation and Monitoring

Before going live, establish an evaluation set consistent with the business scenario, covering factual accuracy, task completion rate, execution success rate, latency, and cost. After launch, continuously monitor cache hit rates, error type distribution, and user feedback so that prompts, tool configurations, or model versions can be adjusted in a timely manner.

5.4 Safety and Compliance

For scenarios involving sensitive data or high-risk operations, combine access control, audit logs, and human review mechanisms. Even if the model has already undergone safety pruning, it should not be solely relied upon for self-restraint.

6. Conclusion

Ultra-long-context models are moving from “reading long documents” to “completing long-horizon tasks.” Evaluation results in code generation, agentic behavior, 3D modeling, and financial workflows show that the model’s capability boundary is no longer limited to text understanding but increasingly covers executable, verifiable, complex workflows.

At the same time, factual accuracy, safety pruning, inference cost, and tool integration remain key constraints in production deployment. Developers need to find a balance between capability and cost that fits their business, and use sound architectural design and engineering practices to turn model capability into real productivity.