Long Context RL Models: Open Source Architecture and Agent Practices

1. Model Scale and Context Capabilities

This open-weight large language model utilizes a Mixture of Experts (MoE) architecture with a total parameter count of 2.8T and 104B activated parameters. This design maintains high expressiveness while controlling the computational overhead of single inferences through sparse activation. The model supports a 1 million token context window, capable of processing long documents, large codebases, cross-file dependencies, and multi-turn tool call histories.

Regarding input modalities, the model natively supports image-text inputs. This means visual and textual information can be fused within a unified representation space, making it suitable for tasks like UI screenshot understanding, CAD drawing reading, and scanned document parsing. For developers, the combination of long context and multimodal capabilities reduces the need for complex preprocessing tailored to model limitations, such as converting screenshots to text descriptions or segmenting long documents for sequential processing.

From an engineering perspective, a 1 million token context window is not simply a parameter scaling exercise; it is intricately linked to position encoding, attention mechanisms, cache management, and training data distribution. In actual deployment, long contexts significantly increase the memory footprint of the Key-Value (KV) cache. Consequently, the weight provider offers quantized versions to lower the barrier to inference.

2. Mixture of Experts and Sparse Attention Design

The model introduces two significant architectural adjustments: first, replacing Gated Multi-head Latent Attention (MLA) in certain layers with Gated Dynamic Sparse Attention; second, combining this with another type of incremental attention layer. Dynamic Sparse Attention selects a fixed number of tokens from historical tokens to participate in attention calculations, with the selected historical token count set to 2048.

This design addresses efficiency challenges in long-sequence inference. Traditional full attention scales quadratically in computational complexity as sequence length grows. By selecting key historical tokens, sparse attention constrains the attention scope to an acceptable subset while preserving the context information most critical to current generation. For scenarios like code repair, terminal operations, and multi-step tool calls, historical tokens often contain repetitive content, log outputs, or completed steps. Sparse attention reduces computational load without significantly degrading performance.

It is worth noting that the selection strategy for sparse attention is an integral part of model training. Determining which historical tokens to retain involves factors such as query-key similarity, positional distance, and semantic importance. The strategy learned by the model during the joint reinforcement learning phase directly impacts its information retention capabilities in long-horizon tasks.

3. Joint Reinforcement Learning and Quantized Training

Built upon the base model, this model undergoes large-scale joint reinforcement learning (RL) post-training, covering hundreds of downstream domains. Unlike traditional Supervised Fine-Tuning (SFT), joint RL mixes multiple task types under a single training objective, including code generation, tool calling, mathematical reasoning, document QA, and UI operations. The advantage of this approach is that the model can achieve capability transfer across different tasks, reducing overfitting to a single task’s data distribution.

Both training and inference phases utilize MXFP4 weight quantization. MXFP4 is a floating-point quantization format that strikes a better balance between numerical range and precision compared to integer quantization. For large parameter models, weight quantization is a critical measure for reducing memory usage and loading costs. Quantized weights find a compromise between inference speed and model quality, making them suitable for teams requiring local deployment or self-hosted inference services.

For enterprise users, quantized weights mean services can be launched with fewer hardware resources. Under identical GPU configurations, a quantized model can support higher concurrency or longer contexts. However, quantization introduces some precision loss, particularly in scenarios requiring fine-grained numerical reasoning or complex tool calls. It is recommended to validate with real business data before going live.

4. Thinking Budgets and Agent Working Modes

Model inference offers three thinking budget tiers: low, high, and maximum. The thinking budget controls the depth of internal reasoning before the model provides a final answer. Lower tiers are suitable for simple queries, short responses, or high-concurrency scenarios; high and maximum tiers fit tasks requiring multi-step planning, code debugging, long document analysis, and extended tool calling chains.

In agent working modes, the model typically requires multi-turn interactions with external tools. For example, in a terminal operation task, the model first understands the user’s instruction, generates a command, observes the execution result, determines if adjustments are needed, and proceeds to the next step. Each tool call appends its result to the context. The model must accurately assess the current state within the expanded context and decide on the next action.

A common engineering practice is to structurally organize system prompts, tool definitions, historical thinking processes, and tool return results into a message sequence for the model to continue reasoning. During multi-turn tool calls, the model’s thinking process should not be truncated, as these hidden reasoning steps may contain task plans, state assessments, and error analyses. Directly deleting the thinking content makes the model prone to losing context in subsequent steps, leading to repetitive operations or erroneous decisions.

Below is a simplified message structure illustrating how tool call history is organized:

system: You are a terminal operation agent, you can use the following tools...
user: Check the files in the current directory and find the most recently modified log file.
assistant: I need to list the current directory contents first.
tool_call: list_files(path=".")
tool_result: ["app.log", "server.log", "README.md"]
assistant: The most recently modified log file is server.log.

In actual systems, tool calls typically exist as structured function calls rather than plain text. Regardless of format, the key principle remains consistent: completely preserving historical thinking and tool call messages benefits the stability of long-horizon tasks.

5. Performance in Code Development and 3D Creation

Official evaluations cover tasks like code repair, terminal operations, and CAD generation. In terminal benchmarks, the model scored 65.7 under the maximum thinking tier; it scored 51.7 in the Software Engineering Marathon evaluation, and 75.5 in the Deep Software Engineering evaluation. These metrics are obtained based on specific evaluation frameworks, and actual performance will be influenced by the agent framework, prompts, tool definitions, and execution environment.

Code repair tasks require more than just writing syntactically correct code; they involve understanding existing project structures, locating error sources, and verifying results post-modification. Long context is particularly crucial here, as multiple files within a repository may have reference dependencies. The model must accurately recall function signatures, variable naming, and calling conventions from relevant files within a limited attention budget.

In 3D creation, the model scored 92.7 in the CAD benchmark with tools. This task usually requires the model to understand spatial constraints, generate modeling instructions, and complete geometry creation and modification through tool calls. Multimodal input capabilities play a vital role here: the model can read design drawings or UI screenshots and convert them into executable modeling operations.

6. Performance in Knowledge Work and Retrieval-Augmented Generation

Knowledge work tasks include web search, financial research, and office document processing. The model scored 92.1 in the Web Search Reasoning evaluation and 64.4 in the Office Document QA evaluation. A common characteristic of these tasks is the need to synthesize multiple information sources, extract key facts, perform logical reasoning, and output structured conclusions.

In Retrieval-Augmented Generation (RAG) workflows, the model itself does not possess real-time information, relying instead on retrieval tools to fetch external content. A typical workflow involves: the model analyzing the question and breaking it down into sub-queries; calling retrieval tools to fetch relevant pages or document segments; deduplicating, filtering, and integrating the returned content; and finally generating an answer with cited sources.

Office document processing emphasizes structural understanding of long documents. Contracts, reports, and financial spreadsheets often contain numerous clauses, figures, and cross-references. The model must distinguish between factual and inferential information, avoiding treating unexpressed content as definitive conclusions. When designing prompts, explicitly requiring the model to provide source evidence can reduce hallucination risks.

7. Deployment and API Compatibility

The open weights support multiple inference engines, including general-purpose LLM inference frameworks. The model provides an API format compatible with common chat protocols, allowing existing applications to integrate quickly. Teams already using standard interfaces can migrate simply by replacing the model endpoint and name.

When self-hosting inference services, consider the following recommendations:

1. Memory Planning: Evaluate the KV cache size based on context length and concurrency. Enable prefix caching or paged attention if necessary.
2. Quantization Choice: MXFP4 weights suit most scenarios, but for business pipelines requiring high numerical precision, using higher precision inference is recommended.
3. Timeout Control: Long-horizon agent tasks are time-consuming. Set reasonable timeouts for Time-to-First-Token (TTFT) and complete responses.
4. Tool Call Logging: Thoroughly log each turn of tool calls and model thinking processes for easier debugging and subsequent prompt and tool optimization.
5. Streaming Output: For interactive applications, enable streaming output to reduce user perceived latency.

8. Licensing and Usage Boundaries

The model weights are released under a proprietary license, rather than a standard permissive open-source protocol. Users must carefully review the license terms to confirm whether commercial use is allowed, whether derivative models must be open-sourced, and whether model outputs are restricted from training other models. The proprietary license may impose restrictions on specific industries, use cases, or scales. Enterprise users should evaluate this with legal or compliance personnel.

For teams looking to integrate the model into their products, beyond licensing, attention must be paid to data security, output compliance, and content moderation. Especially in sectors like finance, healthcare, and law, model outputs must undergo strict manual review and risk control processes and should not be used directly as the final decision basis.

9. Engineering Implementation Recommendations

From a practical standpoint, this long context RL model is well-suited for the following scenarios:

• Long-horizon Code Agents: Cross-file refactoring, dependency upgrades, bug localization, and repair.
• Multi-step Tool Calling: Terminal operations, browser operations, data pipeline orchestration.
• Complex Document Analysis: Long contract review, research report generation, multi-source information summarization.
• Assisted Design: 3D modeling and parameter adjustment based on natural language or blueprints.

During implementation, it is advisable to first build a small-scale evaluation set to compare performance and latency under different thinking budgets using real business tasks, before deciding on a default tier. High-frequency, simple requests can be routed to lower tiers or smaller models to reduce unit costs; complex tasks can then be escalated to the maximum thinking tier.

In summary, the combination of long context, sparse attention, and joint reinforcement learning provides new foundational capabilities for agent development. However, model capability is only one part of an agent system. Tool design, state management, error recovery, evaluation systems, and license compliance are equally important. Only by building robust engineering infrastructure can model capabilities be transformed into stable, production-ready services.