A Technical Deep Dive into Open-Weight Long-Horizon Agent Models: Architecture and Deployment Guide

1. The Technical Inflection Point of Long-Horizon Agents

As AI agent applications transition from short conversational Q&A to multi-step task execution, the core challenges faced by models have fundamentally changed. Context lengths have surged from thousands of tokens to hundreds of thousands—or even a million. Task chains have expanded from single-turn reasoning to long-horizon planning involving dozens of tool calls. Input modalities have evolved from pure text to mixed image-and-text formats. These shifts impose entirely new requirements on a model’s attention efficiency, reasoning budget scheduling, and tool-call consistency.

Recently, an open-source community released a large-scale Mixture-of-Experts (MoE) model tailored for long-horizon agent scenarios, opening its full weights for research and enterprise deployment. Building upon its previous foundation model, this release underwent large-scale joint reinforcement learning (RL) post-training. It features a MoE architecture with 2.8T total parameters and 104B activated parameters, supporting a 1-million-token context window and native image-text input. This article analyzes the model across four dimensions: architecture design, training methodology, evaluation performance, and deployment practices, offering a systematic reference for developers looking to understand and implement long-horizon agent models.

2. Architecture Design: Sparse Attention and Mixture-of-Experts

2.1 The Trade-off Logic of MoE Parameter Configuration

The model employs a sparse MoE architecture with 2.8T total parameters and 104B activated parameters. The core consideration here is that total parameters dictate the model’s knowledge capacity and expressive power, while activated parameters determine the computational cost of a single forward pass. The 2.8T total parameter volume provides a knowledge reserve covering hundreds of downstream domains, whereas the 104B activated parameter count keeps the per-token computational overhead within acceptable limits, making million-token long-horizon tasks engineering-feasible.

For deployment teams, this architecture means memory footprint and compute throughput must be planned independently: weight loading requires accommodating all 2.8T parameters (which still demands substantial VRAM or RAM even under MXFP4 quantization), while inference compute is estimated based on the 104B activated parameters. When doing capacity planning, developers should distinguish between the “weight residency cost” and the “per-inference compute cost” as two separate metrics.

2.2 Evolution of the Sparse Attention Mechanism

In long-context scenarios, the quadratic complexity of standard attention becomes a major bottleneck. This model introduces two key adjustments at the attention level:

• Gated Sparse Attention Replacement: Compared to the previous foundation model, the new version replaces 24 layers of Gated Multi-head Latent Attention (Gated MLA) with Gated DSA (Dense Sparse Attention), combined with 69 layers of a linear attention variant. Sparse attention selects only 2,048 historical tokens to participate in the attention computation for each calculation, compressing the attention overhead of long sequences from quadratic growth to controllable sparse sampling.
• Historical Token Selection Strategy: Selecting 2,048 historical tokens per computation means the model must dynamically decide during inference which historical information is relevant to the current token. This mechanism is crucial for the coherence of long-horizon tasks—after dozens of tool calls, the model still needs to accurately retrieve key constraints and intermediate results from earlier turns.

This hybrid design of “sparse selection + linear attention” significantly reduces computational complexity while preserving long-range information accessibility. Developers must understand that sparse attention does not simply discard historical information; rather, it learns through a gating mechanism “when to look back at which part of the history.” This directly impacts task consistency for long-horizon agents.

2.3 Native Image-Text Input

The model supports native image-text input, meaning visual and textual information are jointly modeled within the same representation space, rather than being post-stitched via an external vision encoder. For agent tasks involving UI operations, document understanding, or chart analysis, this capability reduces the information loss typically associated with cross-modal conversion.

3. Training Methodology: Joint RL and Quantization

3.1 Joint RL Across Hundreds of Downstream Domains

The model’s post-training phase utilizes joint reinforcement learning covering hundreds of downstream domains. Unlike the traditional two-stage approach of “general pre-training followed by domain-specific fine-tuning,” joint RL mixes reward signals from multiple domains within the same training process. This forces the model to learn general problem-solving strategies across domains rather than overfitting to a single task.

This approach is particularly critical for long-horizon agent scenarios: real-world tasks often span terminal operations, code repairs, and document processing. The model needs a foundational ability to transfer planning capabilities across heterogeneous tasks. Joint training allows the model to internalize multi-domain behavioral patterns within a single strategy, reducing policy conflicts during task switching.

3.2 MXFP4 Weight Quantization

The weights are released in the MXFP4 quantization format. MXFP4 is a block-level floating-point quantization scheme that compresses weight storage to the 4-bit level while maintaining numerical dynamic range. For a model with a 2.8T parameter scale, quantization directly determines the deployment threshold. Full-precision weights require data center-level resources far beyond the budget of most teams, but MXFP4 makes loading the full weights feasible under limited hardware conditions.

It is worth noting that quantization introduces numerical precision loss. For numerically sensitive tasks (such as precise calculations or strict format output), developers should verify through evaluation whether the performance degradation of the quantized version falls within an acceptable range.

4. Capability Dimensions and Evaluation Analysis

The officially released evaluations cover typical long-horizon agent scenarios, defaulting to the highest reasoning budget setting. Some evaluations were completed using agent frameworks like intelligent programming tools. The results are categorized by capability dimension below:

Capability Dimension	Evaluation Benchmark	Score	Key Takeaways
Terminal Operation	Terminal Benchmark 4.0	65.7	Evaluates planning and execution of multi-step command-line tasks
Long-Horizon SWE	SWE Marathon v1.1	51.7	Evaluates continuous modification capabilities across files and turns
Code Repair	Deep SWE v1.1	75.5	Evaluates accuracy in defect localization and patch generation
Web Search	Browse Understanding Benchmark	92.1	Evaluates multi-hop retrieval and information integration
CAD Generation	Tool-integrated CAD Benchmark	92.7	Evaluates 3D design tasks combined with tool usage
Office Knowledge Work	Office QA Benchmark v2	64.4	Evaluates document processing and structured output

The score distribution reveals several characteristics. Tool-integrated CAD generation and web search scored high, indicating stable performance in “tool invocation + information integration” tasks. The score for the Long-Horizon SWE Marathon is relatively conservative, reflecting that continuous engineering tasks spanning multiple turns remain a current technical challenge. Terminal operation sits at a mid-tier level, suggesting that the fault tolerance for multi-step command execution still has room for improvement.

Developers should note three things when referencing this data: First, the evaluations default to the highest reasoning setting; actual latency and cost in production environments need separate assessment. Second, some scores rely on the orchestration logic of external agent frameworks, meaning framework quality will affect the final outcome. Third, benchmark scores do not represent performance in specific business scenarios—teams should build evaluation sets tailored to their own tasks before deployment.

5. Reasoning Budget: Three-Tier Thinking Mechanism

The model provides three tiers of reasoning budgets (low, high, max) that can be dynamically configured based on task complexity. The engineering value of this design lies in turning “reasoning cost” into an adjustable parameter:

• Low tier: Suitable for deterministic tasks like format conversion, simple extraction, and short Q&A. Prioritize this for latency-sensitive scenarios.
• High tier: Suitable for routine tasks requiring minor reasoning chains, such as single-file code modifications or standard retrieval Q&A.
• Max tier: Suitable for long-horizon planning, complex defect localization, and multi-constraint satisfaction. Official evaluations utilized this tier.

In practice, a “tiered routing” strategy is recommended: estimate complexity based on task type at the application layer, route simple requests to the low tier and complex requests to the max tier. This prevents the cost overruns that occur when all requests uniformly use the maximum budget.

# Example: Selecting reasoning budget by task complexity (pseudo-code)
def route_task(task):
    if task.type in ("format_convert", "simple_extract"):
        return {"budget": "low", "max_tokens": 2048}
    elif task.type in ("code_patch", "doc_qa"):
        return {"budget": "high", "max_tokens": 16384}
    else:  # Long-horizon planning, multi-step tool calls
        return {"budget": "max", "max_tokens": 65536}

6. Deployment Practices and Key Points

6.1 Inference Engine and API Compatibility

The weights are available in MXFP4 format and support mainstream inference engines (such as vLLM, SGLang, and specialized inference acceleration solutions). They also provide OpenAI- and Anthropic-compatible API interfaces. API compatibility lowers migration costs: existing applications built on these standard APIs can switch their model backend with minimal code changes.

During deployment, note that MXFP4 quantized weights require specific support from the inference engine’s quantization operators; ensure the chosen engine version supports this format. Additionally, a million-token context places high demands on KV Cache management, requiring long-context optimization mechanisms like PagedAttention.

6.2 Message Fidelity in Multi-Turn Tool Calls

The official guidelines specifically note: multi-turn tool calls must retain the complete historical reasoning content and tool call messages. This is the most easily overlooked engineering detail in long-horizon agent deployment. In many agent frameworks, developers truncate historical messages to save context, keeping only the tool return results. However, for this model, the historical reasoning chain is a crucial basis for the sparse attention mechanism to select historical tokens. Truncating reasoning content breaks the model’s planning continuity, leading to repeated calls or forgotten constraints in subsequent turns.

The correct approach is to fully retain the reasoning content, tool call requests, and tool return results for each turn. Only when the context approaches its limit should summary compression be used instead of direct truncation.

# Example: Dialog history structure retaining full reasoning and tool messages
messages = [
    {"role": "user", "content": task_input},
    {"role": "assistant", "reasoning": "...", "tool_calls": [...]},  # Keep reasoning
    {"role": "tool", "content": tool_result},                        # Keep results
    {"role": "assistant", "reasoning": "...", "tool_calls": [...]},
    # ... Accumulate turn by turn, apply summary compression near the limit
]

6.3 License Compliance

The weights are released under a specialized license, not a permissive open-source license like Apache 2.0 or MIT. Enterprises must have their legal teams review the license terms before commercial deployment, focusing on commercial scope, redistribution conditions, derivative model clauses, and liability limitations. Equating “open weights” with “free for commercial use” is a common compliance pitfall.

7. Conclusion

The technical roadmap of this open-weight model highlights several key engineering directions for long-horizon agents: using sparse attention to control long-context costs, leveraging joint RL to cultivate cross-domain planning capabilities, applying quantization schemes to lower the deployment barrier for ultra-large parameter models, and utilizing adjustable reasoning budgets to balance performance and cost. For developers, the focus of implementation should not be on replicating benchmark scores, but rather on: establishing tiered evaluation sets aligned with business needs, scheduling reasoning budgets by task complexity, strictly ensuring message integrity across multi-turn tool calls, and mapping out a commercial deployment path within compliance frameworks. Only by executing these engineering details properly can the capabilities of long-horizon agents be truly transformed into deliverable productivity.