A New Paradigm for Open-Source LLMs: A Technical Breakdown of Long-Horizon Agent Capabilities

1. Background: Why Long-Horizon Agents Are the New Capability Divide

Over the past year, competition among large language models has shifted rapidly from “single-turn answer quality” to “multi-step task completion.” A long-horizon agent is a model that can maintain goal consistency across dozens or even hundreds of reasoning steps—repeatedly calling external tools, reading intermediate results, and revising its own plan—until a complex task is complete. Typical tasks include terminal operations, cross-file code repair, parametric 3D model generation, and multi-source information retrieval and synthesis.

The difficulty of long-horizon tasks is not just “longer context.” It comes down to three things:

1. The conflict between context length and compute cost. Attention computation grows quadratically with sequence length. Pushing context to the million-token level naively makes inference cost prohibitive.
2. State maintenance across multi-turn tool calls. Every request, response, and reasoning step in a tool call must be fully preserved, or the model loses the basis for its decisions.
3. Elastic control of the reasoning budget. Simple tasks do not need deep thinking; complex tasks need more “thinking tokens.” A one-size-fits-all reasoning strategy either wastes compute or falls short on capability.

A recently released open-source model offers a systematic answer to all three problems: a Mixture-of-Experts (MoE) architecture with very large total parameters and a much smaller activated-parameter count, support for million-token contexts and native image-text input, and a combination of sparse attention and joint reinforcement learning that pushes long-horizon agent capability to a usable level. This article analyzes the model along four dimensions—architecture, training, evaluation, and deployment—and closes with engineering recommendations.

2. Architecture: MoE Plus Sparse Attention

2.1 The Trade-offs Behind the MoE Design

The model uses a MoE architecture with roughly 2.8T total parameters and about 104B activated parameters. The core logic of this configuration: total parameters carry knowledge capacity, while activated parameters control per-inference cost.

In a MoE, each token is routed to only a few expert networks, so the parameters actually involved in computation are far smaller than the total. This lets the model hold an enormous knowledge base while keeping memory footprint and latency relatively manageable. For long-horizon agent workloads this matters especially: long contexts already put heavy pressure on the KV cache, and a high activation count would push inference cost out of control quickly.

The main engineering challenges of MoE lie in routing balance and communication overhead. With expert parallelism, distributing tokens across devices adds latency, and real-world throughput depends on the routing strategy and batching approach.

2.2 Sparse Attention: Making Million-Token Contexts Feasible

In the new version, Gated Multi-head Latent Attention (Gated MLA) in some layers is replaced with Gated Sparse Attention (Gated DSA), combined with dozens of Delta Attention layers. DSA selects only about 2,048 historical tokens for each attention computation.

The technical implications break down as follows:

• MLA (Multi-head Latent Attention): compresses the KV cache through low-rank projection to reduce memory usage—the baseline optimization for long contexts.
• DSA (Sparse Attention): instead of letting every token attend to all history, it dynamically picks the most relevant historical tokens. This reduces attention complexity from O(n²) to O(n·k), where k is the number of selected tokens.
• Delta Attention: a linear-attention variant that maintains long-range memory through incremental state updates, with near-linear complexity.

Together, the three form a layered strategy: some layers handle high-precision local modeling, while others handle low-cost long-range memory. Only with this combination does million-token context become practically deployable.

One caveat: the effectiveness of sparse attention depends heavily on the quality of the token selection strategy. If the selected history misses critical information, the model “sees but cannot recall.” The selection network itself therefore also needs to be optimized during training.

3. Training: Joint Reinforcement Learning and Quantization

3.1 Joint RL Across Domains

On top of the base model, the team ran large-scale joint reinforcement learning post-training spanning hundreds of downstream domains. “Joint” here has two meanings:

• Joint tasks: code, retrieval, office documents, tool calling, and more are mixed during training to prevent overfitting to a single domain.
• Joint rewards: different tasks use different reward signals (e.g., execution correctness, retrieval hit rate, format compliance), requiring a unified optimization objective that keeps them from interfering with each other.

For long-horizon agents, reinforcement learning matters more than supervised fine-tuning. Multi-step tasks rarely have a single “correct answer”; supervised signals cannot cover every reasonable path, whereas outcome- or process-based rewards can guide the model toward better strategies.

3.2 Quantization and Deployment Friendliness

Training used MXFP4 weight quantization, and weights are released at the same specification. MXFP4 is a 4-bit floating-point format with better dynamic range than conventional INT4, well suited to large models with widely varying weight distributions.

Quantization directly cuts memory usage dramatically, allowing an ultra-large model to run on fewer accelerators. But engineering teams should still note:

• Quantization may hurt long-tail capabilities (low-resource languages, rare code patterns), so validate it in your target scenario.
• The inference framework must support the corresponding quantized kernels; otherwise it falls back to higher-precision computation and the benefit disappears.

4. Benchmarks: Reading the Numbers

Official benchmark results (using the highest thinking budget by default; some evaluations were combined with agent frameworks):

Benchmark	Score	Capability Area
Terminal-Bench 4.0	65.7	Terminal environment operations
SWE Marathon v1.1	51.7	Long-horizon software engineering
DeepSWE v1.1	75.5	Complex code repair
BrowseComp	92.1	Web retrieval and information synthesis
BenchCAD (with tools)	92.7	3D CAD generation
OfficeQA Pro v2	64.4	Office document Q&A

The distribution shows standout performance on retrieval and tool-augmented tasks, indicating strong tool-calling and multi-step planning. Scores on ultra-long-horizon software engineering are more moderate, which is consistent with the industry overall—error accumulation over long tasks remains a universal challenge.

Keep in mind that benchmark scores and real-world results differ. Evaluation environments are controlled, whereas in production, unstable tools, noisy inputs, and permission restrictions all significantly affect performance.

5. Engineering Recommendations

5.1 Tiered Reasoning Budgets

The model offers low, high, and max thinking-budget tiers. In practice, route by task type:

• low: format conversion, simple extraction, single-turn Q&A.
• high: multi-step retrieval, moderately complex code changes.
• max: long-horizon agents, cross-file refactoring, complex planning.

A routing layer can pick the tier automatically based on the task description, avoiding the cost of running every request at the highest tier.

5.2 State Management for Multi-Turn Tool Calls

The official guidance is explicit: multi-turn tool calls must preserve the complete thinking and tool-call messages. This is the core engineering constraint for long-horizon agents. Use a structured message format in session storage, for example:

{
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "...", "tool_calls": [...]},
    {"role": "tool", "content": "...", "tool_call_id": "..."}
  ]
}

When truncating history, truncate whole turns. Never keep a tool result while dropping the corresponding tool call, or the model will be unable to tell where the result came from.

5.3 API Compatibility and Framework Choice

The model is served through an API compatible with mainstream interface protocols and supports multiple inference frameworks. For teams with existing agent systems, this means low migration cost—usually just swapping the service endpoint while keeping the existing tool-orchestration logic.

Note that inference frameworks differ in their support for sparse attention and quantized kernels. Run small-scale load tests on your target framework first to confirm throughput and latency under long contexts.

5.4 Licensing and Compliance

The weights use a custom license rather than a common permissive open-source license. Enterprises should read the terms carefully before commercial use—usage scope, redistribution restrictions, and derivative-model policies—to avoid compliance risk. This point is often overlooked, but it is critical for scaling deployment.

6. Takeaways

The value of this open-source model lies not in any single metric, but in packaging “ultra-long context + sparse attention + joint reinforcement learning + quantized deployment” into a reusable long-horizon agent recipe. Three directions deserve developers’ attention:

1. How the sparse-attention selection strategy affects long-range memory, and whether the selection network can be fine-tuned on your own data.
2. How to combine tiered reasoning budgets with business-level routing to balance cost and quality.
3. How to design storage and truncation strategies for multi-turn tool calls so that long-horizon tasks never lose information.

Long-horizon agents are still evolving fast, and both architectures and training methods continue to iterate. For engineering teams, building an observable evaluation system early delivers more long-term value than chasing any single model’s scores.