Open-Weight MoE LLMs: A Technical Analysis of Long-Horizon Agent Capabilities

A sparse Mixture-of-Experts (MoE) model with 2.8T total parameters, of which only 104B are activated per forward pass, has officially released its weights. The model was purpose-built around context length, native multimodal input, and long-horizon agent task execution, and ships with a custom license plus support for multiple inference backends. This article analyzes the release across five areas — architectural redesign, training strategy, inference budget control, agent engineering practice, and deployment selection — to help developers understand the technical essentials of long-horizon agent models and how to adopt them.

1. What a 2.8T Sparse MoE Architecture Really Means

The model follows a typical sparse MoE design. The 2.8T total parameters define the upper bound of the model’s knowledge capacity, while the 104B parameters activated per forward pass determine the compute cost per token. This “large capacity, small activation” design lets the model cover a much broader range of knowledge and skills while keeping inference costs under control.

For developers, there are two key engineering implications:

1. Memory and bandwidth are the primary bottlenecks. The weight footprint is enormous. Even with low-bit quantization, single-machine deployment still requires multi-GPU sharding and expert parallelism. In practice, uneven expert distribution can skew the load, which means you will depend on the inference framework’s expert-routing load-balancing strategy.
2. Activated parameters set throughput expectations. With 104B activated parameters, the per-token compute cost is close to that of a ~100B dense model, but thanks to MoE sparsity, the output per unit of compute is typically better than that of a dense model with the same total capacity. When estimating throughput, benchmark against the activated parameter count, not the total.

The model supports a 1M-token context and native image-text input. The engineering value of long context: an agent can keep the full tool-call history, code repository snippets, and document context within a single session, reducing the information loss caused by repeated summarization. The flip side is that attention computation grows quadratically with sequence length — which is exactly what motivates the sparse attention redesign below.

2. Attention Redesign: From Gated MLA to Gated Sparse Attention

Compared with its predecessor, the new version makes two key substitutions in the attention structure: the 24 layers of Gated Multi-head Latent Attention (Gated MLA) are replaced with gated sparse attention (Gated DSA), combined with 69 layers of a linear attention variant.

The core motivation is keeping attention computation tractable under ultra-long context. With full attention, both the attention matrix computation and memory footprint grow with N² as sequence length N increases — at the million-token scale, full attention is simply not feasible. Sparse attention takes a different approach: instead of attending to every previous token, each token selects a fixed number of historical tokens through a routing mechanism.

The specific setting: Gated DSA selects 2,048 historical tokens for each attention computation. This number is a precision-versus-efficiency tradeoff — 2,048 historical tokens are enough to cover recent context plus key remote information, while keeping per-token attention overhead constant, which is what makes million-token inference practical in the first place.

Architecturally, stacking sparse attention layers together with linear attention layers is a common pattern in current long-context models: linear attention provides global information aggregation at near-constant cost, while sparse attention adds selective focus for precision. When evaluating models of this type, pay close attention to two things:

• Whether the sparse selection strategy is interpretable and maintains stable recall on long-range dependency tasks, such as cross-file code fixes;
• Whether the ratio of the two layer types affects latency on short-context tasks.

3. Training Strategy: Joint Reinforcement Learning and Quantization

The model was post-trained on top of the previous base model with large-scale joint reinforcement learning (RL) covering hundreds of downstream domains, and is released in the MXFP4 weight quantization format.

Joint RL means bringing multiple task types into a single unified optimization objective: reward signals from terminal operations, code repair, web retrieval, document processing, CAD generation, and more are trained together. As a result, the model is more inclined to maintain plan consistency across multi-step, long-horizon tasks instead of overfitting to single-step performance. For agent scenarios, this directly maps to the ability to stay on track across many rounds of tool calls.

MXFP4 quantization is a deployment-driven engineering decision. FP4-level weight quantization significantly reduces memory footprint and weight-loading bandwidth, lowering the barrier to deploying such a large model in multi-GPU environments. One caveat: low-bit quantization can introduce precision loss on numerically sensitive tasks, such as precise code generation or numerical reasoning. Run before/after consistency evaluations for your specific task type, and apply mixed precision to critical layers if necessary.

4. Inference Budgets: Three Thinking Levels in Practice

The model exposes three thinking budget levels — low, high, and max — letting you set inference cost dynamically based on task complexity. This is a highly practical engineering interface. Suggested usage:

Level	Use cases	Engineering guidance
low	Simple classification, format conversion, short Q&A	Latency-sensitive; default choice for batch calls
high	Medium-complexity code completion, single-turn tool calls	Balances quality and cost; sensible general default
max	Long-horizon agent planning, multi-step code repair, complex retrieval and reasoning	Enable only for hard samples; pair with timeout and retry policies

Official benchmarks default to the max level, which means the published scores represent the model’s performance ceiling, not its default deployment cost. When doing capacity planning, estimate your average thinking budget by weighting the actual task distribution of your business — do not build cost models on max-level token consumption.

5. Agent Engineering: The Key Constraint in Multi-Turn Tool Calling

In agent scenarios, one easily overlooked but high-impact engineering detail is message history completeness. The model explicitly requires that multi-turn tool calling preserve the full thinking messages and tool-call messages in the history.

The reason: models of this type are trained to treat the thinking process and tool-call results as part of a continuous decision chain. If thinking segments are dropped when assembling the history, or only tool returns are kept while call parameters are omitted, the model loses its sense of prior decision intent. The typical symptoms are repeated calls, parameter drift, or premature termination.

Here is a recommended way to organize messages (illustrative pseudocode):

messages = []
messages.append({"role": "user", "content": task})
while not finished:
    resp = model.chat(messages, thinking_budget="max")
    messages.append(resp.full_message)   # Keep thinking + tool call requests
    for call in resp.tool_calls:
        result = dispatch(call)
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,      # Must match the call request exactly
            "content": result
        })

Three points to remember: first, thinking content must be sent back along with the message and never truncated; second, tool results must be paired with their requests via the call ID to avoid misordering; third, when the context approaches its limit, compress the raw content returned by tools (logs, web page text, etc.) first rather than deleting thinking and call records.

The model also provides an interface compatible with mainstream API styles, making it easy to plug into existing agent frameworks. For multi-turn scenarios, combine it with idempotent tool wrappers, per-step timeouts, and failure retries so that a single tool exception does not derail an entire long-horizon task.

6. Reading the Benchmarks and Choosing a Deployment

The officially published benchmarks cover terminal operations, code repair, long-horizon software engineering, web retrieval, CAD generation, and office document Q&A, all run at the default max level, with some evaluations completed via agent frameworks. When interpreting this data:

1. Agent frameworks matter a lot. Terminal-operation and code-repair scores depend heavily on orchestration quality; differences between frameworks can shift results by several percentage points, so confirm identical evaluation conditions before making cross-model comparisons.
2. Long-horizon metrics are more informative than single-step metrics. Multi-step task success rates better reflect how stable the model will be in a real agent pipeline.
3. CAD and office-document scores reflect tool-integration ability, not pure generation ability, so landing these capabilities requires the corresponding toolchain.

On deployment: the weights are released in MXFP4 format, supported by multiple mainstream inference backends, with an API-compatible interface. When choosing a stack, focus on how well the inference framework supports sparse attention and expert parallelism, its memory footprint curve under long context, and the measured impact of quantization on your business metrics.

7. License and Compliance Notes

One important reminder: the model weights are released under a custom license, not a permissive open-source license. Enterprises should review the license terms in full before commercial use, paying particular attention to the scope of commercial use, redistribution conditions, requirements for derivative models, and attribution obligations. Teams planning to embed the weights into external products should have legal and engineering complete the compliance assessment together to avoid downstream risk from misreading the terms.

Conclusion

Releasing the weights of such a very large sparse MoE model effectively moves long-horizon agents from the demo stage to the engineering stage: million-token context reduces state-management complexity, the three thinking budgets provide a cost-control knob, and the requirement for complete message history in multi-turn tool calling reminds developers that agent reliability depends not only on the model itself, but also on the engineering quality of message orchestration and tool governance. A suggested adoption path: establish a baseline with the high budget on medium-complexity tasks, introduce the max level only for hard samples, and build out message completeness, idempotent tools, and timeout/retry handling as foundational infrastructure first.