How an Open-Source 2.8T MoE Model Supports Long-Horizon Agents

1. Background: The Shift from Single-Turn Chat to Long-Horizon Agents

Over the past two years, the competitive frontier for large language model applications has moved beyond single-turn Q&A and copy generation. Those one-shot tasks are largely mature. What now separates systems is the ability to handle long-horizon agent workloads: locating and fixing a cross-module bug in a terminal, producing a financial research report according to a predefined framework, or generating trustworthy conclusions through a retrieve-browse-cross-validate-write pipeline that spans dozens of steps.

These tasks impose four requirements that are fundamentally different from conventional chatbot use:

1. A context window long enough to hold complete task trajectories, tool outputs, and historical documents.
2. Stable multi-turn tool calling with reliable formatting and semantically consistent behavior.
3. Controllable reasoning cost: simple tasks should not waste compute, while complex tasks should support deeper reasoning.
4. Deployability: accessible weights, quantization-friendly design, and compatibility with mainstream inference frameworks.

A recently released open-weight, very large-scale MoE model makes these requirements its core design focus. It has 2.8T total parameters, roughly 104B activated per token, supports a 1M-token context window and native image-text input, was post-trained with large-scale joint reinforcement learning on top of a previous version, and is released in MXFP4 quantization format. This article uses its publicly available technical documentation as a reference to break down the architecture and engineering practices behind this class of “agent-native” large models.

2. MoE Architecture: Decoupling Knowledge Capacity from Inference Compute

The core idea behind Mixture of Experts (MoE) is to use a learnable routing network that dispatches each token to only a small number of expert sub-networks. In this model, out of 2.8T total parameters, only about 104B are activated per token, an activation ratio below 4%. This yields two key benefits:

• Capacity side: Total parameter count determines knowledge capacity. The model can store more world knowledge, codebases, and domain experience.
• Compute side: Per-token forward computation depends only on the activated parameters. Inference FLOPs remain at the scale of a model with tens of billions of active parameters.

This decoupling matters especially for agent workloads. Long-horizon task inputs are typically concatenations of system prompts, long histories, and tool outputs, often reaching hundreds of thousands of tokens. With a dense architecture, inference cost for long contexts can quickly become prohibitive. By keeping per-token cost low through MoE and combining it with the sparse attention described below, the model makes “million-token context plus frequent tool calls” affordable in practice.

MoE also introduces deployment pressure: all weights must reside in memory, and expert parallelism, routing load balancing, and cross-device communication all require careful engineering. This is why models of this kind are usually released alongside dedicated integration support for mainstream inference frameworks.

3. Hybrid Sparse Attention: Making Million-Token Context Affordable

Standard softmax attention scales quadratically with sequence length, while the KV cache grows linearly. At the million-token scale, both become hard bottlenecks. The model’s approach is not simply extending the window length, but restructuring the attention stack:

• 24 layers of Gated Sparse Attention (DSA): The previous version’s 24 layers of Gated Latent Attention (MLA) are replaced with DSA in this release. For each query token, a lightweight indexer first scores the full history, then selects approximately 2048 of the most relevant historical tokens to participate in the actual attention computation. Attention cost shifts from scaling with the full history to scaling with a fixed set of 2048 tokens, while still preserving the ability to retrieve critical evidence precisely. This is essential for agents because long-horizon tasks often require retrieving only a few key fragments from a massive history, such as a specific error message, file path, or data record.
• 69 layers of Gated Incremental Linear Attention: This variant compresses the entire history into a fixed-size recurrent state, with constant per-token cost. It excels at maintaining a global summary-style memory.

This hybrid stack of sparse retrieval layers and linear compression layers is essentially a layered trade-off between precise recall and lossy compression. Evidence that needs exact localization goes to the sparse layers, while context that requires holistic understanding goes to the linear layers. Compared with making all layers linear attention, which would sacrifice precise retrieval, or keeping full attention everywhere, which would be cost-prohibitive, the hybrid design is a more pragmatic engineering route for current long-context models.

4. Post-Training: Joint Reinforcement Learning Across Hundreds of Downstream Domains

Pretraining determines what a model can do. Post-training determines how usable it actually is. The central theme of this version’s iteration is large-scale joint reinforcement learning on top of the previous version’s weights:

1. Task coverage: The RL environment spans hundreds of downstream domains, including terminal operations, code repair, web retrieval, document processing, CAD generation, and other agent-style tasks, rather than focusing only on chat preference alignment. The key to agent-oriented RL is verifiable rewards: whether a test passes, whether a task completes, whether a retrieval result is correct. These checkable signals drive the model to learn a “multi-step planning, execution, and correction based on feedback” behavioral pattern.
2. Joint training: Hundreds of domains are trained simultaneously rather than fine-tuned one by one. This avoids the “improve one area, regress another” seesaw effect and preserves general capabilities.
3. Thinking budget tiers: Inference exposes three thinking budget levels: low, high, and max. This gives developers an explicit cost-quality dial: batch tasks can use low to reduce latency, while complex software engineering tasks can use max to improve success rates. In production, routing thinking tiers by task complexity is one of the most direct ways to control overall agent cost.

5. Quantization and Deployment: MXFP4 Weights and Engineering Details

The weights are released in MXFP4 format. MXFP4 is a fine-grained scaled floating-point format: each block of weights shares a scaling factor, and values within the block are encoded with 4 bits. Compared with 16-bit weights, memory usage drops to roughly one quarter, allowing a 2.8T-parameter model to be loaded on fewer accelerators. The fine-grained scaling design is also more stable than naive integer quantization, keeping accuracy loss more controlled for large models.

Several deployment-side engineering details are worth noting:

• Framework integration: The release includes official support for mainstream server-side inference frameworks as well as lightweight inference solutions, covering both high-throughput cluster serving and resource-constrained scenarios.
• Protocol compatibility: The API is compatible with mainstream chat completion and message-based protocol formats, meaning existing applications can migrate with minimal changes.
• History completeness for multi-turn tool calling: The official guidance explicitly requires that multi-turn tool calling must preserve complete historical reasoning and tool-call messages. Many teams have encountered this pitfall: to save tokens, they delete intermediate reasoning blocks or tool returns from the context, causing the model to lose memory in later turns, repeat its reasoning, or even contradict itself. The correct approach is structured pruning, such as compressing tool output content while preserving call signatures and conclusions, rather than bluntly deleting reasoning and call records.

A history-preserving agent main loop looks roughly like this:

history = [system_prompt, user_task]
for step in range(MAX_STEPS):
    resp = llm.chat(messages=history,
                    thinking_budget=route(task),
                    tools=TOOL_SCHEMAS)
    history.append(resp.message)              # reasoning + text, preserved as-is
    if resp.stop_reason == "tool_call":
        for call in resp.tool_calls:
            result = sandbox_run(call)         # execute tool in a sandbox
            history.append(tool_msg(call, result))  # append results in order
    else:
        return resp.final_answer

For deployment, teams can either self-host the open weights on their own GPU clusters or use cloud GPU compute and managed model services to get up and running quickly, focusing engineering effort on agent business logic rather than inference operations.

6. Benchmark Interpretation: Reading Numbers with Caution

The officially published scores include, across different task categories, with the max thinking tier by default and some tasks completed with the help of mainstream coding agent frameworks: 65.7 on a long-horizon terminal operations benchmark, 51.7 on a marathon-style software engineering repair benchmark, 75.5 on a deep software engineering benchmark, 92.1 on a web retrieval benchmark, 92.7 on a CAD generation benchmark with tools, and 64.4 on an office document question-answering benchmark.

These numbers signal two things. First, evaluation focus has shifted from knowledge Q&A to long-horizon task completion rates. Second, agent scores depend heavily on evaluation conditions. Developers should interpret the results with three caveats:

1. Tier and cost: Scores at the max thinking tier usually come with higher token consumption and latency. When reproducing results, re-test under your own budget constraints.
2. Framework dependency: The same model with different agent scaffolding can produce significantly different results. The object being evaluated is effectively a “model plus framework” combination.
3. Private evaluation sets: Public benchmarks carry a risk of training contamination. Before deployment, build a small private evaluation set on your own business tasks and measure success rate, average steps, and token cost.

7. Licensing and Compliance: Open Weights Do Not Mean Unrestricted Use

A particularly important note: this model’s weights are released under a custom license, not a permissive open-source license like MIT or Apache 2.0. Enterprises evaluating the model should review the terms carefully, paying attention to whether commercial use is allowed, whether hosted resale is restricted, what additional conditions apply to redistribution, whether attribution or usage-restriction notices are required, and how ownership and liability for outputs are allocated. Open weights lower the technical barrier, but legal compliance costs do not disappear. This should be front-loaded in procurement reviews.

8. Implementation Recommendations: A Long-Horizon Agent Engineering Checklist

Based on the architectural characteristics above, here is an actionable checklist:

1. Context management: Prune history by preserving reasoning and tool-call skeletons while compressing tool output content. Combine with checkpoints and external memory storage.
2. Thinking budget routing: Route among low, high, and max tiers based on task complexity. Use a small evaluation set to calibrate the success-rate-to-cost curve for each tier.
3. Tool protocol: Write clear parameter constraints and return formats in tool schemas. Execute tools in a sandbox with timeouts, retries, and idempotency handling.
4. Observability: Log reasoning, calls, returns, and final answers across the full pipeline to replay and locate failure steps.
5. Cost governance: Enable prefix caching for long-context requests, monitor per-task token and compute overhead, and set budget circuit breakers.

Conclusion

From the 2.8T-parameter/104B-activated MoE architecture, to the hybrid attention design combining sparse retrieval and linear compression, to joint reinforcement learning across hundreds of domains and the release of MXFP4 open weights, this generation of open models clearly outlines the full technology stack of an “agent-native” large model. For developers, understanding the motivation behind these design choices matters more than memorizing benchmark numbers. It determines how we should select models, control costs, and design context and tool protocols when building long-horizon agent systems. Architectural innovation and engineering practice around long-horizon agents will likely remain a central thread in the evolution of large model technology for the foreseeable future.