A 2.8T Sparse MoE Model Goes Open Source: Architecture and Long-Horizon Agent Capabilities

1. Background and Model Positioning

A large-scale sparse Mixture-of-Experts (MoE) model with 2.8T total parameters has officially released its weights. Building on its predecessor foundation model, it achieves a capability leap through large-scale joint reinforcement learning post-training. The model has 104B activated parameters, supports a 1M-token context window, and accepts native image-text multimodal input. For Agent applications that require long-horizon reasoning and multi-step tool calling, this “large total parameters, small activation” architecture offers a viable path that balances capacity with inference cost.

From an engineering perspective, this release sends several notable signals: first, the weights of an ultra-large MoE are opened in a low-precision quantized format, lowering the deployment barrier; second, the model is explicitly aligned for “long-horizon Agent” scenarios rather than optimizing purely for single-turn question answering; third, the inference side introduces configurable thinking budget tiers, letting developers make explicit trade-offs between latency and quality. The following sections cover architecture changes, training methods, benchmark performance, and deployment considerations.

2. Architecture: Gated Sparse Attention Mixed with Linear Attention

2.1 From Dense Gating to Sparse Selection in Attention

Compared with the previous generation, the new version makes a key replacement at the attention layer: the original 24 layers of Gated Multi-head Latent Attention (Gated MLA) are replaced with Gated Sparse Attention (Gated DSA), combined with 69 layers of a linear attention variant. The core mechanism of DSA is that each attention step selects only a fixed number of tokens from the historical sequence to participate in computation; the officially stated selection size is 2,048 historical tokens.

The motivation is straightforward: when context scales to the million-token level, the time and memory cost of standard full attention grows quadratically with sequence length, and inference cost quickly becomes unmanageable. Sparse attention keeps per-step computation at a roughly constant level through “selective lookback,” while a gating mechanism retains the ability to dynamically filter key historical information. The linear attention layers further handle information aggregation over long sequences. The two are stacked together to form a division of labor: “close reading locally, coarse scanning globally.”

2.2 Engineering Implications of Sparse Attention

It is worth emphasizing that fixed-budget sparse selection does not mean information loss is uncontrollable. The actual effect depends on whether the selection strategy can reliably hit key segments across multi-turn conversations and tool-call chains. In Agent scenarios, historical messages often mix tool return results, intermediate reasoning, and user instructions. If sparse recall favors recent tokens, it may miss task constraints set early on. Therefore, for long-horizon tasks, developers should place key constraints early or restate them periodically to work with the recall characteristics of sparse attention.

3. Training: Joint Reinforcement Learning and Low-Precision Quantization

3.1 Joint Reinforcement Learning Across Hundreds of Domains

The post-training stage uses a joint reinforcement learning scheme covering hundreds of downstream domains. Unlike the traditional serial route of “general alignment first, then domain-specific fine-tuning,” joint training exposes the model to heterogeneous tasks—code, terminal operations, document processing, retrieval QA, and more—within the same optimization process. This helps form general cross-domain tool-use habits and reduces the see-saw effect where improving one domain degrades another.

For Agent capabilities, the key to reinforcement learning lies in reward signal design: the final outcome of multi-step tasks is often sparse and delayed, requiring process rewards or verifiable result checks (such as passing test cases or correctly generating files) to stabilize training. Joint multi-domain training effectively broadens the diversity of reward signals, giving the model stronger transfer potential on unseen tasks.

3.2 MXFP4 Weight Quantization

Weights are released quantized in MXFP4 format. MXFP4 is a microscaling floating-point format that compresses numerical representation by sharing exponents across blocks, significantly reducing storage and bandwidth requirements while maintaining a relatively high dynamic range. For a model with 2.8T total parameters, quantization is a prerequisite for making the weights “downloadable and deployable.” Developers should note that low-precision weights require matching quantized operator support at inference time; mixing precisions may introduce numerical deviation.

4. Inference Control: Three Thinking Budget Tiers

The model offers three thinking budget configurations—low, high, and max—allowing inference investment to be set dynamically based on task complexity. The practical value of this mechanism:

• low: Suited to latency-sensitive scenarios such as structured extraction, simple classification, and short QA, reducing unnecessary intermediate reasoning tokens;
• high: Suited to routine code completion, document rewriting, and single-turn tool calls;
• max: Suited to long-horizon software engineering tasks, multi-step terminal operations, and complex CAD generation that require thorough planning. Official benchmark scores are measured at this tier by default.

From a cost perspective, the thinking budget directly determines output token volume and is the primary variable in inference cost. It is recommended to route tasks by type in production rather than using the highest tier globally.

5. Capability Coverage and Benchmark Performance

Official evaluations cover three core scenario categories and report the corresponding scores (max thinking tier by default; some evaluations use external Agent frameworks):

Evaluation Dimension	Benchmark	Score
Terminal operations	Terminal-Bench 4.0	65.7
Long-horizon software engineering	SWE Marathon v1.1	51.7
Code repair	DeepSWE v1.1	75.5
Web retrieval	BrowseComp	92.1
CAD generation (with tools)	BenchCAD	92.7
Office document QA	OfficeQA Pro v2	64.4

Several patterns emerge from the data distribution: retrieval tasks and tool-assisted CAD tasks perform notably well, indicating the model is well trained at “calling external tools to supplement its own knowledge gaps.” Long-horizon software engineering benchmarks score relatively conservatively, reflecting that cross-file, multi-iteration engineering tasks remain a bottleneck for current Agent capabilities. Terminal operations and code repair are at a moderately high level, suitable as productivity tools for daily development assistance, though high-risk changes still require human review.

Note that the above scores are official figures. Differences in test environments, prompt templates, and Agent frameworks all affect reproduction results. Readers should conduct secondary evaluation using their own business data.

6. Deployment and Integration Considerations

6.1 Inference Frameworks and API Compatibility

The weights support loading by mainstream open-source inference frameworks, including vLLM for high-throughput serving, SGLang for flexible scheduling, and low-latency inference engines. OpenAI- and Anthropic-style compatible API interfaces are also provided, making it easier for existing applications to integrate with minimal modification. A typical call looks like this (illustrative):

# Sending a request with tool definitions via a compatible interface (illustrative)
client = Client(base_url="http://localhost:8000", api_key="EMPTY")
resp = client.chat.completions.create(
    model="large-moe-agent",
    messages=[
        {"role": "system", "content": "You are an engineering assistant that can call tools"},
        {"role": "user", "content": "Locate the cause of the build failure and provide a fix patch"}
    ],
    tools=[{"type": "function", "function": {...}}],
    extra_body={"thinking_budget": "max"}  # Three-tier budget (illustrative)
)

6.2 State Management for Multi-Turn Tool Calls

The official guidance specifically notes: when making multi-turn tool calls, the complete historical thinking content and tool-call messages must be preserved. The reason is that sparse attention relies on historical tokens for selective recall. If intermediate reasoning or tool returns are discarded when assembling context, the model loses continuity of task state, manifesting as repeatedly calling the same tool or forgetting confirmed constraints. In engineering implementation, it is recommended to:

1. Maintain the complete message trajectory rather than lossy compression that “keeps only the final answer”;
2. For very long trajectories, apply segmented summarization but preserve tool-call IDs and result mappings;
3. When context approaches the window limit, prioritize trimming low-information chitchat rather than the reasoning chain.

6.3 License Considerations

The weights use the model series’ dedicated license, not a permissive one such as Apache 2.0 or MIT. Before commercial use, carefully review the license terms regarding distribution, attribution, and scope of use to avoid compliance risk. This is especially important for enterprise deployment; legal and architecture teams should evaluate jointly.

7. Practical Recommendations and Outlook

Taken together, the value of this class of ultra-large sparse MoE models is not a single-point capability refresh, but rather combining “long context + tool calling + adjustable reasoning budget” into an engineerable Agent foundation. For deployment, consider the following path:

• Evaluate before integrating: Run small-traffic comparisons at the low and high tiers using your own business samples, confirm the benefit, then open up the max tier;
• Route tasks by tier: Dispatch by complexity to different budget tiers to control per-task cost;
• Strengthen trajectory management: Treat context and tool-call logs as first-class citizens when designing storage and trimming strategies;
• Watch quantization compatibility: Confirm the inference framework’s operator support version for low-precision weights to avoid precision regression.

As hybrid sparse-attention and linear-attention architectures mature, million-token context is moving from a “demo feature” to a “default configuration.” For developers, the real barrier has shifted from “whether the model works” to “whether the context and toolchain can be managed well.” That may be the most noteworthy long-term proposition of this open-source release.