Open-Source Sparse MoE at 2.8T Parameters: How It Supports Long-Horizon Agents

Long-horizon agents are becoming the primary battleground for large model deployment. Unlike single-turn question answering, agent tasks often span dozens or even hundreds of steps, continuously reading and writing tool results, maintaining intermediate state, and repeatedly locating key information across long histories. This places demands on context length, attention computation efficiency, and inference budget control that far exceed those of chat scenarios. A sparse MoE model with 2.8T total parameters has recently been released with open weights, and its design focus clearly targets heavy workloads such as long-horizon coding agents, software engineering, and 3D creation. This article systematically reviews the publicly available technical information about the model across four dimensions: architecture, training, evaluation, and deployment.

1. Overall Specifications: Sparse MoE and Million-Token Context

The model builds on its predecessor through large-scale joint reinforcement learning post-training. Core specifications are as follows:

• 2.8T total parameters, 104B activated parameters: It uses a Mixture of Experts (MoE) architecture, activating only about 104B parameters per forward pass. This maintains large knowledge storage capacity while controlling per-step inference cost.
• 1M-token context: Designed for the long-history requirements of long-horizon agents, supporting a complete tool-call trajectory and document content within a single session.
• Native image-text input: Multimodal capability is built in, directly handling mixed image and text input for scenarios such as interface understanding and diagram parsing.

From an engineering perspective, the combination of “large total parameter count + small activated parameter count” is a mainstream trade-off in current sparse architectures: total parameters determine the model’s knowledge ceiling and expressive capacity, while activated parameters determine the compute cost per token. The 2.8T/104B ratio implies an activation rate of roughly 3.7%, a relatively aggressive sparse configuration that places higher demands on routing network stability and load balancing.

2. Attention Mechanism: Gated DSA and Incremental Attention

The core bottleneck for long context lies in attention computation. The most notable structural change from the predecessor is the replacement of 24 layers of Gated MLA with Gated DSA (sparse attention), used in combination with 69 layers of incremental attention. Key design points include:

1. Dynamic historical token selection: During each attention computation, DSA selects 2,048 tokens from the historical sequence to participate, rather than applying dense attention over the full history. This means per-step attention computation no longer grows linearly with context length but is constrained to a fixed budget.
2. Gating mechanism: The gated structure regulates the output strength of the sparse attention branch, helping the model dynamically balance between “retrieved local information” and “global state,” mitigating the information fragmentation caused by sparse selection.
3. Layer-wise hybrid: Not all layers use sparse attention. The hybrid configuration of 24 layers on the sparse path and 69 layers on the incremental attention path balances long-range retrieval efficiency with continuity of local representations.

The following pseudocode illustrates this layer-wise selection logic:

# Illustrative pseudocode, not a real interface
for layer in range(num_layers):
    if layer in sparse_layers:          # 24 layers
        history = select_topk_tokens(
            query, memory, k=2048)      # select 2,048 historical tokens each time
        out = gated_sparse_attention(query, history, gate)
    else:                               # 69 layers
        out = delta_attention(query, state)
    state = update_state(state, out)

It is worth emphasizing that the benefits of sparse attention are not unconditional: if the selection strategy has insufficient hit rate, the result manifests as missed key information in long documents. Such architectures therefore typically need to teach the model “how to select” during training, rather than relying only on static truncation at inference time.

3. Training Strategy: Cross-Domain Joint Reinforcement Learning and Low-Bit Quantization

Two aspects of training deserve attention:

• Joint reinforcement learning covering hundreds of downstream domains: Unlike the traditional path of general pre-training followed by single-task fine-tuning, this model uses joint reinforcement learning in the post-training stage across terminal operations, code repair, retrieval QA, office document processing, and other domains. The difficulty of joint training lies in the enormous variance in reward scales across tasks, requiring reward normalization and sampling ratio control; otherwise, one domain may dominate policy updates.
• MXFP4 weight quantization: Weights are released in 4-bit floating-point format, significantly reducing storage and VRAM footprint. For 2.8T total parameters, low-bit quantization is one of the prerequisites for making deployment practically feasible.

4. Capability Areas: Three Typical Task Categories

Public information divides capabilities into three blocks, corresponding to different agent forms:

Code development and 3D creation: Targets long-horizon coding agents, software engineering, and 3D design tasks. Evaluation covers terminal operations, code repair, and CAD generation, emphasizing the ability to complete multi-step tasks with tools. The common trait of these tasks is a long “action-feedback” loop, where the model must retain memory of the original goal across multiple tool returns.

Agents and knowledge work: Supports web retrieval, financial research, office document processing, and tool calling. Such scenarios place higher demands on “selective forgetting” of context—irrelevant web content quickly occupies the window, making the value of sparse attention more direct here.

Three inference budget tiers: The inference side offers low, high, and max thinking budgets, configurable by task complexity. Simple tool calls use the low tier to reduce latency, while complex multi-step planning uses the high tier to gain accuracy. This tiered control is very practical for managing online serving costs.

5. Public Benchmark Data

Officially published evaluation results are as follows (using the max thinking tier by default; some evaluations were completed with external agent frameworks):

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

Three points should be noted when interpreting this data. First, terminal operation and code repair metrics reflect combined “multi-step execution + tool collaboration” capability, not single-turn code generation quality. Second, the with-tools CAD evaluation is notably higher than pure-text tasks, indicating that the toolchain significantly affects how much of the model’s capability is unlocked. Third, different evaluations use different agent frameworks, so cross-comparisons should be made cautiously; simple rankings against other models’ public numbers are not advisable.

6. Deployment and Integration Practices

Alongside the open weights, the official documentation provides a fairly complete deployment path:

• Inference engine support: Compatible with mainstream high-throughput inference frameworks and dedicated inference engines, facilitating selection based on existing infrastructure.
• API compatibility: Provides OpenAI- and Anthropic-style compatible interfaces, lowering migration costs for existing applications.
• Key constraint for multi-turn tool calling: In multi-turn tool-calling scenarios, the complete historical thinking content and tool-call messages must be preserved. This point is easily overlooked in engineering implementations—if intermediate thinking segments or tool returns are trimmed when assembling requests, the model loses its state basis, manifesting as repeatedly calling the same tool or terminating tasks prematurely.
• License: The weights use a model-specific license, not a permissive agreement such as Apache 2.0 or MIT. Enterprises should have legal counsel confirm the license terms before large-scale commercial use, particularly restrictions on distribution, hosted services, and derivative models.

An easily missed implementation detail is the construction order of the message sequence. Illustration:

# Illustrative pseudocode: preserve the full trajectory
messages = []
messages.append({"role": "user", "content": task})
while not done:
    resp = call_model(messages, thinking_budget="max")
    messages.append(resp)                  # includes thinking content, must not be trimmed
    if resp.tool_call:
        result = run_tool(resp.tool_call)
        messages.append({"role": "tool", "content": result})

7. Applicability Boundaries and Selection Advice

Overall, the model’s positioning is quite clear: it is not a general-purpose model optimized for lightweight chat, but one designed for long-horizon, multi-step tasks with heavy tool collaboration. Three dimensions can guide selection:

1. Task step count: If a single task averages more than ten tool interactions, the benefits of long context and sparse attention will truly materialize; in short-task scenarios, large activated parameters instead bring unnecessary cost.
2. Infrastructure capacity: Even after 4-bit quantization, 2.8T total parameters still impose high requirements on VRAM and parallelism strategy, so inference cluster deployment density should be assessed in advance.
3. Compliance and licensing: Model-specific licenses differ markedly from permissive open-source agreements, so terms must be verified before commercial use.

In summary, the significance of this model goes beyond parameter scale: it puts the combination of “sparse attention + tiered inference budgets + long-horizon agent training” into developers’ hands as open weights. For teams building long-horizon agent systems, it is a base model option worth serious evaluation; for developers tracking the evolution of sparse architectures, its layer-wise design of Gated DSA and incremental attention also offers engineering ideas worth borrowing.