2.8T Sparse MoE Architecture: How a New Open-Source LLM Powers Long-Horizon AI Agents

1. Background: What Long-Horizon Agents Demand from the Underlying Model

Over the past year, AI agents have evolved rapidly from “being able to chat” to “being able to get work done.” But once agents are put into production, developers quickly run into three walls:

1. The context wall. Multi-turn tool calls generate large amounts of intermediate state—past reasoning, tool outputs, file snippets, and error logs. When the context window runs out, the agent “forgets” and the task chain breaks.
2. The inference cost wall. Every decision an agent makes requires a model call, often dozens or even hundreds of steps. If every call runs the full parameter set, costs quickly spiral out of control.
3. The long-horizon planning wall. Short tasks can get by with a step-by-step approach, but long tasks require planning ahead and dynamically correcting course—placing extreme demands on the model’s reasoning depth and stability.

A newly open-sourced flagship model is an engineering answer to these three walls. It was post-trained with large-scale joint reinforcement learning on top of the previous-generation architecture, adopts a sparse MoE design with 2.8T total parameters and 104B activated parameters, and natively supports 1M-token context and mixed image-text input. The rest of this article examines it from four angles: architecture, capabilities, evaluation, and deployment.

2. Architecture: How Sparse Attention and Sparse Experts Work Together

2.1 MoE: Decoupling Total Parameters from Activated Parameters

2.8T total parameters sounds staggering, but only 104B activated parameters take part in any single forward pass. That is the core value of sparse MoE—total parameter count buys knowledge capacity, while activated parameter count controls inference cost.

For developers, this means two things:

• The model delivers knowledge breadth approaching that of trillion-parameter-scale models;
• Per-token compute stays in the tens-of-billions range, keeping inference latency and memory footprint more manageable.

2.2 Reworking the Attention Layers: Gated MLA → Gated DSA

Compared with the previous generation, the new version makes one key substitution: 24 layers of Gated MLA are replaced with Gated DSA (sparse attention), used in combination with 69 layers of Delta Attention.

DSA works as follows: each attention computation selects only 2,048 tokens from the history to participate. This is a classic sparse-attention approach—rather than fully connecting every token to the entire history, the model first filters for relevance, then attends over that subset.

The benefit is most visible in high-context scenarios:

• Full attention over 1M tokens means quadratic growth in computational complexity, which is practically unaffordable;
• Sparse selection caps each attention computation at a fixed size, making long context engineering-feasible.

Keep in mind that sparse attention is not a free lunch: the quality of the selection strategy directly determines the outcome. If the wrong historical tokens are chosen, the model loses critical information. Architectures like this therefore typically require targeted training strategies that teach the model “what to pick.”

2.3 Quantization: MXFP4 Weights

The weights are released in the MXFP4 quantization format. This is a block-level floating-point quantization scheme that compresses weights to roughly 4 bits while preserving numerical dynamic range, significantly reducing memory usage and load times. For anyone deploying the model, this directly determines whether it fits on a single GPU and how many GPUs are required.

3. Capability Matrix: From Code Development to Knowledge Work

3.1 Code Development and 3D Creation

Aimed at long-horizon coding agents, software engineering, and 3D design tasks, the official evaluation covers three directions:

• Terminal operations: completing multi-step tasks in a command-line environment;
• Code repair: locating defects and submitting patches;
• CAD generation: producing structured geometry with tool support.

The common thread in these tasks is multi-step execution with tool dependencies. The model cannot simply output a block of text; it must plan action sequences, parse tool outputs, and adjust its next step based on feedback. That is what “long-horizon” really means.

3.2 Agents and Knowledge Work

On the knowledge-work side, the model supports web search, financial research, office document processing, and general tool calling.

One design choice matters a great deal for engineering practice: reasoning offers three thinking-budget tiers—low / high / max—configurable per task.

The essence of a thinking budget is an explicit trade-off between quality and cost:

• Use low for simple tasks to cut latency and overhead;
• Use max for complex planning to buy a more thorough reasoning chain.

This suits production environments better than a one-size-fits-all reasoning configuration, because it hands control back to the caller.

3.3 Context Management for Multi-Turn Tool Calling

The official documentation highlights an important constraint: multi-turn tool calls must retain the complete history of thinking and tool-call messages.

This is a constraint that is easy to overlook and easy to trip over. Many frameworks trim history to save tokens, which causes the model to lose its prior reasoning state, leading to repeated calls and logical contradictions. The correct approach is to fully preserve the chain of thought and tool messages so the model can advance the task over a continuous state.

4. How to Read the Benchmark Numbers

Officially published results (defaulting to the max thinking tier; some evaluations use a general-purpose agent framework):

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

A few things to keep in mind when interpreting these numbers:

1. Inconsistent tiers materially affect comparability. The default max tier means these figures represent “ceiling capability,” not performance at default cost. Results may look different if you run the low tier in production.
2. With tools and without tools are different tasks. BenchCAD is marked “with tools,” meaning it measures the joint capability of “model + tools” rather than the raw model alone.
3. Frameworks affect scores. Different agent frameworks organize prompts, manage history, and handle error retries differently, so final scores will vary.

The right way to use these numbers is as a reference for capability magnitude, not as the sole basis for model selection. Real judgment requires small-scale validation on your own task distribution.

5. Deployment and Integration Practices

5.1 Inference Framework Support

The model ships with MXFP4 weights, and major inference frameworks already support it, exposing OpenAI-compatible / Anthropic-compatible APIs.

The value of compatible APIs is low migration cost—existing agent frameworks, prompt templates, and tool-calling protocols can largely be reused; you simply swap the service endpoint.

A typical call configuration sketch:

# Pseudocode: choose a thinking budget based on task complexity
def build_request(task_type: str) -> dict:
    budget_map = {
        "simple_qa": "low",
        "code_fix": "high",
        "long_horizon_agent": "max",
    }
    return {
        "model": "<model-id>",
        "thinking_budget": budget_map.get(task_type, "high"),
        "messages": history,   # Key: preserve the full history
        "tools": tool_specs,
    }

5.2 License Considerations

The weights are released under a custom license, not permissive terms such as Apache 2.0 or MIT. This means:

• Before commercial use, read the license terms line by line to confirm the scope of use, redistribution conditions, and restrictions;
• Do not assume the compliance expectations of permissive open-source licenses carry over.

For enterprise adoption, this often matters more than technical metrics. Legal and engineering teams should evaluate it together.

6. Practical Recommendations for Developers

First, plan your context budget. 1M tokens is a ceiling, not a target. Estimate the per-task volume of history messages and tool outputs, and design state-compression approaches beyond simple truncation.

Second, tier thinking budgets by task. Use low for high-frequency lightweight tasks and reserve max for scenarios that genuinely require long-horizon planning—the most direct lever for cost control.

Third, fully preserve thinking and tool messages. Do not casually truncate history to save tokens; doing so breaks the agent’s state continuity.

Fourth, validate on your own task set. Official benchmarks reflect general capability; your business distribution is the final judge. Run A/B tests on 20–50 real tasks and track success rate, step count, and cost.

Fifth, handle license compliance early. A custom license means the terms must be confirmed before deployment to avoid rework later.

7. Takeaway

The model’s core idea can be summed up in one sentence: sparse MoE provides knowledge capacity, sparse attention provides context length, and adjustable thinking budgets provide cost control. Only the combination of all three moves long-horizon agents from demos to deployable engineering.

For developers, the number worth watching is not 2.8T itself but the design philosophy behind it: decoupling total parameters from activated parameters, decoupling attention compute from context length, and decoupling reasoning quality from reasoning cost. Understanding these three points delivers more long-term value than memorizing any single benchmark score.