A 2.8T-Parameter Sparse MoE Model Opens Weights: Architecture and Long-Horizon Agent Capabilities Explained

1. Why This Release Matters

A sparse mixture-of-experts (MoE) model with a total parameter count of 2.8T has officially opened its weights. Compared with the previous base model, this version was post-trained through joint reinforcement learning across hundreds of downstream domains. It activates roughly 104B parameters per token, supports a context window of about 1 million tokens, and natively accepts mixed image-text input.

For teams working on long-horizon agents, the significance is not simply that another model is available. It is that a very large sparse architecture, long-context processing, and tool-calling capability are now accessible to teams that need self-hosted deployment.

One important caveat: open weights do not mean permissive licensing. The model is released under a custom license, not a widely used open-source license. Teams must review the terms carefully before commercial use, especially around redistribution, derivative model naming, and commercial boundaries.

2. Architecture: Sparse Attention Combined with Linear Attention

2.1 The MoE Baseline

A 2.8T total parameter count with 104B activated parameters means each token is routed to only a small subset of experts. The direct benefit is that memory usage scales with total parameters, while compute cost scales with activated parameters. For self-hosted deployments, the main budget item is memory capacity rather than full dense matrix multiplication across the entire model.

2.2 Attention Layer Changes

The new version replaces 24 gated multi-head latent attention layers with gated sparse attention layers, combined with 69 linear attention layers. The core behavior of sparse attention is that each attention computation selects only 2048 historical tokens instead of attending to the entire context.

This change targets the cost bottleneck of long-context processing. Full attention over 1 million tokens scales quadratically with sequence length. The combination of sparse selection and linear attention reduces complexity to near-linear in most layers, while preserving precise retrieval in a small number of layers. A useful way to think about it is that most layers perform coarse-grained reading, while a few layers perform fine-grained reading.

2.3 Weight Quantization

Training and release use MXFP4 weight quantization. The 4-bit floating-point format maintains numerical dynamic range while significantly reducing memory footprint, which is a key reason a 2.8T-parameter model can be deployed at a practical scale. In practice, inference quality under quantized weights depends heavily on the runtime implementation. Teams should run task-level regression on the target inference engine rather than relying only on perplexity.

3. Inference Budgets: Three Thinking Levels

This release provides three thinking budgets: low, high, and max. These are not simply response-length switches. They constrain the overall reasoning chain length and the number of tool-calling rounds.

• low: suitable for deterministic tasks such as classification, extraction, and format conversion, with a focus on latency.
• high: suitable for medium-complexity tasks such as multi-step code edits and document processing.
• max: the default setting in official evaluations, suitable for long-horizon tasks such as terminal operation, complex retrieval, and CAD generation.

From an engineering perspective, the recommended approach is dynamic routing by task type rather than enabling max globally. One possible policy:

BUDGET_POLICY = {
    "intent_classify": "low",
    "doc_summarize": "high",
    "repo_patch": "max",
    "web_research": "max",
}
def pick_budget(task_type: str) -> str:
    return BUDGET_POLICY.get(task_type, "high")

4. Capabilities: From Coding Agents to Knowledge Work

The official evaluation covers several representative areas:

• Terminal operations and code repair: Terminal-Bench 4.0 scored 65.7, SWE Marathon v1.1 scored 51.7, and DeepSWE v1.1 scored 75.5. These tasks test the full loop of reading a repository, locating issues, editing code, running tests, and revising.
• Web retrieval: BrowseComp scored 92.1, indicating strong performance in multi-hop retrieval and information aggregation.
• CAD generation: BenchCAD with tools scored 92.7, reflecting multi-step spatial tasks combined with tool use.
• Office document QA: OfficeQA Pro v2 scored 64.4, leaning toward structured document understanding and extraction.
• 3D and design tasks: the release explicitly lists 3D creation as one target scenario.

These results should be read in context. The scores use the max thinking budget by default, and some evaluations were conducted with the help of agent frameworks such as coding tools. That means the numbers reflect a combination of model, framework, and budget rather than raw single-turn model performance. For reproducible comparisons, keep the framework version, tool set, and thinking budget fixed.

5. Deployment and API: Key Details for Multi-Round Tool Calling

The release provides MXFP4 weights and supports mainstream inference engines, along with APIs compatible with OpenAI and Anthropic styles. This lowers integration cost, but there is a common pitfall in long-horizon agent scenarios:

Multi-round tool calling must preserve the complete history of thinking messages and tool-call messages. If an intermediate layer truncates thinking blocks or tool outputs to save context, subsequent decisions can degrade noticeably. The correct approach is to pass thinking blocks, tool-call requests, and tool results as an inseparable sequence.

{
  "messages": [
    {"role": "user", "content": "Fix the build failure"},
    {"role": "assistant", "thinking": "...", "tool_call": {"name": "run_tests"}},
    {"role": "tool", "name": "run_tests", "content": "3 failed"},
    {"role": "assistant", "thinking": "...", "content": "Identified a dependency version conflict"}
  ]
}

Although the 1 million token context is usable, cost increases with sequence length. A practical strategy is to combine summarization, retrieval-based memory, and staged execution. Treat the long context as a fallback capability rather than the default working mode.

6. Evaluation and Adoption Guidance

Teams building agent systems that require multi-step planning, tool orchestration, and code operations should put this model on the evaluation list, particularly for private deployment scenarios that need near-closed-source inference quality. Before adoption, three steps are recommended:

1. License review: confirm the exact requirements of the custom license for commercial use, redistribution, and derivative works.
2. Quantization regression: run the business task suite on the target inference engine to validate real-world MXFP4 behavior.
3. Budget and context strategy: select thinking budgets dynamically by task type to avoid runaway costs from using max everywhere.

The core value of this release is that it packages very large sparse architecture, sparse attention, and long-horizon agent capabilities into a self-hostable form. It does not solve engineering problems by itself, but it does expand the available options for teams building agentic systems.