Open-Source 2.8T MoE Model Supports 1M-Token Context and Long-Horizon Agents

1. Model Positioning and Core Specifications

A large-scale Mixture-of-Experts (MoE) model designed for long-horizon agent tasks has released its weights. Built on a previous-generation foundation, the model underwent large-scale joint reinforcement learning post-training. Key specifications:

• 2.8T total parameters, 104B activated parameters: A high-sparsity MoE architecture that significantly expands model capacity while keeping inference costs manageable.
• 1M-token context length: Designed for long-document understanding, repository-level code operations, and multi-turn tool calling.
• Native image-text input: Supports mixed text and image input, providing foundational capability for UI understanding, chart analysis, and CAD-style tasks.
• Open MXFP4 weight quantization: Reduces VRAM footprint and deployment barriers, compatible with mainstream inference frameworks.

In terms of positioning, this generation does not simply pursue parameter scale. The focus is on stability in long-horizon tasks — whether the model can maintain goal consistency, avoid context loss, and support error rollback across dozens or even hundreds of steps of tool calls, code edits, and retrieval verification. This is precisely the key bottleneck preventing agents from moving from demos to production.

2. Architecture Evolution: From Full Attention to Sparse Attention

2.1 Hybrid Stacking of Gated MLA and Gated DSA

Compared to the previous generation, the new model makes a structural replacement in its attention layers: 24 layers of Gated Multi-head Latent Attention (Gated MLA) are replaced with Gated Dynamic Sparse Attention (Gated DSA), combined with 69 layers of a linear attention variant to form a hybrid attention stack.

The motivation behind this design can be understood from two dimensions:

First, the computational economics of long contexts. Standard full attention has O(N²) complexity for sequence length N. When context extends to millions of tokens, attention computation and KV cache usage become primary bottlenecks. Sparse attention reduces complexity to approximately linear by limiting the visible history range for each query position.

Second, selective retention of long-range dependencies. Pure sliding windows or fixed-pattern sparsity would lose critical distant information. The DSA mechanism used in this model dynamically selects 2,048 historical tokens for each attention computation, with selection based on content relevance rather than fixed positional patterns. This means the model can “retrieve on demand” key constraints written early in a million-token context — such as interface specifications set at task start, requirements corrected mid-task by the user, or configuration file contents read dozens of steps earlier.

2.2 Engineering Implications of the Hybrid Attention Stack

Stacking sparse attention and linear attention in layers is essentially a layered trade-off between “precise retrieval” and “constant overhead”: some layers handle high-fidelity content addressing, while others handle low-cost global state propagation. For developers, the direct benefit is a flatter VRAM curve under long contexts and reduced degradation — such as “forgetting the initial goal mid-task” — in ultra-long tasks.

It should be noted that the dynamic selection mechanism in sparse attention places higher demands on inference framework kernel implementations. Deployment should prioritize inference backends that have already adapted to this attention pattern; otherwise, performance may degrade to inefficient implementations.

3. Inference Budget: Three Thinking Modes for Task Adaptation

The model offers three thinking budget tiers: low, high, and max, allowing inference overhead to be dynamically set based on task complexity. The value of this mechanism lies in enabling a single set of weights to cover widely varying workloads:

Tier	Use Case	Characteristics
low	Simple Q&A, format conversion, single-step tool calls	Latency-first, low token consumption
high	Multi-step reasoning, medium-scale code edits	Balanced quality and cost
max	Long-horizon agents, complex bug localization, multi-constraint design tasks	Quality-first, allows long chain-of-thought

Official benchmark scores default to the max tier, which serves as a reminder: benchmark numbers reflect capability ceilings, not performance under default configurations. In production, tier routing by task type is recommended — for example, pinning document summarization requests to low, while elevating code repair and terminal operation requests to high or max, to achieve a balance between overall cost and success rate.

4. Capability Coverage and Official Benchmark Interpretation

4.1 Code Development and 3D Creation

The model is specifically optimized for long-horizon code agents, software engineering, and 3D design tasks. Official evaluations cover three scenario categories — terminal operations, code repair, and CAD generation — and support multi-step tasks combined with external tools. Relevant scores:

• Terminal-Bench 4.0: 65.7 — Measures the ability to complete operational tasks in real terminal environments, involving command composition, environment exploration, and error recovery.
• SWE Marathon v1.1: 51.7 — Targets long-horizon software engineering tasks, emphasizing goal consistency across multi-file, multi-turn edits.
• DeepSWE v1.1: 75.5 — Comprehensive software engineering capability evaluation.
• BenchCAD (with tools): 92.7 — CAD generation tasks combined with toolchains, demonstrating coordination between image-text input and tool calling.

The SWE Marathon-style evaluation deserves particular attention: it tests not single-point code generation, but whether the model can maintain an overall grasp of repository structure, test feedback, and requirement constraints across dozens of operations. This echoes the sparse attention design mentioned earlier — performance in long-horizon tasks depends heavily on context management capability.

4.2 Agents and Knowledge Work

In the agent and knowledge work domain, the model supports web retrieval, financial research, office document processing, and tool calling. Corresponding benchmark scores:

• BrowseComp: 92.1 — Web retrieval and information synthesis capability.
• OfficeQA Pro v2: 64.4 — Office document understanding and question answering.

It should be noted objectively that the official documentation mentions some evaluations used external agent frameworks (such as agentic coding tool frameworks) to host the tool-calling loop. This means scores include a “model + framework” combined effect. Readers comparing results across models should pay attention to differences in evaluation configurations and avoid attributing framework capabilities to the model itself.

5. Training Approach: Joint Reinforcement Learning and Quantization

The model’s post-training uses joint reinforcement learning covering hundreds of downstream domains. Compared to single-domain fine-tuning, the core challenge of joint RL is the enormous variance in reward signal scales across tasks — binary success/failure for terminal operations, test pass rates for code repair, semantic match scores for document QA — requiring a unified training strategy to prevent inter-task interference. The significance of joint training across hundreds of domains lies in improving the model’s generalization and transfer capability, enabling it to maintain basic planning and error-correction behavior when encountering unseen tool combinations.

At the weight level, MXFP4 quantization is used. This format represents weights in 4-bit floating point, significantly compressing model size while retaining floating-point dynamic range characteristics, which is more conducive to maintaining numerical stability than fixed-point quantization. For a model with 2.8T total parameters, quantization is one of the prerequisites for enabling inference serving on reasonably sized clusters.

6. Deployment Practicalities

6.1 Inference Frameworks and API Compatibility

The open weights support mainstream open-source inference frameworks (such as vLLM, SGLang, and high-performance inference backends like TokenSpeed), and provide OpenAI / Anthropic compatible APIs. This means existing applications built on these two interface specifications can migrate at relatively low cost. An illustrative configuration:

# Example using an OpenAI-compatible interface (illustrative, not a real endpoint)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="large-moe-agent-model",
    messages=[
        {"role": "system", "content": "You are a code agent skilled at long-horizon tasks."},
        {"role": "user", "content": "Locate and fix the failing tests in the repository."}
    ],
    extra_body={"thinking_budget": "max"}  # Set thinking tier per task
)

6.2 Key Constraints for Multi-Turn Tool Calling

The official documentation explicitly states: multi-turn tool calling requires preserving the complete historical thinking content and tool-call messages. This is one of the most easily overlooked engineering details in agent deployment. If intermediate tool returns or thinking segments are trimmed when assembling context, the model loses memory of “what has been tried and which paths failed,” leading to repeated calls, circular trial-and-error, or even goal drift. Practical recommendations:

1. In context management, preserve tool-call messages and their corresponding results as indivisible units.
2. In long-chain tasks, prefer relying on the model’s own sparse attention for context compression rather than aggressive external truncation.
3. Set up monitoring for ultra-long sessions, observe the correlation between token growth curves and task success rates, and adjust tiers or split tasks accordingly.

6.3 License Considerations

The model weights use a custom license, not a permissive license such as Apache 2.0 or MIT. Enterprises should carefully review the license terms before commercial use, with particular attention to: scope of commercial use, redistribution conditions, rights and obligations for derivative models, and whether additional restrictions apply to specific use cases. This should be factored into evaluation during the model selection phase to avoid compliance rework later.

7. Selection Advice and Summary

Overall, this model’s technical direction is clear: sparse attention for long-context economics, joint reinforcement learning for cross-domain agent generalization, and multi-tier inference budgets for cost stratification. It is suited for:

• Long-document analysis and repository-level understanding requiring million-token contexts.
• Automated workflows requiring dozens of tool-calling steps, such as terminal operations, bug repair, and retrieval research.
• Enterprises seeking private deployment via open weights with explicit requirements for quantization and inference cost.

At the same time, its boundaries should be clearly understood: benchmark scores are obtained by default at the max tier under specific framework configurations. Real-world performance depends on inference backend adaptation, context management strategy, and task routing design. Converting model capability into a stable production system requires no less engineering effort than the model itself.