A 2.8T-Parameter Open MoE Model for Long-Horizon AI Agents

Introduction: From Model Scale to Agent Capability

As large language model technology matures, parameter count is no longer the only measure of capability. Industry practitioners increasingly recognize that real-world agent applications depend on a model’s ability to plan, remember, and invoke tools across long sequences.

A recently released open-weight MoE model addresses these requirements with 2.8T total parameters and 104B active parameters. Its core highlights include a 1M-token context window, native image-text input, and deep optimization for long-horizon agent tasks. This release marks a meaningful step forward for open models in complex task handling.

Architecture: Sparse Attention and Efficient Parameter Coordination

1. Parameter Efficiency in a MoE Design

The model uses a classic Mixture-of-Experts design. Although it contains 2.8T total parameters, only 104B are activated for a single inference pass. The philosophy behind this approach is to distribute knowledge across numerous expert sub-networks, preserving strong expressive capability while significantly reducing computational overhead.

For developers, this means that under the same compute budget, the model can process longer contexts or serve more concurrent requests.

2. Dual Attention Mechanism

Compared with the previous generation, this model introduces a major change in its attention layers. The original 24 Gated MLA layers are replaced by Gated DSA, which works alongside 69 Delta Attention layers. During each inference step, the DSA mechanism dynamically selects only 2048 historical tokens for attention computation instead of attending to all history positions.

This design alleviates the quadratic computation bottleneck commonly associated with long sequences, making 1M-token contexts practical while preserving the model’s ability to capture key historical information.

3. Joint Reinforcement Learning Post-Training

The training strategy is also notable. After initial pre-training, the team applied joint reinforcement learning post-training across hundreds of downstream domains. Unlike traditional supervised fine-tuning, this approach allows the model to align with human preferences and task completion simultaneously on heterogeneous tasks such as code generation, terminal operations, and office document processing.

The weights are released in MXFP4 format, reducing the model size to roughly one quarter of the original with nearly no performance loss and significantly lowering the deployment barrier.

Core Capabilities: From Code to Knowledge Work

1. Long-Horizon Code Agents and Software Engineering

Benchmark results reveal the model’s potential in real software engineering scenarios. It achieves 65.7 on Terminal-Bench 4.0, 51.7 on SWE Marathon v1.1, and 75.5 on DeepSWE v1.1. These tasks require more than generating code snippets: the model must understand repository structures, execute command-line instructions, read compilation errors, and iteratively fix issues. It can decompose multi-step tasks into tool invocation sequences, such as calling a code interpreter, querying API documentation, or executing version control commands.

2. Agents and Knowledge-Intensive Work

On BrowserComp, the model scores 92.1, demonstrating its ability to navigate dynamic web pages, complete forms, and extract information. For scenarios such as financial research and office document processing, the model can invoke external tools for data organization and report generation. In the BenchCAD tool-based evaluation, a score of 92.7 indicates that it can understand 3D modeling constraints and generate editable CAD operation sequences.

3. Controllable Reasoning Budget

The model provides three reasoning budget levels: low, high, and max. This design lets developers balance latency and quality according to the task. For simple questions such as basic arithmetic, the low setting delivers fast responses. For tasks involving multiple rounds of function calls and numerical reasoning, such as financial analysis, the max setting activates a longer internal chain of thought, improving decision accuracy.

Benchmarks: Official Results and Interpretation

The following table lists results reported by the model team. Scores use the max reasoning setting, and some evaluations rely on common agent frameworks.

Benchmark	Score	Task Type
Terminal-Bench	65.7	Terminal interaction, CLI execution and repair
SWE Marathon	51.7	Complex engineering tasks across multi-file codebases
DeepSWE	75.5	Deep software engineering and dependency error repair
BrowseComp	92.1	Web browsing, information retrieval and content extraction
BenchCAD	92.7	Parametric 3D model generation with tools
OfficeQA Pro	64.4	Office document Q&A, table parsing and generation

It is worth noting that benchmark scores are influenced by test sets and agent frameworks. When reproducing results, developers should pay attention to how the model is invoked, such as whether complete historical reasoning traces are retained, to obtain the most accurate performance picture.

Engineering Practice: Deployment and Inference Optimization

1. Inference Framework Compatibility and Serving

The MXFP4 weights are compatible with common high-performance inference frameworks such as vLLM, SGLang, and TokenSpeed. Developers can set up a service with the following pattern:

# Example command: load model weights into an inference engine
python -m vllm.entrypoints.openai.api_server \
    --model /path/to/weights/ \
    --quantization mxfp4 \
    --max-model-len 1000000

2. Key Considerations for Multi-Turn Tool Calling

When building agent applications, it is essential to preserve full historical context across multi-turn tool calls. After each tool result is returned, developers must append the previous system prompt, reasoning trace, tool call messages, and tool output to the next request. Truncating history causes the model to lose execution state and produce incorrect subsequent actions.

3. API Compatibility and Ecosystem Integration

The model service provides OpenAI- and Anthropic-compatible API interfaces. This allows existing ecosystem tools such as LangChain and LlamaIndex to integrate without modifying core business logic.

License and Compliance Considerations

The model weights are released under a custom license, not a permissive license such as Apache 2.0 or MIT. Before integrating the model into commercial products, developers should carefully review the license terms regarding commercial use, obligations for derivative works, and attribution requirements to avoid legal risks.

Summary and Outlook

This 2.8T-parameter MoE model validates the effectiveness of combining large-scale sparse activation, sparse attention, and cross-domain reinforcement learning for long-horizon agent tasks. For technical teams, it offers not only a performance reference but also an engineering paradigm for efficiently using 100B-level active parameters to process million-token contexts through techniques such as MXFP4 quantized storage and dynamic sparse attention.

As tool-calling protocols and inference infrastructure continue to mature, we can expect more applications built on such foundation models in areas including automated programming, intelligent research, and design generation. For developers, now is an opportune time to explore long-context agent applications in depth.