Open-Source 2.8T-Parameter MoE Model Targets Long-Horizon Agent Tasks

Model Overview and Positioning

A new large language model with 2.8 trillion total parameters has released its weights to the open-source community. Rather than a brute-force parameter scale-up, the model is a reasoning engine deeply customized for long-horizon agent tasks. Its core architecture uses a sparse mixture-of-experts (MoE) design that activates only about 104 billion parameters per inference, preserving enormous capacity while keeping compute costs under control.

The context window extends to the 1-million-token level, enabling multi-file codebase analysis, long-form technical document comprehension, and complex multi-turn conversations. Even more notable, the model natively supports mixed image-and-text input, paving the way for cross-modal agent operations. Its design explicitly targets three core scenarios: long-horizon coding and software engineering, tool-based agent task execution, and demanding knowledge-worker office workflows.

Architecture Deep Dive: Sparse Attention and Layer Design

From Global Attention to Dynamic Sparse Attention

Compared with its predecessor, the open-source version introduces a significant overhaul of the attention mechanism. Layers that previously used gated multi-head latent attention (Gated MLA) have been replaced with gated dynamic sparse attention (Gated DSA), which now works alongside the remaining 69 Delta Attention layers in a hybrid architecture. The core rationale is balancing long-context capability against inference efficiency.

Under the DSA mechanism, each layer dynamically selects only 2,048 historical tokens for attention score computation. Instead of weighting every historical position equally, the model learns to identify key information nodes. For everyday developers, this directly reduces memory usage and latency during long-sequence inference, making ultra-long contexts feasible on constrained hardware. The mechanism is especially effective in retrieval-heavy agent scenarios, where agents frequently need to revisit earlier instructions or system messages during tool calls.

The Engineering Trade-offs of the MoE Design

Choosing a 2.8-trillion-parameter MoE configuration reflects the team’s pragmatic handling of the tension between knowledge capacity and inference cost. Total parameters determine how much knowledge the model can store, while activated parameters dictate the compute complexity of each forward pass. With roughly 104 billion activated parameters—among the leading tier of comparable open-source models—the design leaves ample compute headroom for complex reasoning while avoiding the linear cost growth of running all parameters on every request.

Training Strategy: Joint Reinforcement Learning on Real-World Tasks

Training did not stop at conventional pretraining and fine-tuning. The core stage is large-scale joint reinforcement learning (RL) post-training built on the previous-generation base model. This aligns with the current mainstream trend in AI alignment: letting the model autonomously refine its decision policies through reward signals while operating in simulated real-task environments.

Generalization Across Hundreds of Domains

According to the release documentation, the joint RL stage covered hundreds of downstream domains. The key advantage is that the model avoids overfitting to a single benchmark and instead learns general problem-solving patterns across diverse task distributions. For example, on code-repair tasks it learns not just syntax fixes but also how to interpret build logs and test feedback; on office document processing it acquires a combined skill set spanning format compliance, information extraction, and logical restructuring.

Quantization-Aware Training

The released weights use the MXFP4 quantization format. This was not post-hoc compression—MXFP4 is deeply integrated into the training and fine-tuning pipeline. As a microscaling floating-point format, MXFP4 maintains a high dynamic range at 4-bit width, reducing the model’s footprint to roughly a quarter of its original size while retaining as much precision as possible. For deployment, this means developers can experiment with loading the model on a single GPU or a small cluster, rather than requiring a large multi-GPU setup.

What the Benchmark Numbers Reveal

Open-source communities tend to judge models by reproducible benchmark results. Officially published scores show standout performance across several task dimensions:

• Terminal interaction and system operations: 65.7 on Terminal-Bench 4.0, validating the model’s ability to understand command-line semantics and execute multi-step operations—the foundation for autonomous agent work in Unix environments.
• Software engineering marathons: 51.7 on SWE Marathon v1.1 and 75.5 on DeepSWE v1.1, reflecting generalization on high-difficulty tasks such as resolving real GitHub issues and reviewing pull request code. These benchmarks carry weight because they simulate developers’ long, messy daily workflows rather than isolated algorithm puzzles.
• Information retrieval and web interaction: 92.1 on BrowseComp, indicating highly precise web information localization and comprehension—critical for building automated web research agents.
• 3D design assistance: 92.7 on BenchCAD (with tools), a figure worth noting. CAD generation requires translating natural language descriptions into parametric operation sequences, so this score suggests emerging geometric reasoning and engineering drawing literacy.
• Office document automation: 64.4 on OfficeQA Pro v2, covering common pain points such as spreadsheet handling, presentation generation, and complex document layout.

Keep in mind that most of these results were obtained at the highest inference budget tier (max), and some suites used widely adopted industry agent frameworks for tool-call orchestration. Treat these numbers as a reference for the model’s reasoning ceiling, not as absolute throughput metrics for production environments.

Inference Deployment and Tool-Calling Practice

From an engineering standpoint, the official weights already support mainstream inference backends such as vLLM and SGLang, along with lightweight inference engines, and the model is exposed through an API shape compatible with industry standards. Developers can therefore migrate existing applications from closed-source endpoints to a local deployment stack relatively smoothly.

Key Implementation Notes

When integrating multi-turn tool calling, one detail is critical: context retention integrity. Because the model relies on the full history of reasoning traces and tool-return messages for its next decision, disable any form of automatic conversation trimming or history summarization in your API configuration. Forcibly compressing early context causes the model to lose track of the task objective, breaking the agent execution chain.

For users targeting lower-VRAM environments, MXFP4 weights provide a starting point, but they should be combined with optimizations such as KV cache quantization and continuous batching. In practice, a single GPU with at least 80GB of memory (e.g., mainstream A100/H100-class accelerators or equivalents) is needed to ensure smooth inference for medium-length sessions.

License and Usage Boundaries

This release follows a dedicated model license rather than a common permissive open-source license such as Apache 2.0 or MIT. Developers must review the specific terms on use restrictions, derivative model licensing, and branding before commercial use or redistribution. Compliance is the first hard line for bringing a model to production; enterprise users are advised to involve their legal teams before integrating the model into core business workflows.

Conclusion

This release marks a significant milestone in the development of AI foundation models. It demonstrates the substantial potential of the “sparse architecture plus large-scale reinforcement learning” route for general agent tasks, and gives small and mid-sized research teams a rare material foundation for frontier experimentation. From an industry perspective, the open release of MoE weights at this scale will further lower the entry barrier for long-horizon automation solutions, pushing AI from conversational Q&A toward autonomous execution. For developers, now is the ideal time to study the model’s technical report in depth and test the boundaries of its capabilities firsthand.