Open-Source LLM Breakthrough: 2.8T-Parameter MoE Architecture Powers Long-Horizon AI Agents

Introduction: The LLM Race Enters the Era of Long-Horizon Agents

As large language models (LLMs) are applied to increasingly complex tasks, simple conversational Q&A no longer meets industrial demands. Developers today face real challenges: maintaining goal consistency across dozens or even hundreds of interaction turns, accurately extracting relevant information from massive contexts, and reliably orchestrating external tool calls to complete multi-step workflows. These requirements have given rise to the concept of the “long-horizon agent”—an autonomous system with persistent memory, planning capabilities, and tool-use proficiency.

A newly released open-weights LLM has drawn significant community attention. It employs a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B active parameters, extends the context window to 1 million tokens, and demonstrates strong performance in specialized domains including software engineering, 3D generation, and financial research. This article provides a comprehensive technical overview across five dimensions: architecture design, training methodology, core capabilities, evaluation results, and engineering deployment.

1. Architecture Analysis: The Design Philosophy Behind 2.8T Parameters

1.1 The Asymmetric Balance Between Total and Active Parameters

The most prominent specification of this model is its 2.8T total parameter count. However, the metric that truly determines inference efficiency and cost is the number of active parameters—only 104B are activated during each forward pass. The core value of this design lies in a deliberate balance between storage capacity and computational overhead: the massive parameter pool acts as a large-scale knowledge repository, while the sparse activation mechanism ensures that only a small fraction is used for any given task. This approach is conceptually similar to how the human brain activates only certain neural regions rather than all areas at full intensity.

For developers, 104B active parameters mean that inference can be deployed on a server equipped with 8 high-end GPUs (e.g., 80GB memory per card) rather than requiring a distributed cluster with thousands of cards. This significantly lowers the barrier to private deployment, making frontier model capabilities accessible to mid-sized enterprises.

1.2 Coordination of Gated Sparse Attention and Deep Attention

The model’s innovation in attention mechanisms deserves particular attention. Across a total of 24 Transformer layers, the model uses a newly designed gated sparse attention layer, while the remaining 69 layers rely on the more established deep attention mechanism. The design logic is straightforward: different layers handle information extraction at different granularities.

The gated sparse attention layer employs a dynamic selection mechanism—during each computation, only 2,048 historical tokens are selected to participate in attention weight calculation. This contrasts sharply with traditional dense attention, which computes over the full history. The engineering benefits are threefold:

• Significantly reduced memory usage in long-sequence scenarios, making million-token context windows feasible;
• Suppression of attention pollution from irrelevant information, allowing the model to focus on critical historical nodes;
• Higher inference throughput, as unnecessary matrix multiplications are avoided.

At the same time, retaining deep attention in the majority of layers ensures that the model’s capacity for modeling complex semantic relationships is not compromised by sparsification. This is especially important for cross-paragraph logical reasoning or multi-turn tool-call state tracking, where complete attention paths remain traceable.

2. Training Methodology: Joint Reinforcement Learning for Multi-Domain Alignment

2.1 From Pretraining to Post-Training: A Capability Leap

Building on the base model, the developers implemented an extensive joint reinforcement learning (RL) post-training strategy. Unlike traditional supervised fine-tuning (SFT), which merely requires the model to output in a “correct format,” RL iteratively refines the policy through reward model scoring, optimizing behavior across dynamic evaluations such as multi-step reasoning, code execution result verification, and tool-call success rates.

The “joint” aspect of the joint RL approach emphasizes that the training data covers hundreds of vertical domains, including but not limited to terminal command operations, repository-level code repair, 3D CAD parametric modeling, structured extraction from financial reports, and office suite macro operations. The advantages of this multi-task joint training include:

• Prevention of catastrophic forgetting: New task capabilities are gained while base capabilities remain stable;
• Promotion of cross-task transfer: For example, patterns learned in code error correction can be partially applied to SQL query optimization;
• Improved consistency in instruction following: The model’s response style across different domain user instructions becomes more uniform.

2.2 Thinking Budget Mechanism: Controllable Inference Cost

The model introduces a three-tier “thinking budget” setting during inference: low, high, and highest. This design directly addresses a key developer pain point: not all requests require deep reasoning. For simple factual Q&A or immediate information extraction, the low thinking tier may suffice. For multi-step code refactoring, complex logical reasoning, or long-horizon agent task planning, the highest tier enables the model to generate longer internal reasoning chains, significantly improving task success rates.

In engineering practice, we recommend building a request routing strategy that dynamically adjusts the thinking tier based on task complexity, latency tolerance, and cost constraints—optimizing overall system efficiency.

3. Core Capability Scenarios: From Code Development to Knowledge Work

3.1 Code Development and Software Engineering Automation

Official evaluation data shows the model scored 65.7 on terminal operation benchmarks and 51.7 on software engineering benchmarks (involving cross-file code modifications, test case generation, and repair). These results indicate the model can handle medium-complexity issue-fixing tasks in real repositories. In practice, this capability supports the following workflows:

1. Defect localization: Developers input error messages and relevant code snippets as context; the model assists in identifying root causes;
2. Patch generation: The model generates precise code patches based on contextual understanding, rather than generic code fragments;
3. Terminal command execution: For tasks such as environment configuration or dependency installation, the model can produce executable Shell command sequences and process result feedback.

3.2 3D Design and CAD Parametric Modeling

3D design is a domain most general-purpose models cannot handle well, yet this model performs strongly here. Its core capability lies in parsing natural language design intent and invoking CAD tools to generate parametric feature sequences. On the tool-supported CAD benchmark, the model achieved an impressive score of 92.7. This capability has significant implications for automation in industrial design: engineers can adjust model dimensions, modify extrusion features, or generate assembly constraints through conversational interaction, substantially reducing the time cost of manual parameter adjustments.

3.3 Agent Tasks and Knowledge-Intensive Workflows

For tasks such as browsing and price comparison, financial research report writing, and multi-document comparative analysis, the model demonstrates robust tool-calling and long-context comprehension abilities. It scored 92.1 on browsing and price-comparison tasks and 64.4 on office document processing. This is enabled by the million-token context window, allowing the model to read hundreds of pages of PDFs, multiple Excel spreadsheets, and web pages within a single session, then execute information extraction, cross-validation, and structured output according to user instructions.

4. Interpreting Evaluation Data from an Engineering Perspective

4.1 Evaluation Conditions and Default Configuration

Important note: all evaluation scores reported above were obtained under the “highest thinking tier” setting, and some scenarios (such as terminal operations and code repair) leveraged third-party agent execution frameworks (common industry Agent frameworks, not built-in model capabilities). Developers seeking to reproduce these results must ensure their deployment environment includes a similar external tool-orchestration layer.

4.2 Impact of Sparse Attention on Evaluation Results

The introduction of gated sparse attention results in a degree of “content position sensitivity” in long-context evaluation performance. When key information is located early in the token history or is highly dispersed across different sections of a long document, the model’s performance shows a minor gap compared to dense attention models. However, thanks to the dynamic selection mechanism, this gap is kept within a very small range—evaluations show that on million-token document Q&A tasks, recall of critical information remains at the industry-leading level.

5. Deployment and Engineering Practice Guide

5.1 Weight Quantization and Inference Framework Support

The model’s official release includes MXFP4 (micro floating-point 4-bit) quantized weights. This data format preserves near-BF16 accuracy while compressing model size to roughly a quarter of the original, significantly reducing memory pressure and inference bandwidth requirements. Mainstream inference frameworks (e.g., vLLM, SGLang) already support this weight format, and several high-performance runtime tools have adapted the model. Developers can choose their inference backend based on operational preferences.

5.2 API Compatibility and Tool-Calling Considerations

The model provides two API-compatible paths: one compatible with the widely used OpenAI message format, and another compatible with Anthropic-style prompts. This means developers do not need to rewrite existing LLM application-layer code; a simple adaptation layer allows seamless replacement with this new model, keeping migration costs very low.

A critical engineering detail: for application scenarios requiring multi-turn tool calls, developers must preserve the complete conversation history in the frontend, including the model’s reasoning content and tool execution return messages. If reasoning traces or tool-call records are stripped during message passing, the model will lose contextual continuity in subsequent turns, leading to repeated calls or incorrect inferences. This is the most common configuration error in long-horizon agent deployments.

5.3 License Agreement Key Points

Special attention is required: this model uses a proprietary license rather than common open licenses such as Apache 2.0 or MIT. This means its usage may be subject to additional compliance constraints, including but not limited to commercial-use registration and derivative model release restrictions. Before integrating it into a production environment, developers should read the specific terms carefully and consult their legal teams to ensure compliant usage.

6. Future Outlook: A New Paradigm for Long-Horizon Agent Architectures

The release of this model marks a new phase in the open-source LLM ecosystem: scale is no longer merely a parameter-count competition, but a comprehensive trade-off involving engineering efficiency, inference controllability, and domain adaptability. The combination of 2.8T total parameters and 104B active parameters demonstrates the feasibility of “large but sparse” architectures for industrial deployment. Meanwhile, joint reinforcement learning and the thinking budget mechanism represent a shift from “static knowledge storage” to “dynamic policy adaptation” in model training.

For developers and enterprise technology teams, now is an optimal window to evaluate this model’s effectiveness in your vertical scenarios. We recommend pilot testing along three dimensions:

• Select one small, well-defined scenario (e.g., internal support ticket classification or automatic code comment generation);
• Build an automated evaluation set (covering at least 50 real task samples);
• Measure the relationship between thinking tiers and inference latency to find the sweet spot between performance and cost.

Technological evolution never stops, and those who know when to harness new tools will maintain a first-mover advantage in the wave of intelligence.