Open-Source 2.8T-Parameter MoE Model: A Deep Dive into Long-Horizon Agents and Multimodal Capabilities

Introduction: The Paradigm Shift from “Conversation” to “Long-Horizon Task Execution”

The capability boundary of large language models is rapidly expanding from single-turn Q&A and short-text generation toward complex agent tasks that require multi-step planning, tool invocation, and long-range context dependencies. While traditional sparse Mixture-of-Experts (MoE) models can achieve breakthroughs in scale, they often face bottlenecks from the quadratic computational complexity of attention mechanisms and inference state management when dealing with codebases spanning hundreds of thousands of tokens, multi-turn web retrieval with tool coordination, and 3D CAD design scenarios. Recently, a mainstream AI lab officially open-sourced its next-generation flagship model—an MoE architecture with a total parameter count of 2.8T and only 104B activated parameters. This model not only sets multiple new SOTA results on standard benchmarks but also demonstrates remarkable engineering maturity on the core track of “long-horizon agents.”

Based on publicly available technical reports and weight release information, this article provides a systematic technical breakdown of the model’s architecture design, joint reinforcement learning training strategy, sparse attention mechanism innovations, inference deployment solutions, and typical application scenarios, while discussing implications for enterprise-level AI application deployment.

Model Architecture: “Slimming Down” Activation Under 2.8T Total Parameters

The model’s parameter scale reaches an impressive 2.8T, but thanks to the MoE (Mixture of Experts) architecture, each forward pass requires activating only 104B parameters. This “total parameters to activated parameters” ratio (approximately 27:1) represents an extremely high sparsity design in the industry, meaning the model maintains enormous knowledge capacity while keeping inference-time computational overhead (FLOPs) and memory bandwidth requirements within acceptable ranges.

Specifically, the model continues the hybrid attention mechanism framework of its base version and introduces key upgrades on top of it. The core innovation lies in replacing the original 24 layers based on Gated MLA (Multi-head Latent Attention) with Gated DSA (Dynamic Sparse Attention), which works in coordination with the remaining 69 layers of Delta Attention structure. This “dynamic sparse + incremental” hybrid attention design is the key to the breakthrough in long-context processing capability achieved by this open-source release.

From an engineering implementation perspective, MoE load balancing and expert parallelism strategies are also important factors determining training stability. The model employs fine-grained expert sharding and routing strategies during training, effectively avoiding the “expert collapse” problem (where some experts are overloaded while others sit idle), ensuring efficient collaborative training of 2.8T parameters across tens of thousands of accelerator cards.

Sparse Attention Mechanism: Gated DSA and Delta Attention Working in Concert

Long-context capability is the foundation supporting long-horizon agent tasks. The model supports a 1 million token context window, far exceeding the 128K or 200K levels of typical commercial models. The physical basis for achieving this lies in attention mechanism innovation.

Gated DSA (Gated Dynamic Sparse Attention) is designed with the following logic: in each layer’s computation, rather than computing attention between the current token and all historical tokens in the sequence, a learnable gating mechanism dynamically selects the most relevant 2,048 tokens from the historical token pool to participate in attention computation. This is equivalent to equipping each query token with a “targeted retriever” rather than performing exhaustive global matching. This strategy dramatically reduces computational complexity for long sequences, making linear complexity possible.

Meanwhile, the model retains 69 layers of Delta Attention (Incremental Attention) , which focuses on capturing local relationships and incremental changes between consecutive tokens, responsible for maintaining fine-grained perception of local context. The division of labor between the two attention mechanisms is clear: DSA handles global sparse association capture, while Delta Attention handles local dense information retention. This “local dense + global sparse” combination enables the model to precisely locate key information within contexts as long as one million tokens without losing the continuity of micro-context.

Notably, DSA’s token selection is not a fixed pattern but content-aware dynamic routing. In code generation tasks, it tends to select function definitions and variable references related to the current function call; in long-document Q&A, it prioritizes sentences containing entity keywords. This dynamic nature is the core advantage distinguishing this model from fixed-window sparse attention (such as sliding windows).

Training Strategy: Large-Scale Joint Reinforcement Learning Covering Hundreds of Domains

In terms of training methodology, the model does not limit itself to the traditional pre-training-then-fine-tuning two-stage paradigm but emphasizes large-scale joint reinforcement learning post-training on top of the base model. The objective function of this training approach is no longer merely minimizing cross-entropy loss but directly optimizing task-level metrics (such as code pass rates, tool invocation success rates, and answer accuracy).

The joint reinforcement learning process covers hundreds of downstream domains, spanning code development, software engineering, data analysis, agent interaction, mathematical reasoning, and multimodal understanding. This broad training domain coverage brings two significant benefits: first, the model possesses stronger task generalization ability without overfitting to specific capability sets; second, through unified reinforcement learning reward signals, the model learns to transfer strategies across different stages of complex tasks.

For example, in a typical “terminal operation + code repair” composite task, the model must not only generate correct code snippets but also understand the current state of the terminal environment, parse error logs, and plan the next terminal command. Joint reinforcement learning enables the model to integrate these heterogeneous skills within a single policy network, rather than relying on chaining multiple independent models.

Inference Optimization and Deployment Practices: MXFP4 Quantization and TokenSpeed

Since 2.8T total parameters far exceed single-GPU memory capacity, the model simultaneously provides an MXFP4 (Microscaling Floating Point 4-bit) weight quantization version upon open-source release. MXFP4 is an advanced 4-bit quantization format designed for large model inference that applies block-wise dynamic scaling to weight tensors through microscaling factors, compressing model size to approximately 1/4 of the original FP16 (16-bit floating point) while maximally preserving the dynamic range of critical weights. In practical testing, the accuracy loss from MXFP4 quantization is typically kept within a very small range, while inference throughput gains reach several multiples.

In terms of inference framework support, the model is compatible with mainstream vLLM, SGLang, and TokenSpeed backends. For deploying agent-type applications, special attention must be paid to state management across multi-turn tool invocations. The official recommendation is to retain the complete conversation history (including thinking processes and structured messages of tool calls) rather than only keeping final summary text. This is because in long-horizon tasks, the model needs to trace back to earlier decision paths.

Deployment Considerations:

• KV Cache Optimization: For 1 million token contexts, KV Cache memory usage is a factor that must be considered. Enabling PagedAttention or TokenSpeed’s KV Cache management features can significantly reduce memory fragmentation.
• API Compatibility Layer: Provides both OpenAI and Anthropic API-compatible interfaces for seamless integration with existing agent frameworks.
• Multi-Turn Conversation State: It is recommended to separately store “thinking/reasoning” and “tool execution results” at the application layer, and carry the complete tool call chain in requests.

Capability Boundaries and Benchmark Performance: Interpreting Hard Data

The officially published benchmark data is highly compelling. Here are several key metrics for analysis:

• Terminal-Bench 4.0 (65.7 points) : This benchmark evaluates the model’s ability to execute multi-step commands (such as Git operations, file retrieval, and environment configuration) in real Linux terminal environments. The high score of 65.7 indicates the model possesses deep understanding of system states and command sequence planning capabilities.
• SWE Marathon v1.1 (51.7 points) and DeepSWE v1.1 (75.5 points) : These are hardcore evaluations in the software engineering domain, requiring the model to resolve GitHub Issues in real open-source repositories. In particular, DeepSWE’s 75.5 score demonstrates that the model can not only modify individual files but also coordinate cross-module modifications and handle regression tests.
• BrowseComp (92.1 points) : Complex tasks involving web retrieval and information aggregation. The high score proves the model’s strong capabilities in multi-hop retrieval, noise filtering from web content, and information distillation.
• OfficeQA Pro v2 (64.4 points) : An intelligent assistant test oriented toward office software (such as spreadsheets and presentations) operations. This reflects the model’s ability to parse and generate structured documents.

All the above results were obtained using the highest thinking budget (max) setting. The model offers three thinking budget levels—low, high, and max—to accommodate different user trade-offs between latency and accuracy. For interactive chat scenarios, the low setting is recommended to reduce time-to-first-token latency, while for code review or financial modeling scenarios requiring deep reasoning, switching to the max setting is advisable.

Application Scenarios and Deployment Practices

1. Long-Horizon Code Agents and Software Development Automation

The model’s core battleground lies in the intelligent transformation of software engineering. With a 1 million token context, the model can serve as a “pair programming agent” that resides in the development environment, reading the entire remote code repository in real time (including text representations of large binary resource files). When handling cross-module feature development, it can precisely locate key functions across dozens of files through the DSA mechanism and generate suggested modifications that conform to the project’s existing code style.

2. 3D Modeling and CAD/CAM Integration

In the 3D design domain, the model generates CAD operation command sequences by parsing parametric modeling history. The BenchCAD score of 92.7 in official evaluations indicates that the model can understand boundary representation (B-rep) data formats and generate executable design command streams, which will greatly lower the learning curve for professional 3D modeling software.

3. Financial Research and Enterprise Knowledge Management

In the financial domain, the model can be used to process massive volumes of earnings report PDFs, market research reports, and macroeconomic data. Through a “deep research” mode, it can autonomously plan research paths: first retrieving data sources, then extracting key financial indicators, and finally generating structured research reports. For internal enterprise operations, the model supports complex Office document batch processing—from spreadsheet data auditing to PPT outline generation.

Commercial License and Usage Boundaries

It is important to note that while the model has open-sourced its model weights, the license it adopts is not a permissive license such as Apache 2.0 or MIT. Instead, it uses a dedicated custom license for this model (Kimi K3.1 Pro License). This means that when using it in commercial products or redistributing it, the specific provisions regarding “commercial scope,” “derivative restrictions,” and “terms of service limitations” must be strictly reviewed. It is recommended that legal counsel and compliance teams be involved in assessing compliance, especially concerning specific details related to cloud service provisioning or model hosting.

Summary and Outlook

Through its 2.8T total parameter MoE architecture, innovative Gated DSA sparse attention mechanism, and joint reinforcement learning spanning hundreds of domains, this open-source model establishes a new benchmark for long-horizon agent tasks. For developers, this is not only a powerful foundation model but also a complete, engineering-validated solution for long-context processing and tool invocation.

With MXFP4 quantization and improved support from mainstream inference frameworks, the deployment barrier for this model has been significantly lowered. In the future, as larger-scale model parameters and rapid iteration inference optimization strategies develop in parallel, the role of large models in software engineering, scientific discovery, and enterprise automation will not merely be an auxiliary tool but will become a true autonomous executor. Enterprises and developers should closely monitor the development of this technical route and begin conducting proof-of-concept validation and cost assessment in specific business scenarios. From a technology evolution perspective, the combination of million-level context and dynamic sparse attention has already laid a solid technical foundation for the next generation of workflow automation.