Inside the 2.8T MoE Model: Long-Term Agent Capabilities and Sparse Attention

1. Background: New Architectural Requirements for Long-Term Agents

Over the past year, Agent-based applications have rapidly evolved from simple single-turn Q&A with tool calling to multi-step planning, cross-session memory, long-document comprehension, and repository-level code operations. These long-term tasks impose three hard requirements on the underlying models:

First, the context window must be sufficiently large. A complete software engineering task might involve reading, modifying, and verifying dozens of files. Combined with tool call histories, intermediate reasoning results, and error backtracking information, this can easily exceed hundreds of thousands of tokens. If the context window is insufficient, the Agent is forced to discard historical information at critical nodes, leading to a sharp degradation in decision quality.

Second, the reasoning process must be controllable and budgetable. Task complexity varies significantly. Executing a terminal command might require only a single fast inference pass, while CAD generation or cross-file code repair demands deep thinking. The model needs to offer different tiers of “thinking budgets,” allowing developers to flexibly allocate compute based on the task type.

Third, the attention mechanism must maintain efficiency at large context windows. Standard full-attention mechanisms see compute and memory consumption scale quadratically at million-token contexts, making inference costs unacceptable. Sparse attention is the necessary path forward, but maintaining model quality while sparsifying is the core engineering challenge.

Recently, a team open-sourced a 2.8T total parameter MoE architecture model with 104B activated parameters, providing a systematic answer to the three issues above. This article will analyze the model across four dimensions: architectural modifications, training strategies, inference budgeting, and benchmark performance.

2. Architecture Analysis: Layered Replacement Strategy for Sparse Attention

2.1 MoE Foundation

The model employs a Mixture of Experts (MoE) architecture with 2.8T total parameters, activating approximately 104B parameters per forward pass. The core idea of MoE is to split FFN layers into multiple experts, using a routing network to select a few experts for activation per token. This controls the compute cost of a single inference pass while maintaining the capacity of a large model.

The 104B activation size represents an upper-mid scale for MoE models. This means the compute overhead of a single inference pass is comparable to a 100B-scale dense model, but the total model capacity is far greater. This design is particularly advantageous for long-term Agent tasks: domain-specific knowledge can be distributed across different experts, and the routing network dynamically selects them based on input, avoiding the interference between different domain knowledge that often occurs in dense models.

2.2 Sparsification of Attention Layers

The model introduces a key modification to the attention mechanism: replacing 24 layers of the original Gated MLA (Multi-head Latent Attention) with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention.

Two core concepts are essential to understand here:

How DSA Works: During each attention computation, DSA selects 2,048 historical tokens to participate in the calculation. This selection is dynamic—not a simple sliding window or fixed-step sampling, but a filtering process based on the relevance between the current Query and historical Keys. The selection size of 2,048 tokens is an engineering compromise: too few risks losing critical information, while too many diminish the benefits of sparsification.

Combination of Gated MLA and Delta Attention: The gating mechanism controls the strength of information flow, while Delta Attention focuses on capturing differential information between tokens. Theoretically, combining both maintains the modeling capability for long-range dependencies while achieving sparsification.

From an engineering perspective, this layered replacement strategy is noteworthy. Not all layers use the same attention mechanism; instead, different attention schemes are configured based on the layer’s position and functional role. This “hybrid attention” design is gradually becoming a mainstream approach for long-context scenarios.

2.3 Quantization and Deployment

The weights utilize the MXFP4 quantization format. MXFP4 is a 4-bit floating-point quantization scheme that, compared to INT4, preserves a better dynamic range in very small numerical intervals, making it suitable for parameters like attention weights that have a wide distribution span. The quantized model can be deployed on mainstream inference frameworks and provides API interfaces compatible with OpenAI and Anthropic styles.

It is important to note that in multi-turn tool calling scenarios, the complete thinking history and tool call messages must be retained. This means that when constructing a request, you cannot simply keep the final round of user messages; you must pass in all previous Assistant thinking content, Tool call requests, and Tool return results. This raises the engineering requirements for request construction logic and context management.

3. Training Strategy: Joint Reinforcement Learning and Multi-Domain Coverage

3.1 Post-Training from Base Model to Agent Capabilities

Based on a previous generation base model, this model underwent large-scale joint reinforcement learning (RL) post-training. The “joint” aspect is reflected on two levels:

Domain Jointness: Training covers hundreds of downstream domains, including code development, terminal operations, web retrieval, financial research, office document processing, and CAD generation. Traditional approaches fine-tune a separate model for each domain and then ensemble them. Joint training allows a single model to optimize across multiple domains simultaneously, avoiding the latency and consistency issues caused by model switching.

Task Jointness: Training tasks are not limited to single-turn Q&A but include Agent-specific tasks like multi-step tool calling, long-term planning, and error recovery. The RL reward signals are derived from the final state of task completion, rather than step-by-step token supervision. This approach closely aligns with the actual usage scenarios of Agents.

3.2 Three-Tier Thinking Budget Design

Inference provides three thinking budget tiers: low, high, and max. This is not simply a “max token limit” but involves depth control over the reasoning path:

• Low tier: Suitable for simple tool calls, format conversions, and single-step queries. The reasoning path is short with low latency.
• High tier: Suitable for multi-step code repair and medium-complexity analysis tasks, allowing the model to unfold multiple rounds of internal reasoning.
• Max tier: Suitable for long-term Agent tasks, complex CAD generation, and cross-file software engineering, enabling the model to perform thorough planning, verification, and backtracking.

Developers can dynamically select a tier based on the task type, balancing latency and success rate. Official evaluation data shows that defaulting to the max tier yields high scores across all domains.

4. Benchmark Performance and Capability Boundaries

Official benchmarks cover scenarios such as terminal operations, code repair, web retrieval, CAD generation, and office document processing. Key data includes:

Benchmark	Score	Description
Terminal-Bench 4.0	65.7	Terminal operations and command execution
SWE Marathon v1.1	51.7	Long-term software engineering tasks
DeepSWE v1.1	75.5	Deep code repair
BrowseComp	92.1	Web retrieval and information extraction
BenchCAD (with tools)	92.7	CAD generation tasks
OfficeQA Pro v2	64.4	Office document comprehension and Q&A

It should be noted that some evaluations utilized Agent frameworks (e.g., code Agent tools) alongside the model. These scores reflect the overall capability of the “model + toolchain” rather than the raw model running in isolation. Developers referencing this data should pay close attention to the specific evaluation configurations and tool dependencies.

Looking at the data distribution, the model performs exceptionally well on retrieval and CAD generation tasks, while there is still room for improvement in long-term software engineering tasks. This is closely tied to the inherent difficulty of long-term Agent tasks: cross-file dependencies, error propagation, and state management remain engineering challenges that are not fully resolved.

5. Engineering Implementation Recommendations

For developers planning to integrate this model into Agent systems, consider the following:

Context Management Strategy: A million-token window does not mean you can inject content limitlessly. It is recommended to perform pre-filtering at the request construction layer, prioritizing the most relevant files, historical messages, and tool results to reduce the occupation of attention resources by invalid tokens.

Dynamic Selection of Thinking Budget: Do not use the max tier for all tasks. Adaptive tier switching can be implemented based on signals such as task type, historical success rate, and latency requirements.

Message Retention for Multi-Turn Tool Calls: Ensure every request includes the complete chain of thought and tool call records. Some frameworks default to keeping only the final result, which breaks the model’s reasoning continuity.

License Compliance: The model uses a proprietary license, not a commonly permissive open-source protocol. Commercial use requires carefully reading the license terms to ensure the use case complies with the requirements.

6. Conclusion

The combination of a 2.8T MoE architecture, million-token context, sparse attention modifications, a three-tier thinking budget, and joint RL across hundreds of domains points to a clear goal: equipping open-source models with deployable long-term Agent capabilities. Benchmark data shows substantial progress in this direction, but state consistency, error recovery, and tool coordination in long-term tasks remain engineering topics requiring continuous investment. For teams building Agent systems, understanding the trade-offs behind these architectural choices is more valuable in the long run than merely focusing on benchmark scores.