Technical Deep Dive into an Ultra-Large Sparse MoE Open-Source Model: From 2.8T Parameters to Long-Horizon Agent Capabilities

The open-source large language model ecosystem is shifting from merely scaling parameters to optimizing effective compute. As total parameter counts cross into the trillions, a model’s usability is no longer dictated by sheer size. Instead, it depends on the activated parameter ratio, sparse attention mechanisms, and the ability to maintain state consistency across long-horizon tasks. A recently open-weighted ultra-large Mixture-of-Experts (MoE) model exemplifies this trend: featuring 2.8T total parameters, 104B activated parameters, a 1-million-token context window, native multimodal inputs, and systematic post-training tailored for long-horizon coding agents and knowledge work scenarios. This article analyzes the model across four dimensions—architecture, training, evaluation, and deployment—offering developers an actionable technical reference.

1. Overall Design: Balancing Trillion-Scale Parameters and Billion-Scale Activation

The model utilizes an MoE architecture with 2.8T total parameters and approximately 104B activated parameters per forward pass. This ratio allows the model to retain a massive knowledge capacity while keeping the compute cost per token near the level of a 100B-scale dense model, representing a mainstream engineering approach for current ultra-large models.

Core capability metrics include:

• Context Length: Supports up to 1 million tokens, sufficient to cover large code repositories, extensive document collections, or long-spanning agent interaction histories.
• Input Modality: Natively supports mixed image and text inputs, capable of processing visual information like UI screenshots, charts, and design drafts.
• Inference Budget: Offers three thinking budget tiers—low, high, and max—allowing dynamic allocation of compute overhead based on task complexity.
• Weight Format: Released with MXFP4 quantized weights, lowering the barriers for VRAM and bandwidth.

It is crucial to emphasize that a long context window does not automatically equate to effective long-context performance. The true value of a million-token window relies on sparse attention mechanisms; otherwise, computational complexity grows quadratically with sequence length, rendering inference costs unacceptable.

2. Sparse Attention: Combining Gated DSA and Delta Attention

Compared to its predecessor, the new model implements a critical modification in the attention layers: replacing 24 layers of Gated MLA (Multi-head Latent Attention) with Gated DSA (a sparse attention variant), combined with 69 layers of Kimi Delta Attention.

The core mechanism of DSA is that each attention computation selects only 2,048 historical tokens to participate in the calculation, rather than performing dense attention across the entire window. This design yields two direct benefits:

1. Computational Scalability: As sequence length increases, attention overhead grows approximately linearly rather than quadratically, making a million-token window engineering-feasible.
2. Information Selectivity: Through a gating mechanism that filters historical information, the model can actively focus on fragments relevant to the current task, reducing interference from distant noise.

This hybrid structure of “sparse selection + linear attention” effectively strikes a balance between expressiveness and efficiency: a few sparse layers handle precise retrieval of critical history, while the majority of linear attention layers maintain global state compression. For long-horizon agents, this combination is particularly vital—agents need to accurately trace back early conventions, variable names, and task constraints even after dozens or hundreds of tool calls.

3. Training Strategy: Joint Reinforcement Learning Across Hundreds of Domains

The model undergoes large-scale joint reinforcement learning (RL) post-training based on its predecessor base model, with RL covering hundreds of downstream domains. The significance of this strategy includes:

• Capability Aggregation Over Single-Point Optimization: Single-domain RL often leads to trade-offs where one capability improves at the expense of another. Multi-domain joint training encourages the model to develop transferable reasoning patterns across different tasks.
• Agent-Behavior-Oriented Training: Tasks like terminal operations, code repairs, and retrieval research are inherently multi-step decision-making problems, naturally aligning with RL’s sequence decision modeling.
• Weight and Inference Synergy: MXFP4 weight quantization is applied during the training phase, allowing the model to adaptively absorb quantization errors during training and avoid the precision collapse typically caused by post-training quantization.

For teams looking to build upon these open weights for secondary development, this means the model already possesses good numerical stability in its quantized state, reducing the workload required for custom calibration.

4. Capability Profile: What the Evaluation Data Tells Us

Officially released benchmark scores cover two main tracks: agents and professional tasks (utilizing the max thinking tier by default, with some evaluations completed using an AI coding tool framework):

Benchmark	Score	Capability Indicator
Terminal-Bench 4.0	65.7	Multi-step command operations in terminal environments
SWE Marathon v1.1	51.7	Continuous execution of long-horizon software engineering tasks
DeepSWE v1.1	75.5	Complex code repair and engineering comprehension
BrowseComp	92.1	Web retrieval and information locating
BenchCAD (with tools)	92.7	3D/CAD generation and tool coordination
OfficeQA Pro v2	64.4	Office document comprehension and processing

There are three key takeaways from this data:

First, there is a noticeable performance gradient between long-horizon and short-horizon tasks. Evaluations like SWE Marathon, which require multi-round modifications and maintaining task state consistency, score significantly lower than single-round retrieval tasks. This indicates that long-horizon consistency remains a bottleneck for current agent capabilities, serving as the direct driver for the continuous evolution of context and attention design.

Second, tool coordination evaluations show outstanding performance. BenchCAD achieved 92.7 under tool-augmented conditions, demonstrating the model’s strong capability in process orchestration for “calling external tools to complete multi-step tasks,” which is closely related to native tool-calling training.

Third, evaluation conditions must be transparently documented. Different evaluations utilize different thinking tiers and agent frameworks. When comparing horizontally, attention should be paid to condition consistency to avoid directly comparing numbers derived from different configurations.

5. Inference Budget: The Engineering Value of Three-Tier Thinking Scheduling

The three thinking budget tiers—low, high, and max—essentially turn “inference-time compute” into a configurable resource. In engineering practice, the following strategy can be referenced:

# Pseudo-code: Selecting thinking tier based on task complexity
def select_budget(task):
    if task.type in ("simple_lookup", "format_convert"):
        return "low"      # Low latency priority, suitable for high-frequency lightweight calls
    elif task.type in ("code_patch", "doc_summary"):
        return "high"     # Balance quality and cost
    else:                  # Long-horizon agents, complex reasoning, CAD generation
        return "max"      # Quality priority, accepting higher latency

The practical value of this design lies in cost controllability: batch and retrieval tasks can use the low tier to compress unit costs, while complex engineering tasks can switch to higher tiers, avoiding the compute waste caused by a “one-size-fits-all” approach. For teams building agent pipelines, it is recommended to incorporate tier selection into the orchestration layer rather than hardcoding it into the calling logic.

6. Deployment and Integration: How to Use the Open Weights

The weights are released in MXFP4 format, compatible with mainstream high-performance inference engines, and provide OpenAI- and Anthropic-compatible API formats. Several engineering details require attention during implementation:

1. History Preservation for Multi-Round Tool Calls
In multi-round tool calling scenarios, it is essential to completely preserve historical thinking content and tool call messages. This is because sparse attention relies on historical fragments for selective retrieval. If the orchestration layer truncates or summarizes the history, the model may lose critical intermediate decision-making context, leading to repeated calls or state drift. It is recommended to explicitly distinguish between “compressible chat history” and “non-compressible decision chains” in session management.

2. VRAM Planning for Quantized Weights
MXFP4 weights significantly reduce VRAM usage, but expert parallelism in MoE models still requires accounting for load fluctuations caused by uneven routing distribution. During deployment, combine the engine’s expert parallelism configuration to reserve peak VRAM headroom, and monitor alerts to observe the actual utilization of each expert.

3. Cost Control for Long Contexts
A million-token window should not be used indiscriminately. It is recommended to perform input filtering at the application layer: prioritize injecting code fragments, document paragraphs, and tool return results strongly relevant to the current task, while downgrading low-frequency history to summaries. The longer the context, the greater the selection pressure on sparse attention; signal-to-noise ratio management is more critical than window size.

4. License Compliance
The weights are released under a proprietary license, not a permissive one like Apache 2.0 or MIT. Before commercial use, legal teams must confirm the scope of use, redistribution conditions, and derivative model clauses to avoid equating “open-source downloadable” with “unrestricted commercial use.”

7. Practical Recommendations for Developers

Synthesizing the analysis above, here are several actionable recommendations:

• Start Small, Scale Up: First validate the task pipeline using the low thinking tier and a smaller context window, then gradually expand the tier and window to identify performance bottlenecks.
• Document Evaluation Conditions: Record the tier, framework, and toolset used for each evaluation to ensure internal data comparability within the team.
• Prioritize State Management: Failures in long-horizon agents mostly stem from state loss rather than single-step reasoning errors. Session storage and history pruning strategies should be treated as first-class design citizens.
• Validate Quantization First: Run a precision baseline using MXFP4 weights on the target inference engine before evaluating whether a higher precision format is necessary.
• Compliance First: Define license boundaries clearly during the architecture design phase to avoid late-stage rework.

The combination of trillion-scale parameters, billion-scale activation, million-token context, and sparse attention signifies that open-source LLMs are transitioning from “capability demonstration” to “engineering usability.” For enterprise users, true competitiveness lies not in being the first to integrate a specific set of weights, but in the ability to build a stable, observable, and cost-controlled agent infrastructure around inference budgets, context management, and tool orchestration. This is the most worthwhile technical direction to invest in at the current stage.