Sparse MoE Architecture Moves Forward: A 2.8T-Parameter Open-Weight Model Built for Long-Horizon Agents

In early November, a sparse Mixture-of-Experts (MoE) large language model with 2.8 trillion total parameters was released as open weights. Building on its predecessor with large-scale joint reinforcement learning post-training, the model uses a MoE architecture with 104 billion activated parameters and natively supports a 1-million-token context window with mixed image-text input.

For agent developers who have long been constrained by context window limits and inference costs, this release is notable not just for its parameter count, but for its systematic improvements in long-horizon task planning, tool-calling stability, and sparse attention engineering. This article analyzes the release from four angles: architecture, training, evaluation, and deployment.

Why Long-Horizon Agents Need a New Attention Mechanism

Long-horizon agent tasks—such as cross-file code repair, multi-step terminal operations, and iterative design work involving CAD tools—place two conflicting demands on a model:

• Context needs to be long enough: The agent must retain historical reasoning, tool-call results, and intermediate states, often spanning hundreds of thousands of tokens.
• Compute cost needs to stay manageable: Standard attention scales quadratically with sequence length, making million-token contexts nearly prohibitive.

The common workaround is to introduce sparse or blocked attention, which skips redundant computation while preserving key historical information. This release offers a fairly complete engineering solution in that direction.

From Gated MLA to Gated DSA

Compared with its predecessor, the new model replaces the original 24 layers of Gated MLA (multi-head latent attention) with Gated DSA (dynamic sparse attention), combined with 69 layers of Kimi Delta Attention.

The core logic of DSA is straightforward: at each attention step, the model dynamically selects the 2,048 most relevant tokens from the history to participate in computation, rather than running dense attention over the entire history. This design brings two immediate benefits:

• Attention computation drops significantly from the O(n²) regime, making a 1M-token context feasible in practice.
• A gating mechanism preserves stable selection of critical historical information, avoiding the information forgetting that can accompany aggressive sparsification.

For agent scenarios, this means the model can accurately trace back to early task goals and constraints after multiple rounds of tool calls, instead of gradually drifting off course in long conversations.

MoE Parameter Scale and Activation Efficiency

The model uses a 2.8T total parameter configuration with 104B activated parameters. This ratio means only about 3.7% of parameters are activated per forward pass, keeping per-inference compute costs relatively controlled while maintaining the knowledge capacity of a large model.

Coupled with MXFP4 weight quantization, memory footprint during deployment is further compressed, creating room for local and private deployment scenarios.

Joint Reinforcement Learning: From Writing Code to Completing Tasks

Parameter scale and attention mechanisms address the question of whether a model can handle long contexts. The core of agent capability, however, lies in whether it can turn long context into correct multi-step decisions. That depends on the reinforcement learning strategy used in post-training.

Joint Training Across Hundreds of Downstream Domains

The model underwent large-scale joint reinforcement learning post-training on top of the K3 base, with training signals covering hundreds of downstream domains. This joint strategy, rather than single-point optimization, aims to prevent the model from overfitting to one task category and developing capability gaps in real-world agent workflows.

Official evaluations cover the following task types:

• Terminal operations: Multi-step execution in real command-line environments.
• Code repair: Cross-file defect localization and patch generation.
• CAD generation: Structured 3D design tasks involving tool use.
• Web retrieval and knowledge work: Including financial research, office document processing, and tool calling.

Three Thinking Budgets: Letting Agents Allocate Compute on Demand

One easily overlooked but highly practical design is the availability of three thinking budgets: low, high, and max.

This allows developers to dynamically set inference overhead based on task complexity:

• For simple tool calls or format conversion, use low to reduce latency and cost.
• For regular code repair or document processing, use high.
• For complex multi-step agent planning and long-horizon CAD iteration, use max.

This tiered mechanism is closer to real production needs than a one-size-fits-all inference configuration. Not every request warrants maximum compute.

A Look at Official Benchmark Results

The officially published benchmark scores use the max thinking budget by default, with some evaluations using agent frameworks such as intelligent programming tools. Key results are shown below.

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

These results span terminal operations, software engineering, web retrieval, CAD generation, and office document processing. It is worth keeping in mind that benchmark scores still differ from real production performance, especially in multi-round tool-calling scenarios where framework design, history message management, and tool return formats can significantly affect final outcomes.

Deployment and Integration Notes

Quantization Format and Inference Frameworks

The model ships with MXFP4 weights, supports mainstream inference frameworks, and provides OpenAI/Anthropic-compatible APIs. This means existing agent applications built on compatible interfaces can switch models with relatively minor modifications.

Key Constraint for Multi-Round Tool Calling

One official note deserves special attention: multi-round tool calling requires preserving the full history of reasoning and tool-call messages.

This is easy to overlook in practice. Many agent frameworks trim history messages between turns to save tokens, keeping only final results. But for this model—and similar long-horizon agent models—the reasoning process and tool-call records are themselves part of the reasoning chain. If they are stripped out, the model may lose its sense of the overall task state by the Nth turn.

A suggested implementation pattern is shown below.

# Pseudocode: preserve the full agent trajectory
messages = []
# Initial user request
messages.append({"role": "user", "content": user_query})
while not task_done:
    # Model output (may include reasoning + tool calls)
    response = model.chat(messages, thinking_budget="max")
    messages.append(response)  # Keep the full response without trimming
    # Tool execution results
    if response.tool_calls:
        for call in response.tool_calls:
            result = execute_tool(call)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": result
            })

License Notes

The weights are released under a model-specific license, not Apache 2.0 or MIT. Teams should carefully review the license terms before commercial use and confirm that their intended use case complies. This is especially important for teams planning to integrate the model into commercial products.

Use Case Fit and Selection Guidance

Based on the model’s characteristics, several scenario types stand out as a good fit.

Long-horizon code agents: Cross-file refactoring, multi-round test-and-repair loops, and terminal automation. The long-context capability enabled by sparse attention offers an advantage in scenarios that require retaining large amounts of code context.

Knowledge work automation: Web retrieval, financial research, and office document processing. Native mixed image-text input means the model can directly handle documents containing charts and scanned content.

3D design and CAD generation: Multi-step structured design tasks involving tools. The BenchCAD with-tools score of 92.7 suggests targeted optimization in this direction.

Less suitable scenarios: For simple classification or extraction tasks that are highly latency-sensitive, the low thinking budget may be more appropriate, or a smaller specialized model may be worth considering.

Conclusion

2.8T parameters, 104B active parameters, a 1M-token context window, sparse attention, and joint reinforcement learning—behind these numbers is an engineering effort to move long-horizon agents from “capable of conversation” to “capable of completing tasks.” For developers, what matters is not any single metric, but whether the model provides flexible thinking budget control, supports complete agent trajectory preservation, and remains stable in real tool-calling chains.

Open weights mean private deployment and deep customization are possible, but the model-specific license also calls for careful commercial evaluation. Before formal integration, teams are advised to build a small-scale evaluation set from their own business scenarios to validate actual performance in multi-round tool calling.