A New Benchmark for Open-Source LLMs: How a 2.8T Sparse MoE Architecture Powers Long-Horizon Agents

Long-horizon agents are becoming the main battleground for large language model (LLM) deployment. Unlike single-turn Q&A, agent tasks often require dozens or even hundreds of steps involving tool calls, state management, and self-correction. This places stringent demands on context length, reasoning stability, and per-token cost simultaneously. Recently, a newly released open-weight sparse Mixture-of-Experts (MoE) model—featuring 2.8T total parameters, 104B activated parameters, a 1M token context window, and native image-text inputs—has provided an engineering sample worth studying. This article analyzes the model across four dimensions: architecture, training, capabilities, and deployment, offering actionable practical recommendations.

1. The MoE Parameter Math: What 2.8T Total Parameters Actually Means

To understand this architecture, we must first clarify the distinction between “total parameters” and “activated parameters.” The 2.8T total parameters describe the overall scale of the model’s weight matrices, while the 104B activated parameters represent the actual number of parameters involved in the forward computation for a single token. The ratio of roughly 27:1 is a classic fine-grained sparse MoE design.

The core benefit of this design is the decoupling of “knowledge capacity” and “inference cost”:

• Capacity: A larger total parameter count allows the model to store more domain knowledge and patterns, providing better coverage for long-tail tasks, multilingual capabilities, and specialized terminology.
• Cost: The computation per token is determined by the activated parameters. The 104B activation scale keeps inference throughput and VRAM usage within acceptable limits.
• Routing: MoE relies on a gating network to select experts for each token. Routing quality directly determines whether the benefits of sparsification can be realized, making load-balancing strategies during training critical.

It is worth noting that total parameters do not equate to “effective capability.” Whether experts are fully trained, whether routing is stable, and whether knowledge is redundant across experts will all impact final performance. When evaluating such models, focus closely on the activated parameter scale, the number and granularity of experts, and whether routing introduces noticeable token dropping.

2. Attention Layer Overhaul: From Gated MLA to Gated DSA

The first bottleneck in long-context processing lies in attention computation. Standard full attention incurs computational and memory overheads that scale quadratically with sequence length N, making it unfeasible for a 1M token scenario. The model’s approach to the attention layer is particularly noteworthy.

According to public information, the new version replaces the original 24-layer Gated Multi-head Latent Attention (Gated MLA) with Gated Dynamic Sparse Attention (Gated DSA), combined with 69 layers of Kimi Delta Attention. The key mechanism of DSA is that each attention computation only selects 2,048 historical tokens to participate in the calculation, rather than applying weights to the entire history.

This design brings three major implications:

1. Controllable computational complexity: Regardless of the context length, the candidate set size for a single attention step remains fixed, resulting in more predictable latency.
2. Selection strategy is critical: How the 2,048 tokens are chosen—whether based on similarity, block indexing, or learned retrieval—directly determines whether long-range dependencies are preserved.
3. Complementary to linear attention: Delta Attention-style structures excel at maintaining state memory with linear overhead, while sparse attention handles precise retrieval of key fragments. Layering them together is a pragmatic compromise.

The takeaway for developers is that long-context capability cannot be judged solely by “how many tokens are supported”; retrieval precision matters just as much. In practice, it is recommended to construct “needle in a haystack” tests, placing critical information at various depths and positions to observe whether the model can consistently recall it, rather than just verifying the beginning and end of the context.

3. Training Paradigm: Joint Reinforcement Learning and Quantization

The model undergoes large-scale joint reinforcement learning (RL) post-training based on its previous foundational model, covering hundreds of downstream domains. This trajectory reflects a broader trend: general pre-training establishes foundational capabilities, while the multi-step decision-making skills required for agents are largely derived from RL and interaction with tool environments.

The key challenges of joint RL lie in reward design and task proportioning:

• Task diversity: The optimal strategies for terminal operations, code repair, CAD generation, web retrieval, and document processing differ vastly. Rational sampling weights are needed to prevent one category of tasks from dominating the gradient.
• Sparse reward problem: Multi-step tasks often only provide success/failure signals at the end. This requires process rewards, curriculum learning, or better credit assignment mechanisms.
• Environment stability: Tool-call-based training relies on reproducible execution environments. Environmental jitter will introduce noisy rewards.

The weights are released using the MXFP4 quantization format. The advantage of FP4 is that it significantly compresses storage and bandwidth requirements, making multi-GPU deployment of a 2.8T-scale model more realistic. The trade-off is that quantization errors may impact numerically sensitive tasks (such as precise calculations or long-chain reasoning). A practical recommendation is to prioritize validating the quantized version’s performance on target tasks. If significant degradation occurs, consider using higher-precision weights or de-quantizing critical layers.

4. Three-Tier Thinking Budget: Delegating Inference Cost to the Caller

The model provides three tiers of thinking budgets—low, high, and max—allowing developers to adjust reasoning depth based on task complexity. This is a highly engineering-oriented design.

• Low tier: Suitable for deterministic tasks like classification, extraction, and format conversion, prioritizing low latency and high throughput.
• High tier: The default choice for routine agent steps, balancing quality and cost.
• Max tier: Used for complex planning, multi-step debugging, and long-horizon dependency reasoning. This is also the default tier for official evaluations.

It must be emphasized that official benchmark scores default to the max tier, and some evaluations even leverage external agent frameworks. This means the published results reflect the “upper bound under optimal configuration” rather than the average performance under default calls. In production environments, using the max tier across the board will significantly increase costs and latency. A sound strategy is to stratify by task: use the low tier for entry routing and intent classification, the high tier for core steps, and reserve the max tier only for high-value, complex tasks.

5. Capability Coverage: From Code Agents to Knowledge Work

The model’s capability layout can be inferred from its public evaluation benchmarks:

• Terminal operations and software engineering: Terminal tasks, code repair, and long-horizon software engineering tasks assess multi-step execution, error recovery, and repository-level comprehension.
• CAD and 3D generation: High scores in tool-augmented CAD tasks indicate targeted optimization in structured geometric generation and tool synergy.
• Web retrieval and research: Outstanding retrieval evaluation scores reflect the model’s capability in information filtering and multi-source integration.
• Office document processing: Document understanding and Q&A tasks correspond to knowledge work scenarios.

These capabilities collectively point to a specific positioning: rather than acting merely as a conversational assistant, the model is designed as an agent foundation oriented towards “tool calling + multi-step execution.” Consequently, the usage paradigm should shift from “one prompt, one answer” to “defining goals, providing tools, and managing state.”

6. Deployment and Integration Practices

The open weights support mainstream inference frameworks and provide OpenAI- and Anthropic-compatible API formats, reducing integration overhead. During deployment, pay special attention to the following points:

First, preserve full history for multi-turn tool calls. This includes historical reasoning content and tool call messages. If reasoning fragments are truncated when assembling requests, the model will lose its decision-making basis, manifesting as repeated steps or incorrect tool parameters. It is recommended to persist reasoning blocks as first-class citizens in session state management.

Second, VRAM and parallelism strategies. Even with FP4 quantization, the 2.8T total parameters still require multi-GPU deployment. A combination of tensor parallelism and expert parallelism is recommended, paying close attention to the all-to-all communication overhead introduced by expert routing. In long-context scenarios, KV Cache management is also a major factor; standard techniques like PagedAttention should be utilized to control VRAM growth.

Third, context budget allocation. One million tokens is an upper limit, not a target. In practice, proactive context engineering should be applied: front-load stable system instructions and tool definitions, append volatile observation results to the end, and summarize historical data to prevent invalid tokens from diluting attention.

Fourth, license compliance. The weights are released under a specialized license rather than a standard open-source protocol. Before commercial use, carefully review the licensing scope, attribution requirements, and redistribution conditions clause by clause to avoid compliance risks.

7. How to Correctly Interpret Evaluation Data

Public scores cover multiple metrics, including terminal tasks, code repair, long-horizon software engineering, web retrieval, CAD generation, and office Q&A. When interpreting these results, keep three points in mind:

1. Tier and framework variances: Scores achieved using the max tier with external agent frameworks cannot be directly equated to the performance of the base model under default calls.
2. Task construction variances: Different evaluations use varying environments, toolsets, and scoring criteria. Cross-leaderboard horizontal comparisons hold limited significance.
3. Selective disclosure risks: Publicly available metrics highlight the model’s strengths. The unreported areas are equally important. It is advisable to conduct small-scale sampling evaluations based on your own business data distribution before deciding to adopt the model.

Conclusion

The value of this model lies not in a single benchmark score, but in how it combines the paths of “ultra-large sparse capacity, long-context sparse attention, RL post-training, and low-precision quantization release” into a reproducible engineering solution. For developers, the real task is not chasing parameter counts, but rather executing solidly around their own task distributions—optimizing tier scheduling, context engineering, state management, and cost control. The model provides the foundation, but the engineering quality of the agent system is what ultimately determines deployment success.