Ultra-Large MoE Model Open Weights: Architecture, Agent Benchmarks, and Deployment Realities

Recently, a Mixture-of-Experts (MoE) model with a total parameter count of 2.8 trillion was officially released with open weights. As promised during its initial announcement, the weight files are now available on mainstream model hosting platforms. Alongside this release, the model secured the top spot among open-weight models—and 13th overall—on an agent leaderboard operated by an independent evaluation organization. It also ranked first among open-weight models in the work, coding, and chat categories. This article breaks down and analyzes this progress from four technical dimensions: architecture design, evaluation methodology, granular metrics, and deployment realities.

1. Architecture: Trade-offs Between Sparse Activation and Long Context

The model utilizes a Mixture-of-Experts architecture with a total of 2.8 trillion parameters, activating approximately 104 billion parameters per inference pass, and supports a context window of around 1 million tokens. This combination reflects a clear engineering orientation: using a massive total parameter count to store knowledge capacity, while leveraging sparse activation to control the computational overhead of a single inference pass.

Compared to its predecessor, the new generation model underwent large-scale joint reinforcement learning (RL) post-training on the base model, covering hundreds of downstream domains. Structurally, a key adjustment was made: replacing some full-attention layers with gated sparse attention layers to reduce the overhead of long-context modeling.

The motivation behind this change is worth examining. Standard full attention scales quadratically in computational and memory overhead as sequence length increases, placing extreme pressure on 1-million-token contexts. Gated sparse attention dynamically selects relevant key-value subsets for each query, restricting attention computation to sparse connections. This preserves the ability to model long-range dependencies while reducing overhead. The introduction of a gating mechanism allows the model to adaptively determine the sparsity level based on input content, rather than relying on a fixed sparsity pattern. For agent scenarios that require processing long documents, multi-turn tool-calling trajectories, or large codebases, this structural optimization directly determines whether the context window can be fully utilized in real-world tasks.

The significance of the joint RL post-training covering hundreds of downstream domains lies in enabling the model to adapt to multiple task distributions under a unified policy, rather than narrow fine-tuning for a single task. For agentic tasks, this means the model must perform planning, tool calling, error recovery, and natural language interaction within a single session. Joint training helps reduce capability fragmentation across different tasks.

2. Evaluation Methodology: From Pairwise Blind Tests to Causal Tracing

Understanding the model’s ranking requires understanding how this evaluation differs from traditional methods. Conventional leaderboards typically use pairwise blind testing: users are presented with two anonymous responses and vote for the better one, which is then aggregated into a win rate or Elo score. While this paradigm works well for chat scenarios, it has clear limitations for agent tasks. The value of an agent is not reflected in the phrasing of a single-turn response, but in whether it can complete the task, correctly call tools, and recover from errors during a multi-step execution process.

This leaderboard extracts signals from real user agent sessions, including:

• Explicit task confirmations from users;
• User praise and complaints;
• User correction commands;
• Runtime signals such as shell command execution errors.

Using this data, the evaluator estimates the net improvement rate of each model relative to the average level through a randomized inference method called “causal tracing.” The net improvement rate is a relative metric; a positive value indicates that the model delivers better results under an equivalent task distribution. The advantage of this method is that signals come from real usage behavior rather than manually annotated preferences. The downside is that the signal noise is higher and heavily depends on session quality, requiring a sufficiently large sample size for support.

This ranking is based on over 17,500 real, long-cycle agent task sessions from global users. Long-cycle tasks are a critical stress test for agent capabilities: they involve many steps, large state spaces, and early errors that get amplified in subsequent steps, making them better at distinguishing true gaps in planning and execution. On this sample base, the model achieved a net improvement rate of +5.1%, a 4.2 percentage point increase from the +0.9% of the previous generation with a similar configuration. This magnitude of improvement represents a significant leap in a relative metric system, demonstrating that the post-training and structural optimizations yielded measurable returns on real task distributions.

3. Granular Metrics: Where Capabilities Improved

An overall net improvement rate only indicates that a model is “better,” while granular metrics show exactly “where it is better.” The recently published granular data reveals a specific improvement structure:

Bash Reliability: Increased from -4.5% (13th place) in the previous generation to +7.9%, ranking first among open-weight models. This transition to a positive value is particularly critical. Command-line execution is a core action for terminal agents; a single erroneous command can corrupt the workspace state or cause irreversible side effects. Shifting from a negative value to nearly +8% means a significant drop in the model’s error rate for command syntax, parameter selection, and pre-execution validation.

User Satisfaction (Praise vs Complaint): Increased from +1.3% to +7.7%. This metric measures the net difference between praise and complaint signals, reflecting the overall acceptance level of the model at the interaction level.

Steerability: Reached +6.5%, ranking first among open-weight models. This metric measures whether the model can adjust its execution direction in a timely manner based on user feedback. In multi-step tasks, if mid-task corrections from users cannot be effectively absorbed, the task can easily continue down the wrong path. Therefore, this capability is directly tied to the success rate of long-session tasks.

Confirmed Success: +3.9%, ranking fourth among open-weight models. This metric only counts sessions where the user explicitly confirmed task completion. Because its criteria are the strictest, it is normal for the value to be lower than other metrics.

From this data set, it is evident that the most significant improvements are in execution reliability and steerability, rather than just conversational quality. This aligns with the aforementioned joint RL post-training and attention structure optimization directions: the training focus was placed on “getting the task done correctly and stably.”

4. Open Weights Does Not Mean Locally Runnable

The release of open weights is a commendable move, but developers must clearly recognize the distance between “downloadable” and “locally runnable.”

Storing 2.8 trillion parameters in BF16 precision requires weight files of approximately 5.6TB; even with INT4 quantization, it still requires about 1.4TB. This magnitude means deployment is essentially impossible on a single consumer-grade GPU. Practically feasible paths generally fall into three categories:

1. Multi-node multi-GPU distributed inference: Requires high-bandwidth interconnects and supporting parallelism strategies (tensor parallelism, pipeline parallelism, expert parallelism), resulting in high engineering complexity.
2. Small-scale cluster deployment post-quantization: INT4 or lower precision can significantly compress memory footprint, but the actual impact of quantization on long-context and tool-calling accuracy must be evaluated.
3. Managed inference services: Calling upon general model inference services to avoid the operational costs of building a proprietary cluster.

For most teams, the third path is more realistic in terms of cost and stability. For scenarios with strict data compliance requirements or the need for deep customization, the first two paths warrant discussion. It is important to emphasize that distributed inference for MoE architectures has its own specific challenges: expert parallelism requires requests to be routed efficiently among experts, and load imbalances will directly drag down throughput, placing additional demands on the scheduling system.

5. Horizontal Position on Terminal Task Benchmarks

On the terminal agent benchmark Terminal-Bench 4.0, the model scored 65.7%. For reference, on the same benchmark, other models achieved scores of 58.2% and 66.4%, with the latter differing by less than 1 percentage point from the open-weight model discussed here. This indicates that in the specific capability of terminal operations, open-weight models have entered a range closely matching closed-source frontier models, with the gap compressed to a single-digit percentage level.

It is necessary to view this result rationally: a single benchmark score is heavily influenced by task set composition, evaluation configuration, and prompt engineering, and consistency across different benchmarks is limited. True capability assessment should combine multiple benchmarks, real session data, and validation within your own business scenarios, rather than relying on a single ranking.

6. Practical Advice for Developers

Based on the above analysis, here are several actionable recommendations:

• Validate before investing: Before deciding to build your own inference cluster, conduct small-sample validations using your target business’s long-cycle task sets. Focus on observing tool-calling success rates and error recovery capabilities, rather than just general conversational quality.
• Monitor behavioral drift post-quantization: The impact of INT4 quantization on long-context tasks is not uniform. It is recommended to run the same set of terminal tasks before and after quantization to compare changes in Bash execution error rates and task completion rates.
• Incorporate steerability into evaluation: Whether mid-task correction commands from users can be absorbed is key to distinguishing between a “functional” and a “highly usable” agent. Explicitly design test cases involving mid-task interventions in your internal evaluations.
• Prioritize execution safety boundaries: As command-line execution capabilities improve, it is necessary to enforce whitelist constraints and sandbox isolation on the range of commands the agent can execute, preventing the spread of accidental operations.

Conclusion

The release of open weights for this 2.8-trillion-parameter MoE model provides a high-capacity foundation for research and application. Gated sparse attention and large-scale joint RL post-training address two core bottlenecks in agent deployment: long context and multi-task adaptation. Meanwhile, the causal tracing evaluation based on real session signals offers a perspective on measuring agent capabilities that differs from standard pairwise blind tests. At the same time, the terabyte-scale weight size serves as a reminder that open weights solve the problem of “availability,” not “deployability.” For developers, rationally evaluating your own computational resources and task requirements to choose a matching deployment path is of far more practical value than chasing leaderboard rankings.