A 2.8T-Parameter Open Model in Agent Tasks: What the Net Improvement Rate Reveals

From Blind Voting to Causal Tracking: The Shift in Agent Evaluation

Traditional model capability evaluation has relied heavily on human preference voting between two responses. This approach is acceptable for single-turn dialogue, but it breaks down when agents execute multi-step tasks. Users do not care whether a single sentence sounds polished—they care whether the entire task pipeline can be correctly decomposed, executed, and brought to convergence. Major evaluation platforms have shifted toward extracting signals from real user sessions: explicit task completion confirmations, praise or complaints, corrective instructions, and errors during Shell command execution. These signals are processed through randomized inference methods to estimate a model’s net improvement rate relative to the average, producing rankings that better reflect actual usage experience.

The academic value of this approach is that it reframes “is the model smart?” as a quantifiable metric: “how much user frustration does the model reduce in real workflows?” A model might rank highly on knowledge QA leaderboards but show a negative net improvement rate when handling operations tasks that require repeatedly invoking command-line tools, parsing output, and dynamically adjusting strategy—a dimension traditional evaluation simply cannot surface.

Reading the Evaluation Data: What the Net Improvement Rate Shows

Consider a recently released open-weight 2.8-trillion-parameter mixture-of-experts model. In evaluations based on over 17,500 real long-horizon agent task sessions from global users, its net improvement rate reached +5.1%, a 4.2-percentage-point gain over the previous generation’s +0.9%. Breakdown metrics reveal where the improvements came from:

• Bash Reliability: Jumped from -4.5% (ranked 13th) in the previous generation to +7.9%, placing first among open-weight models. This metric measures the model’s ability to generate executable Shell commands and correctly handle execution results. A negative-to-positive shift means the model’s error rate in terminal interactions dropped substantially—a foundation for agents working in real server environments.
• Praise vs Complaint Ratio: Improved from +1.3% to +7.7%. This metric directly reflects how often users proactively express satisfaction after task completion. Compared to intermediate process metrics, it better captures overall experience improvement.
• Steerability: Reached +6.5%, also first among open models. This measures whether the model can quickly adjust subsequent actions when users change requirements mid-task or correct the execution path, rather than stubbornly persisting. In real production environments with ad-hoc requirement changes, steerability often matters more than single-task success rate.
• Confirmed Success: +3.9%, ranking fourth among open models. This metric requires explicit user confirmation that the task was achieved, making it one of the strictest data quality signals.

Notably, improvement magnitudes vary across metrics, suggesting the model was not uniformly enhanced but had specific weaknesses addressed through targeted training strategies. The large Bash reliability gain likely benefited from adding more terminal interaction error-correction samples to training data, while steerability improvement may come from strengthened contradiction-resolution mechanisms in multi-turn dialogue.

Architecture and Training Strategy: Sparse Attention and Layered Reinforcement Learning

The model uses a mixture-of-experts (MoE) architecture with 2.8 trillion total parameters but activates only about 104 billion parameters per inference. This design maintains the model’s capacity ceiling while controlling per-inference compute cost. Importantly, the model was not trained from scratch—it builds on an earlier dense model and undergoes large-scale joint reinforcement learning post-training. The training process covers hundreds of downstream domains and replaces some full-attention layers with gated sparse attention layers.

The introduction of gated sparse attention has engineering significance. In full attention, every token computes attention weights against all tokens in the sequence, with complexity growing quadratically with sequence length. Gated sparse attention uses a learnable routing mechanism to selectively activate key-value pairs for each query token, reducing long-context modeling complexity from quadratic to approximately linear. For a model supporting a context window of roughly 1 million tokens, this modification directly determines the economic viability of inference serving.

The large-scale joint reinforcement learning in post-training performs policy optimization across multiple downstream tasks simultaneously. Traditional reinforcement learning tends to overfit on single tasks, while joint training shares the policy network to transfer knowledge across tasks, balancing performance in coding, workflow management, and conversation. This training strategy aligns with the evaluation result of ranking first among open models in all three categories: work, coding, and chat.

Open Weights and Local Deployment: The 5.6TB Reality Gap

Open-sourcing model weights has positive implications for technology dissemination—researchers and enterprises can examine internal structure, fine-tune, or distill. But there is an order-of-magnitude gap between “downloadable” and “locally runnable.” Weights stored in BF16 precision occupy approximately 5.6TB. Even with INT4 quantization compressing this to about 1.4TB, it still far exceeds the VRAM capacity of single-machine consumer GPUs (mainstream products range from 24GB to 80GB). This means practical deployment typically involves:

• Multi-machine distributed inference: Splitting the model across multiple servers and coordinating computation through high-speed network communication. This requires professional cluster operations expertise and deep understanding of model parallelism strategies.
• API calls: Using model capabilities through cloud provider inference endpoints, which is the practical choice for most developers.
• Distillation and compression: Using the large model to generate training data and training a smaller, locally deployable student model.

For independent developers without infrastructure budgets, the third path may be more practical. However, distillation loses some capability, and the quality of distillation data directly determines the student model’s ceiling.

Terminal Agent Benchmark: A Ruler Close to Real Operations Scenarios

In a benchmark closer to terminal operation scenarios, the model scored 65.7%, higher than another well-known vendor’s flagship model at 58.2%, and within 1 percentage point of the best closed-source model (66.4%). The terminal agent benchmark requires the model to complete composite tasks in a simulated Linux terminal environment—file operations, process management, network requests, configuration changes—while correctly handling edge cases like insufficient permissions, non-existent commands, and abnormal output formats.

The value of such benchmarks is that they force models to demonstrate understanding of underlying operating system behavior, not just reproduce text patterns. For example, when executing rm -rf, can the model recognize the command’s irreversibility and refuse to execute without explicit user confirmation? When grep returns empty output, can the model infer possible encoding issues rather than simply retrying? These details are extremely difficult to cover in traditional evaluation but are exactly what determines whether automation tools can be trusted.

Implications for Developers: How Evaluation Metrics Guide Selection and Optimization

When facing evaluation data, developers need their own interpretation framework:

1. Distinguish “capability ceiling” from “stability”: Metrics like net improvement rate reflect average performance. In practice, combine them with your own task type. If your business involves heavy API calls and programmatic interaction, Bash reliability is more relevant than chat metrics.
2. Pay attention to steerability metrics: In real projects, requirement changes are the norm. A model with strong steerability reduces the number of rework conversation turns, directly impacting overall task completion time.
3. Quantify deployment costs: Open weights do not mean free deployment. Calculate the VRAM, bandwidth, and power costs required for inference, and compare them against API call expenses. For low-frequency tasks, API calls are usually more economical; for high-frequency, latency-sensitive scenarios, self-hosted inference may be more reasonable.

The model’s iteration demonstrates the viability of the open-weight path—more developers can fine-tune on this foundation, driving agent technology adoption in vertical industries. The evolution of evaluation methods also reminds us that the ultimate assessment standard for large models should return to real-world problem-solving efficiency, not a single score on a leaderboard. For developers exploring the boundaries of agent applications, understanding the signals behind these metrics is more meaningful than chasing parameter count numbers.