Trillion-Parameter Open-Weight Models: Technical Breakthroughs and the Shift in Agent Evaluation

From Preference Voting to Causal Tracking: Rethinking Agent Evaluation

For years, large language model evaluation has relied on pairwise blind tests, where users vote on which of two model responses they prefer and win rates are computed from the results. This paradigm works reasonably well for single-turn Q&A scenarios, but it breaks down in complex agentic tasks. A user’s actual intent often only becomes clear through multiple rounds of interaction, tool calls, and error correction. A binary judgment of “which answer is better” cannot capture the dynamic corrections, user frustration, and silent failures that occur during task execution.

A different approach has emerged in current agent evaluation: signal extraction and causal inference from real user sessions. Instead of asking users to perform any additional annotation work, this method mines four key signals from naturally occurring agent interaction logs: explicit user confirmation of task completion (such as “done” or “all set”), praise or complaints about model behavior, corrective instructions when the model heads in the wrong direction, and error codes returned by shell command execution. These signals are fed into a statistical inference framework known as randomized causal tracking. By borrowing design principles from randomized controlled experiments, the framework estimates a “net improvement rate” for each model relative to the average of all evaluated models. A positive net improvement rate means the model performs above average in real tasks; a negative value means the actual experience falls below the mean.

The key advance of this evaluation logic is that it eliminates both the subjective bias and the scaling bottleneck of human judges. Traditional blind testing requires large numbers of annotators to score responses one by one, which is expensive and hard to scale. The new paradigm can directly reuse massive volumes of real user sessions; as long as the user base is large enough, evaluation scale grows naturally with usage. More importantly, it focuses on whether a task is ultimately completed successfully rather than which response sounds more polished, which aligns directly with the core value of agentic products.

Benchmark Results Across 17,500 Real Tasks

A large-scale open-weight mixture-of-experts model recently completed a new round of evaluation. Based on statistics from more than 17,500 real long-horizon agentic task sessions accumulated by global users, the model achieved a net improvement rate of +5.1%, up 4.2 percentage points from +0.9% for its previous generation. This is a substantial jump in this type of evaluation, indicating a qualitative change in the model’s overall execution capability on complex tasks.

The sub-metrics reveal where the improvement came from. On the dimension measuring shell command execution reliability, the model jumped from -4.5% (ranked 13th) in the previous generation to +7.9%, placing first among all open-weight participating models. This metric reflects the model’s ability to parse user intent in a terminal environment, generate correct commands, and handle unexpected output — a litmus test for whether an agent can handle real-world work such as system administration and data pipeline operations. The user satisfaction signal (the ratio difference between praise and complaints) improved from +1.3% to +7.7%, suggesting the new model no longer merely “makes fewer mistakes” but proactively delivers results that exceed user expectations. Another metric called “steerability” measures whether the model can flexibly adjust its execution strategy mid-task when the user changes direction or introduces new constraints. The model scored +6.5% on this dimension, also ranking first among open-weight models. It is worth noting that explicit user confirmation of task completion came in at +3.9%, ranking fourth among open models. This indicates that on certain task subtypes, other models still outperform it; it is not an across-the-board champion.

On a separate agent benchmark focused on terminal operations, the model achieved a score of 65.7%, ahead of a recently released general-purpose flagship model at 58.2%, and within less than a percentage point of a commercial model known for long-horizon task capabilities at 66.4%. Taken together, these data points suggest that open-weight models are beginning to approach, and in some cases surpass, closed-source flagships in agentic capability.

Architecture Deep Dive: A 2.8-Trillion-Parameter Mixture-of-Experts Design

The model adopts a mixture-of-experts (MoE) architecture with a total of 2.8 trillion parameters, activating only about 104 billion parameters per forward pass. The core idea is to decouple model capacity from inference cost. In theory, more parameters allow the model to store richer knowledge and patterns. However, if every inference step required engaging all parameters, compute cost would scale linearly with parameter count and eventually become even more prohibitive than training cost. MoE addresses this through sparse activation: each input token activates only a small subset of expert networks, maintaining large parameter capacity while keeping per-token inference overhead within a reasonable range.

Compared with the previous base version, the new release makes two key structural adjustments. First, some full attention layers are replaced with gated sparse attention layers. Full attention computes attention over all historical tokens, with complexity growing quadratically with sequence length. Sparse attention layers use a gating mechanism to selectively attend to only a subset of critical tokens, significantly reducing compute and memory overhead in long-context scenarios. Second, the model retains a context window of approximately one million tokens, which is essential for processing very large code repositories, complete technical documentation, or multi-turn conversations spanning extended periods.

Post-Training Strategy: Large-Scale Joint Reinforcement Learning on a Strong Base

The training path is worth examining. The model was not trained from scratch. Instead, it used the base version as a foundation and underwent large-scale joint reinforcement learning post-training covering hundreds of downstream domains. The success of this strategy rests on the base model’s already broad knowledge and reasoning capabilities. The post-training phase aims to reshape behavior rather than learn from nothing.

The key to joint reinforcement learning lies in reward signal design. Unlike conventional reinforcement learning from human feedback, joint reinforcement learning often collects feedback from real tool-execution environments, allowing the model to naturally receive positive and negative signals while writing code, manipulating files, and running commands. This training approach mirrors the evaluation paradigm described earlier: both training and evaluation are grounded in real task scenarios rather than static Q&A pairs. Multi-domain joint training also enables capability transfer across downstream tasks; for example, error-tracking skills learned in code execution can be reused in database operations or web request debugging.

The Gap Between Open Weights and Local Deployment

From an academic and developer ecosystem perspective, open weights are unquestionably valuable. But “downloadable weights” does not mean “runnable locally.” There is a significant engineering gap between the two. For a 2.8-trillion-parameter model stored in BF16 precision, the weight files total roughly 5.6 TB. Even with INT4 quantization, which shrinks the size to a quarter, approximately 1.4 TB of storage is still required. This does not account for runtime KV cache, optimizer state, and inference framework overhead. A workstation with four high-end consumer GPUs (24 GB VRAM each) provides only 96 GB of total VRAM. Loading this model at INT4 precision would require roughly 15 times that capacity on a single machine. As a result, what most developers can actually do in a local environment is run a heavily distilled small version, or call cloud compute via API for inference.

This reality underscores that open weights are currently more relevant to research institutions or large enterprises with multi-node GPU clusters. The value lies in fine-tuning autonomy, data privacy, and freedom from API fees — but all of these benefits presuppose sufficient infrastructure investment. For individual developers and small to medium-sized businesses, the pragmatic path remains using cloud-hosted inference services rather than insisting on local deployment.

Terminal Agents in Context

Terminal agents represent one of the most active application areas for large models today — letting models directly operate command lines, IDEs, databases, or operations dashboards, turning suggestions into execution. The evaluations above show that this model has achieved strong results on terminal operation benchmarks, though a slight gap remains with its closest commercial closed-source counterpart. That gap is a reminder that in high-value, low-fault-tolerance tasks like terminal agents, small differences in capability rankings can translate into significant differences in user experience. A single incorrect shell command can cause data loss, while one successful automatic error recovery can save critical business operations.

As mainstream models increasingly embrace complex tool use and long-horizon task execution, agent capability evaluation will inevitably move toward more fine-grained, task-oriented metrics. Regardless of how evaluation paradigms evolve, the real test remains the same: can a model reliably get things done, and done well, without human supervision.