Trillion-Parameter Open-Weight Models: Agent Capability Leap and Deployment Practices

1. Background: Agent Evaluation Is Moving from Preference Votes to Behavioral Signals

Over the past two years, the dominant form of LLM leaderboards has been pairwise blind testing: users are shown two responses, asked to pick the better one, and relative rankings are fitted through statistical models. This approach works well for general chat scenarios, but it has gradually exposed weaknesses when applied to long-horizon agent tasks. The value of an agent does not lie in “which response sounds more polished,” but in whether it can execute reliably across dozens of tool calls, self-correct after errors, and change direction promptly when the user intervenes. These capabilities are difficult to capture through a single preference judgment on one response.

As a result, a new evaluation paradigm has begun to emerge. Instead of asking users for additional votes, it extracts “behavioral signals” directly from real sessions, including explicit user confirmation that a task is complete, proactive praise and complaints, corrective instructions, and command-level execution errors. Randomized inference methods (often referred to in the industry as causal tracing) are then used to estimate the net improvement rate each model contributes relative to the average. The advantage of this approach is that signals come from real usage traces, samples are closer to production environments, and scores are harder to game with “people-pleasing” answers.

Against this backdrop, a new version of an open-weight model achieved first place among open models and thirteenth place overall on its agent leaderboard, and ranked first among open models in all three categories: work, coding, and chat. Its ranking is based on more than 17,500 real long-horizon agent sessions from users worldwide. Its net improvement rate rose from +0.9% in the previous generation to +5.1%, an increase of 4.2 percentage points. These figures deserve a closer look.

2. Sub-metrics: Gains Are Concentrated in Execution Reliability, Not Expression Quality

The overall net improvement rate is only the conclusion; the sub-metrics reveal where the gains come from. According to public data, the most significant improvement is in the command execution reliability metric, which jumped from -4.5% (13th place) in the previous generation to +7.9%, ranking first among open models. This metric measures the success rate and fault tolerance of an agent executing commands in a shell environment, directly corresponding to the hard indicator of “whether the task can run through.”

The other sub-metrics show a consistent trend:

• User satisfaction (praise minus complaints): rose from +1.3% to +7.7%, indicating a clear increase in the proportion of proactive positive feedback from users;
• Steerability (adjusting execution direction based on feedback): reached +6.5%, ranking first among open models, reflecting the model’s compliance quality in multi-turn corrections;
• Explicit confirmation of completion rate: +3.9%, fourth among open models, indicating that task closure capability improved steadily but remains a relative weakness.

The combined meaning of these metrics is clear: the new version’s gains come primarily from “getting things done and done right,” not from “sounding good.” For agent scenarios, this is precisely the most valuable direction of improvement. In real engineering, a single command execution failure often means the entire task chain is interrupted, and the cost is far higher than a response that is not sufficiently fluent.

From an engineering perspective, there is a coupling relationship between execution metrics and steerability metrics. Even if a model has a high single-step execution accuracy, if it continues down the wrong path after receiving a user’s corrective instruction, the overall task success rate will still drop significantly. Therefore, the improvement in steerability from +1.3% to +6.5% likely shares the same post-training mechanism as the improvement in execution reliability—namely, using reinforcement learning to teach the model to “treat user feedback as a new constraint and replan,” rather than simply repeating the previous turn’s actions.

3. Architecture Analysis: 2.8 Trillion Parameters and the Trade-offs of Sparsity

The model uses a Mixture-of-Experts (MoE) architecture with 2.8 trillion total parameters, approximately 104 billion activated parameters per inference, and a context window of about 1 million tokens. Behind these numbers is a typical large-scale sparsity design logic: total parameter count determines the model’s knowledge capacity and upper bound of expressiveness, activated parameter count determines the compute cost of a single inference, and the ratio between the two determines sparsity efficiency.

With 2.8 trillion total parameters and about 104 billion activated parameters, only about 3.7% of parameters are activated per inference. This sparsity ratio means the model can maintain an extremely large capacity pool while keeping per-token compute close to the level of a hundred-billion-scale dense model. For agent scenarios, this design is particularly well suited: agent tasks often require frequent tool calls and repeated reading of long contexts, with many inference passes but short outputs per pass. The lower the activated parameters, the more controllable the inference cost per unit of task.

A context window of about 1 million tokens directly corresponds to the memory requirements of long-horizon tasks. A complete agent session may contain dozens of tool calls, large amounts of command output, and intermediate results. If the context window is insufficient, the model has to perform lossy compression, and early key information is easily lost. A million-token window makes it possible to “retain the original trace throughout,” but it also introduces new overhead problems.

To this end, the new version made two key modifications on top of the previous-generation base:

1. Large-scale joint reinforcement learning post-training: covering hundreds of downstream domains, directly incorporating behavioral objectives such as execution reliability and correction responsiveness into optimization, rather than optimizing only the language modeling loss;
2. Hybrid attention layer replacement: replacing some full-attention layers with gated sparse attention layers to reduce the compute and memory overhead of long-context modeling.

The second modification deserves elaboration. The computational complexity of full attention grows quadratically with sequence length, and the overhead is extremely significant at a million-token window. Gated sparse attention dynamically selects the key-value range each query participates in, compressing effective computation to near-linear while using a gating mechanism to preserve access to critical long-distance information. The engineering difficulty of this kind of modification lies in the fact that the choice of sparsity pattern must match the task distribution. In agent traces, key information tends to be concentrated in the most recent tool return results and the early task objectives, showing a “heavy at both ends, light in the middle” distribution. The sparsity strategy needs to be tuned for this characteristic; otherwise, the result is “saving compute but losing memory.”

4. The Real Distance of Open Weights: Downloadable Does Not Mean Locally Runnable

“Open weights” is often misread as “freely deployable.” From an engineering perspective, there is a hard threshold formed by memory and bandwidth between the two.

At 2.8 trillion parameters, the weight file is about 5.6 TB in BF16 precision; even with INT4 quantization, it still requires about 1.4 TB. This magnitude means single-machine consumer GPUs are essentially not viable for deployment—not only is memory capacity insufficient, but the storage bandwidth and interconnect bandwidth required for weight loading far exceed single-machine motherboard capabilities. Practical deployment forms typically include:

• Multi-machine, multi-GPU distributed inference: splitting weights across dozens of accelerators through tensor parallelism and pipeline parallelism, requiring high-speed interconnect support;
• Quantization plus sparse offloading: offloading cold experts to host memory or remote storage and swapping them in on demand, at the cost of increased latency;
• Cloud-hosted invocation: using the model via API to avoid the hardware investment of local deployment.

The table below gives a rough memory estimation approach (for magnitude illustration only, not specific product parameters):

Precision	Weight Volume Magnitude	Deployment Form
BF16	About 5.6 TB	Multi-machine, multi-GPU, high-speed interconnect required
INT8	About 2.8 TB	Multi-machine, multi-GPU, slightly lower interconnect bandwidth requirements
INT4	About 1.4 TB	Multi-machine, multi-GPU, or hybrid quantization + offloading

It should be emphasized that the memory footprint of an MoE architecture is not equal to “activated parameter count × precision bytes.” All expert weights must either reside in memory or be predictably swapped in; otherwise, routing to an unloaded expert triggers severe latency jitter. Therefore, when planning capacity, the memory lower bound should be estimated based on total parameter count rather than activated parameter count, and then KV cache and intermediate activation usage should be added. For million-level contexts, the KV cache itself may reach hundreds of GB, an overhead that is often underestimated.

Another frequently overlooked dimension is the economics of inference throughput. MoE models can improve per-unit-compute output through expert parallelism when batch sizes are large, but in small-batch, low-concurrency scenarios, imbalanced expert routing load causes some accelerators to sit idle. Agent tasks are precisely typical low-concurrency, multi-turn workloads, which requires deployment teams to perform request aggregation and expert affinity optimization at the scheduling layer; otherwise, actual throughput will be significantly lower than the theoretical value.

5. Benchmark Performance and Selection Implications

On a terminal agent benchmark, the model scored 65.7%, higher than one competitor’s 58.2% and within less than 1 percentage point of another competitor’s 66.4%. This result indicates that open-weight models have approached the level of closed-source leading models on terminal-operation tasks, with the gap entering the single-digit percentage range.

For developers and enterprises, three points deserve attention when selecting models:

1. Look at sub-metrics, not just the total score: models with similar overall rankings may differ greatly in execution reliability and steerability, and evaluation should be weighted according to your own task types;
2. Distinguish “capability achievable” from “cost achievable”: leaderboard results are typically obtained under specific inference configurations, and the quantization level, context length, and concurrency scale of actual deployment will all affect performance;
3. Pay attention to the suitability of the evaluation method itself: evaluations based on real-session behavioral signals are closer to production, but whether the sample distribution matches your own business distribution still needs to be verified independently.

To reproduce a similar behavioral-signal evaluation, you can collect logs in the following dimensions within your own business (illustrative pseudocode):

def extract_agent_signals(session):
    signals = {}
    signals["confirmed"] = session.has_explicit_confirmation()
    signals["praise"] = session.count_positive_feedback()
    signals["complaint"] = session.count_negative_feedback()
    signals["corrections"] = session.count_redirect_instructions()
    signals["exec_errors"] = session.count_command_failures()
    # Estimate relative improvement rate using randomized grouping to avoid confounders
    return signals

It should be noted that this type of inference method depends on sufficient sample size and reasonable randomization design. Net improvement rates fluctuate considerably under small samples and should not be used as the sole basis for decision-making.

6. Summary

The emergence of trillion-parameter open-weight models has pushed the ceiling of agent capabilities higher again, with the focus of improvement falling on execution reliability and steerability—highly consistent with the real pain points of agent deployment. But “open weights” does not automatically equal “deployable,” and the memory, bandwidth, and scheduling challenges brought by parameter scale remain problems that engineering teams must confront directly. A rational approach is to screen capabilities using sub-metrics, calculate costs using quantization and distributed solutions, and then perform final verification with your own business logs.