Evaluating and Deploying Trillion-Parameter Open-Source Models for AI Agent Capabilities

1. Background: A New Era for Open Weights and Agent Evaluation

As the parameter scale of large language models continues to grow, the focus of the open-source community and the tech industry on “open weights” has shifted from mere download availability to more pragmatic concerns: How do these models perform in real-world tasks? Are deployment costs manageable? Recently, a research institution fulfilled its promise by open-sourcing the weights of its new-generation trillion-parameter mixture-of-experts (MoE) model on mainstream model hosting platforms. This model ranked first among open-weight models and 13th overall in a specialized AI agent leaderboard. It also secured the top spot among open-source models across three categories: work, coding, and chat.

The significance of this progress lies not just in the rankings, but in the trends revealed by its evaluation paradigm and engineering constraints: Agent capabilities are becoming a critical dimension for measuring the practical value of large models, and a significant gap remains between open weights being “accessible” and being “executable.”

2. Evaluation Mechanism: From Pairwise Blind Testing to Causal Tracing

Traditional model evaluations often rely on pairwise blind testing, requiring users to vote on their preference between two anonymous responses. This method depends heavily on human judgment, suffers from high sample bias, and struggles to capture real signals in long-horizon tasks. The agent leaderboard discussed here takes a different approach: instead of user votes, it extracts behavioral signals from real user agent sessions, including:

• Explicit user confirmation of task completion (e.g., “done” or “correct”)
• Praise and complaints
• Correction instructions (users asking the model to change its execution direction)
• Runtime feedback such as Shell command execution errors

Based on this, the platform uses a randomized inference method called “Causal Tracing” to estimate the net improvement rate of each model relative to the average level. The core advantage of this method is that the signals come from real-world usage scenarios rather than lab-constructed preference pairs. Multi-turn interactions in long-horizon tasks are fully incorporated into the evaluation, avoiding the limitations of single-turn Q&A benchmarks.

2.1 Understanding the Net Improvement Rate

The net improvement rate is not an absolute accuracy metric, but rather a gain relative to the average performance of participating models. A positive value indicates above-average performance, while a negative value indicates below-average performance. This metric allows for horizontal comparison of different models under the same task distribution and is sensitive to sample size—this ranking is based on over 17,500 real long-horizon agent task sessions from global users, providing statistical significance.

3. Key Metrics Analysis: From Execution Reliability to Steerability

The model’s net improvement rate stands at +5.1%, a 4.2 percentage point increase from the previous generation’s +0.9%. The sub-metrics reveal more specific improvements worth examining in detail.

3.1 Bash Reliability: Shell Command Execution Reliability

This measures the model’s reliability in executing Shell commands during agent sessions. The metric jumped from -4.5% (ranking 13th) in the previous generation to +7.9%, ranking first among open-source models. Shell command execution is a core capability for terminal agents: the model must correctly construct commands, handle paths and permissions, parse outputs, and decide the next step. A negative value means the model frequently makes errors in this phase, leading to task interruptions or requiring human intervention. Turning positive and showing a massive improvement indicates substantial progress in command generation and error recovery.

3.2 Praise vs Complaint: User Satisfaction Signals

This reflects the ratio of user praise to complaints regarding model performance. The metric improved from +1.3% to +7.7%. This change directly correlates with the subjective experience in real-world usage: users file fewer complaints and provide more positive feedback. In agent scenarios, satisfaction depends not only on the quality of a single-step response but also on whether the model maintains goal alignment and proactively clarifies ambiguities during multi-turn interactions.

3.3 Steerability: Controllability and Adaptability

This measures the model’s ability to adjust its execution direction based on user feedback. The model achieved +6.5%, also ranking first among open-source models. In long-horizon tasks, it is common for users to change requirements or correct the direction midway. A highly steerable model can quickly understand correction commands, abandon its original path, and re-plan, rather than stubbornly executing an outdated plan. This capability is particularly crucial for coding assistants and automated IT operations scenarios.

3.4 Confirmed Success: Explicit Task Completion Confirmation

The ratio of users explicitly confirming task completion stands at +3.9%, ranking fourth among open-source models. While this metric is relatively conservative, combined with the leading scores in Bash Reliability and Steerability, it indicates that the model performs stably in closing the loop of “completing the task and getting user confirmation.”

4. Architecture and Training: MoE + Sparse Attention

The model utilizes a mixture-of-experts (MoE) architecture with a total parameter count of 2.8 trillion. It activates approximately 104 billion parameters per inference and supports a context window of around 1 million tokens. Compared to its predecessor, the training strategy features two key adjustments:

1. Large-Scale Joint Reinforcement Learning Post-Training: Starting from the base model, joint reinforcement learning was applied across hundreds of downstream domains. This post-training approach allows the model to optimize across multiple task distributions simultaneously, rather than overfitting to a single benchmark.
2. Gated Sparse Attention Layers Replacing Some Full Attention Layers: The computational overhead of full attention grows quadratically with sequence length in long-context scenarios. Replacing certain layers with gated sparse attention reduces the overhead of long-context modeling while preserving attention precision at key positions.

These two adjustments jointly support the practicality of the 1-million-token context window—long context is not just a capacity metric, but requires a balance between inference cost and precision.

5. Deployment Reality: The VRAM Threshold of 2.8 Trillion Parameters

There is a significant distance between open weights being “downloadable” and “locally runnable.” Storing the 2.8 trillion parameter weights in BF16 precision requires roughly 5.6TB of storage; even with INT4 quantization, it still demands approximately 1.4TB. This means:

• Single machines with consumer-grade GPUs (typically 24GB–48GB VRAM) are practically incapable of deployment.
• Multi-GPU servers require numerous high-VRAM accelerators and must account for interconnect bandwidth and the inference framework’s parallelism strategy.
• Quantization introduces precision loss, requiring a trade-off between task performance and resource consumption.

For teams looking to deploy locally, viable paths include using multi-node distributed inference, adopting higher compression ratio quantization schemes (such as INT3/INT2, though performance degradation must be evaluated), or accessing the model via hosted API services. When selecting a path, it is recommended to first clarify the task’s requirements for context length, concurrency, and latency, and then reverse-engineer the necessary hardware configuration.

6. Terminal Agent Benchmark Performance

On the Terminal-Bench 4.0 benchmark, the model scored 65.7%, higher than the 58.2% achieved by a common industry solution A, and less than 1 percentage point behind another common industry solution B’s 66.4%. This result indicates that in terminal command execution and multi-step task planning scenarios, the open-weight trillion-parameter model is approaching the level of top closed-source solutions, with the gap falling within the statistical margin of error.

7. Practical Recommendations for Developers

1. Evaluate First: Do not just look at the overall ranking on the leaderboard. Focus on sub-metrics that match your specific scenario. For example, prioritize Bash Reliability and Steerability for coding assistants, and focus on Praise vs Complaint for customer service scenarios.
2. Calculate Deployment Costs: The VRAM requirements of a 2.8 trillion parameter model imply a high investment for building a self-hosted inference cluster. If the task volume is low, prioritize hosted APIs. If local deployment is mandatory, plan your quantization scheme and parallelism strategy in advance.
3. Long Context Usage: A 1-million-token context is suitable for scenarios like repository-level code understanding and long document analysis. However, be aware that inference latency and VRAM consumption increase as the context grows.
4. Post-Training and Fine-Tuning: Since the model has already undergone joint reinforcement learning post-training across hundreds of domains, when performing domain-specific fine-tuning on top of this, it is recommended to use a smaller learning rate to avoid breaking the existing alignment capabilities.

8. Conclusion

The open weights of trillion-parameter models signify that the competition in AI agent capabilities has entered a new phase. Evaluation methods are shifting from preference voting to real session signals, and metrics are being broken down from single accuracy scores into dimensions like execution reliability, satisfaction, and steerability. Meanwhile, deployment constraints have shifted from “can it be downloaded” to “can it be run.” For developers, understanding these changes is far more valuable than memorizing rankings: it dictates every step of model selection, cost estimation, and engineering implementation.