Trillion-Parameter Open Models Top the Agent Arena: Evaluation, Post-Training, and Deployment Analysis

1. Background: Open-Weight Models Enter the Agent Capability Competition Phase

As large language models evolve from general-purpose chatbots to terminal agents, the industry’s focus is shifting from “whether it can generate an answer” to “whether it can reliably complete real-world tasks.” For open-weight models, relying solely on knowledge Q&A scores on public leaderboards is no longer sufficient to reflect their engineering value. More critical metrics include the model’s ability to understand multi-turn user feedback, reliably execute terminal commands, maintain reasoning consistency in long sessions, and exhibit stable behavior switching across different downstream environments.

In this context, Agent Arena-style evaluations have become an essential reference for measuring a model’s practical utility. Instead of relying on static Q&A pairs, these evaluations extract feedback signals from real user-agent sessions, including explicit task confirmations, compliments, complaints, corrective instructions, and terminal command execution errors. By using causal tracing methods to estimate the model’s net improvement rate relative to the average, this type of evaluation provides a more direct reflection of the model’s behavioral performance in real-world scenarios.

Meanwhile, open-weight models face a common misconception: downloadable does not mean deployable. When the parameter scale reaches the trillion level, the weight size, VRAM requirements, quantization loss, and inference infrastructure costs become significant practical barriers. Therefore, to form a comprehensive judgment when analyzing these models, it is necessary to simultaneously evaluate their capabilities, training methods, structural design, and deployment conditions.

2. Evaluation Logic and Key Metrics of the Agent Arena

2.1 From Pairwise Blind Testing to Causal Tracing

Traditional model arenas typically employ pairwise blind testing: users see responses from two models simultaneously and vote for the better one. While effective for general conversational scenarios, this approach is not entirely suitable for agentic tasks. The reason is that agentic tasks usually involve multi-turn interactions, command execution, error recovery, and user corrections. Single-turn response quality often cannot represent overall task completion.

The Agent Arena adopts an evaluation approach more inclined towards extracting signals from real sessions. Instead of asking users to vote on two anonymous responses, the system records user feedback behaviors during their interaction with a single model, such as:

• The user explicitly states the task is complete;
• The user compliments or complains about the output;
• The user provides corrective instructions;
• A terminal command execution fails and triggers a retry;
• The user actively terminates the task during a long session.

These events are subsequently used for randomized inference. Specifically, the system compares user feedback differences across models for the same type of task to estimate each model’s net improvement rate relative to the overall average. The advantage of this causal tracing method is that it does not assume users can evaluate two models simultaneously. Instead, it infers relative performance from naturally occurring interactions.

2.2 Three Core Capability Dimensions

Under this evaluation framework, several metrics are particularly noteworthy.

Terminal Command Execution Reliability: This measures the success rate of model-generated commands during actual execution. For terminal agents, a single parameter error, path reference error, or incorrect execution sequence can lead to task failure. The reliability metric directly reflects the model’s stability within the real execution pipeline.

User Satisfaction: Estimated through the ratio of user compliments to complaints. Satisfaction depends not only on whether the answer is correct but also on whether the model is concise, provides actionable next steps, and maintains appropriate expression when user sentiment shifts.

Steerability: Measures the model’s ability to adjust its subsequent execution direction based on user feedback. For example, if a user says “don’t do it this way,” “use a simpler command,” or “ignore the previous step,” the model must quickly correct its behavior without repeating the error.

2.3 Practical Implications of Metric Changes

Based on recent public results, a trillion-parameter open model improved its terminal command execution reliability from a previously negative value to a positive one, ranking first among open-weight models. This indicates that the model has not only shed its “prone to execution failure” status but has also begun to establish an advantage in this dimension.

User satisfaction metrics have also seen a significant rise. This change is typically directly correlated with the model reducing redundant outputs, lowering invalid command retries, and improving task completion rates. The steerability metric entering the positive range suggests that the model underwent more systematic user feedback alignment during the post-training phase, meaning it no longer merely obeys static instructions but can understand dynamic corrections during a task.

3. From Base to Agent Model: Large-Scale Post-Training and Structural Optimization

3.1 Joint Reinforcement Learning Post-Training

This model was not trained from scratch but underwent large-scale joint reinforcement learning (RL) post-training on top of its previous base model. Its training scope covered hundreds of downstream domains rather than being limited to a single vertical task. The purpose was to enable the model to maintain consistent behavior across different types of tasks.

A key challenge in joint RL is the design of reward signals. In terminal agent tasks, a simple success/failure signal is insufficient. The training system typically needs to combine multiple reward sources, such as:

• Whether the command executed successfully;
• Whether the task was completed within a limited number of steps;
• Whether the user provided positive feedback;
• Whether existing context was utilized effectively;
• Whether repeated errors were avoided;
• Whether goal consistency was maintained during long sessions.

These signals must remain relatively balanced across hundreds of downstream domains; otherwise, the model easily overfits to one type of task. For instance, if the training reward is overly concentrated on command success rates, the model might tend to execute high-risk commands. If it leans too heavily toward user sentiment feedback, the model might sacrifice efficiency to generate excessive explanatory text.

3.2 Structural Replacement in Long-Context Modeling

In addition to post-training, the model’s structure was partially adjusted. Specifically, some full attention layers were replaced with gated sparse attention layers to reduce the overhead of long-context modeling.

When processing long sequences, the computational and storage complexity of full attention layers increases significantly. For terminal agent scenarios, the model often needs to simultaneously process system prompts, tool outputs, command histories, error logs, and multi-turn user corrections. In extremely long sessions, the full attention mechanism can become a primary bottleneck for inference latency and VRAM usage.

Gated sparse attention introduces sparse connections and gating mechanisms, allowing the model to avoid computing full attention across all positions when processing long sequences. This preserves information transmission along critical paths while reducing the computational cost of irrelevant or low-value associations. Compared to traditional full attention, this design acts as a compromise, finding a balance between capability loss and long-context efficiency.

3.3 Engineering Trade-offs

It is worth noting that structural replacement often carries certain risks. Full attention layers typically possess stronger global dependency modeling capabilities. While sparse attention is highly efficient, it may underperform in tasks requiring cross-long-distance associations. Therefore, in practice, only a subset of layers is replaced rather than all of them.

For agentic tasks, partial sparsification is an acceptable choice. This is because dependencies within terminal sessions are often localized. For example, the association strength between the most recent few rounds of command outputs and the next operation is much higher than that with irrelevant information from dozens of rounds ago. Furthermore, the model needs to process a large amount of tool output and system messages, which may not require establishing complete associations with all historical positions. Structural replacement essentially compresses redundant long-context computations while preserving core reasoning capabilities.

4. Terminal Agent Benchmark Performance and Practical Significance

4.1 Performance in Terminal Tasks

In terminal agent benchmark tests, this model scored 65.7%, higher than a closed-source general model’s 58.2%, and less than 1 percentage point behind another highly capable closed-source model. Three points are noteworthy regarding these results.

First, open-weight models have approached the leading level of closed-source models in terminal agent tasks. This breaks the previous perception that “agentic capabilities are mainly concentrated in large-scale closed-source models.” Second, the narrowing of the gap primarily occurs in the command execution and error recovery phases, rather than in pure text generation. Third, although the overall scores are close, stability across different task types may still vary. Simply comparing total scores is insufficient to demonstrate that the model possesses equivalent reliability across all terminal sub-tasks.

4.2 Evaluation Differences Behind the Scores

Benchmark scores reflect overall levels, but engineering teams need to pay attention to a more granular task distribution when deploying. For example, sub-tasks like file system operations, permission management, network requests, and software installation/environment configuration have different requirements for the model. A model might perform exceptionally well in file-search tasks but fail during multi-step dependency installations.

Therefore, before integrating the model into actual workflows, a secondary evaluation on a custom task set is usually necessary. Particularly for production environments, permission boundaries, and security-sensitive operations, relying solely on public benchmark results is insufficient.

5. The Realistic Gap Between Open Weights and Local Deployment

5.1 Weight Size and Quantization

The most direct benefit of open weights is that they allow researchers and enterprises to obtain full parameters for fine-tuning, distillation, evaluation, and deployment. However, for trillion-parameter models, there is a massive gap between downloading and running.

Taking BF16 precision storage as an example, the weight file for a 2.8-trillion-parameter model requires approximately 5.6TB of storage space. This is just for the weights themselves, excluding optimizer states, activation storage, and inference-time temporary caches. Even with INT4 quantization, the weight size remains around 1.4TB. Consumer-grade graphics cards typically have only a few dozen GB of VRAM, making single-machine deployment practically impossible.

5.2 Constraints of Multi-Card Distributed Inference

Deploying such models in reality usually requires multi-card collaboration, utilizing a combination of tensor parallelism, pipeline parallelism, and expert parallelism. Even with a sufficient number of high-VRAM accelerators, inter-card communication bandwidth, inference latency, batch processing throughput, and long-term operational stability must be considered.

In terminal agent scenarios, latency is particularly critical. If the model takes too long to generate the next step while a user waits for command execution, overall throughput will be severely impacted. Even with quantization and sparse structural optimizations, a dedicated inference framework and scheduling strategy are required to keep response times within acceptable limits.

5.3 Cloud Inference as a More Realistic Path

For small and medium-sized teams, directly downloading and locally running a trillion-parameter model is not feasible. A more realistic path is to use cloud inference services, delegating weight deployment to platforms capable of multi-card scheduling. When selecting an inference service, teams should focus on:

• Whether low-latency inference for large-parameter models is supported;
• Whether quantization and dynamic batching capabilities are provided;
• Whether tool calling and long-session support required for terminal agents are available;
• Whether stable responses can be maintained under high concurrency.

This is why, although such models have open weights, their actual usage still heavily relies on cloud infrastructure.

6. Engineering Practice Recommendations

6.1 Focus on Both Evaluation and Deployment Costs During Model Selection

When introducing trillion-parameter open models, teams should not focus solely on leaderboard rankings. Evaluation performance, downstream task alignment, inference latency, VRAM requirements, quantization precision loss, and long-term operational costs should all be factored into the selection process. Especially for terminal agent applications, command execution reliability and steerability are often more important than text generation scores.

6.2 Design Agent Workflows Oriented Towards Failure Recovery

Even if a model’s command execution reliability has significantly improved, failures may still occur in production environments due to environmental differences, permission restrictions, or external dependency issues. It is recommended to design explicit failure recovery mechanisms into the agent workflow, including:

• Execution result validation;
• Error log feedback;
• Automatic retry and backoff;
• User confirmation steps;
• Degradation to explanatory output upon failure.

This can mitigate the impact of occasional model errors on business processes.

6.3 Pay Attention to Long-Session Context Management

Agentic tasks are naturally accompanied by long-context growth. Even if the model’s structure has undergone sparsification optimization, the engineering side should still avoid uncontrolled context expansion. Input length can be managed by cleaning irrelevant tool outputs, compressing historical summaries, and truncating expired logs. This not only reduces inference costs but also helps minimize the probability of the model being distracted by irrelevant information.

7. Conclusion

Current trillion-parameter open models have demonstrated capabilities approaching leading closed-source models in agentic tasks, showing significant improvements particularly in terminal command execution reliability, user satisfaction, and steerability. These advancements result from the combined effects of large-scale joint RL post-training and long-context structural optimization.

On the other hand, open weights do not equate to low-cost local operation. The storage, transmission, and inference requirements of trillion-level parameters remain extremely high, and cloud-based multi-card inference will remain the primary deployment method in the short term. For engineering teams, the more pragmatic approach is not to chase the largest parameter models, but to find the balance between evaluation capabilities, task alignment, and deployment costs suitable for their specific scenarios.