Open-Weight Trillion-Parameter Model Tops Agentic Leaderboard: A Technical Analysis

1. A Dual Breakthrough in Open Weights and Agentic Capability

A leading domestic AI lab has fulfilled its commitment by officially releasing the weight files of its latest-generation large language model on a public model-hosting platform. The model uses a Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8 trillion, while activating only about 104 billion parameters per inference, balancing scale and efficiency. The move marks a shift for ultra-large open-weight models from “technical demonstration” to “accessible resource.”

More notably, the model performs strongly on a third-party agent evaluation benchmark: it ranks first among all evaluated open-weight models and 13th overall across both closed-source and open-source models. Especially on agent tasks oriented toward real end-point operation scenarios, its score is very close to the currently recognized strongest commercial closed-source model, with a gap of less than 1 percentage point.

2. Rethinking Evaluation Methodology: From Pairwise Blind Tests to Causal Tracing

Traditional LLM arena evaluations typically use pairwise blind testing, in which users vote on their preference between two models’ responses. This approach has clear limitations when evaluating agents: an agent’s value lies not only in single-turn response quality but also in its combined ability to understand task intent across multi-turn interactions, execute tool calls, and revise strategies based on feedback.

The agent evaluation platform used here introduces a markedly different methodology. Rather than starting from human-designed benchmark questions, it extracts signals directly from long-horizon sessions generated by real users interacting with agents. These signals include:

• Explicit confirmation: the user explicitly states that the task is complete or the goal has been achieved
• Sentiment feedback: user praise, thanks, complaints, or dissatisfaction
• Correction instructions: user requests to correct the agent’s execution direction or results
• System errors: exceptions or failure indicators produced during command-line tool execution

The platform uses a randomized inference method called “causal tracing” to estimate each model’s net improvement rate relative to the average across all evaluated models. The core advantage of this method is that it does not simply count user satisfaction with a given response; it infers, through causal relationships among signals, the degree to which a model actively improves task outcomes.

3. Core Metrics Jump: Data-Driven Capability Evolution

This ranking is based on more than 17,500 real long-horizon agent task sessions from users worldwide, yielding a final net improvement rate of +5.1%. Compared with the previous-generation model’s +0.9%, this is a 4.2-percentage-point increase, implying a qualitative leap in the model’s overall ability to help users achieve goals in real complex tasks.

Sub-metrics reveal a more detailed distribution of improvements:

Shell command execution reliability rose from -4.5% (13th place) in the previous generation to +7.9%, topping the open-weight model ranking. This metric directly measures the model’s ability to execute commands, parse output, and handle exceptions in terminal environments—the core embodiment of an agent’s “hands-on” capability. The shift from negative to positive and the large gain indicate that the model has evolved at the tool-calling level from “occasionally error-prone” to “stably reliable.”

User satisfaction (ratio of praise to complaints) improved from +1.3% to +7.7%. This is not just a numerical increase but reflects a substantially better match between generated content and user expectations. In real scenarios, users tend to express praise for results that exceed expectations and complaints for results that remain ineffective after repeated corrections.

Steerability reached +6.5%, also first among open models. This metric measures whether the model can quickly adjust its execution direction after receiving mid-task correction instructions from the user. In many real tasks, users’ initial requirements are often vague or incomplete, and the model’s responsiveness to immediate feedback directly determines task completion efficiency.

Explicit task confirmation success rate was +3.9%, ranking fourth among open models. This metric is relatively conservative because not all tasks receive explicit user confirmation, but it indirectly validates the model’s ability to maintain goal consistency across multi-turn complex interactions.

4. Architecture and Training: A System Upgrade from Base Model to Reinforcement Learning

The model uses the previous version as its base and underwent large-scale joint reinforcement learning post-training. This process covered hundreds of downstream domains, including code generation, data analysis, system operations, and content creation. Unlike the traditional pretraining-plus-fine-tuning paradigm, joint reinforcement learning lets the model improve multiple capability dimensions simultaneously under the same optimization objective, avoiding the catastrophic forgetting that staged training can cause.

At the architecture level, a key technical change is replacing some full-attention layers with gated sparse attention layers. Standard full attention requires each token to attend to all other tokens in the sequence, and computational complexity grows quadratically with sequence length. Gated sparse attention uses a learnable gating mechanism to dynamically decide which key historical tokens each query token needs to attend to, reducing the computational cost of long-context modeling from quadratic to approximately linear.

This change is critical for supporting a context window of about 1 million tokens. Without sacrificing long-range dependency modeling, gated sparse attention enables the model to process ultra-long documents, complete code repositories, or long multi-turn conversation histories under limited compute.

5. The Practical Gap Between Open Weights and Local Deployment

Although the weight files are available for download, there is a huge engineering distance between “downloadable” and “locally runnable.” Model weights stored in BF16 precision are about 5.6TB in size; even with INT4 quantization compression, about 1.4TB of storage is still required. This means single-machine consumer-grade GPUs (typically with 24GB to 80GB of VRAM) are essentially incapable of direct deployment.

For teams with private deployment needs, realistic paths include:

• Multi-machine distributed inference: connect multiple servers equipped with high-end GPUs over a high-speed network and use model parallelism and pipeline parallelism to distribute weights across many GPUs. This approach requires professional distributed inference frameworks and network infrastructure support.
• CPU memory offloading: use the very large capacity of CPU main memory (up to several TB per machine) to hold weights, with GPUs handling only compute-intensive matrix operations. Inference speed is limited by PCIe bandwidth between CPU and GPU, making this suitable for scenarios with modest throughput requirements.
• Cloud service API calls: for most developers and small-to-medium businesses, calling cloud APIs directly is the most cost-effective approach. Cloud platforms handle underlying high-performance inference cluster scheduling and elastic scaling, so users need not worry about hardware details.

6. Outlook on Model Capability Evaluation Paradigms in the Agent Era

The significance of these evaluation results lies not only in a particular model’s specific scores but also in validating the feasibility of a new evaluation paradigm for agents oriented toward real scenarios. Traditional static benchmarks (such as QA pairs and multiple-choice questions) struggle to capture an agent’s adaptability in dynamic environments, whereas causal tracing based on real session signals offers the industry a scalable, contamination-resistant, and production-relevant evaluation approach.

For model developers, this means the optimization direction is shifting from “improving single-turn response accuracy” to “improving task completion rates and user control experience in multi-turn interactions.” For enterprise model selection, metrics such as net improvement rate, command execution reliability, and steerability are more informative than raw benchmark scores because they map directly to efficiency gains and labor savings in real business scenarios.

As open-weight models surpass the trillion-parameter mark in total size, the industry is entering a new phase: the ceiling of model capability is rising, and the deployment barrier is rising along with it. How to find a balance among model scale, openness, and deployability will be a key question for the LLM field in the period ahead.