Chinese Open-Weight Model Achieves Significant Gains in Agent Capabilities

The Open-Weight Race Shifts: From Parameter Counts to Task Reliability

Over the past year, the competitive focus in the large language model space has undergone a structural shift. Early conversations revolved around static metrics like parameter counts and context window sizes. As the agent paradigm matures, the industry has come to realize that what truly determines whether a model is production-ready is not how fluently it handles single-turn Q&A, but whether it can converge reliably on long-horizon, multi-step tasks involving external tool calls.

Against this backdrop, a Chinese AI firm recently honored its release commitment by open-weighting its latest ultra-large Mixture-of-Experts (MoE) model on a public model-hosting platform. The model packs 2.8 trillion total parameters, activates roughly 104 billion per inference pass, and supports a context window of approximately 1 million tokens. More importantly from an engineering standpoint, it achieved the top position among open-weight models on public agent-oriented benchmarks, leading in all three evaluated categories—work, coding, and chat. The signal is unmistakable: open-weight models have moved past “can it be used?” into “can it be trusted with long tasks?”

Why Pairwise Blind Testing Falls Short for Agent Evaluation

Understanding the weight of this ranking requires understanding the evaluation methodology behind it. Traditional LLM leaderboards rely on pairwise blind testing: two anonymous responses are shown to a user, who votes for the better one. This works well for single-turn text quality but has a clear blind spot for agent tasks—agents derive their value from multi-turn interaction, tool invocation, error recovery, and goal completion, none of which a single-turn preference vote captures.

The agent benchmark in question takes a different approach to signal extraction. Rather than asking users to cast extra votes, it harvests behavioral signals from real user-agent sessions, including:

• Explicit confirmation: the user explicitly states that the task is done or the result is usable.
• Praise vs. complaint: positive or negative feedback on output quality.
• Correction directives: the user asks the model to change direction or fix an error.
• Command execution failures: reliability issues exposed when external commands fail.

On this basis, the evaluators apply a randomized inference method—think of it as “causal tracing”—to estimate each model’s net improvement rate relative to the average. Compared with preference voting, this approach draws signals from real consequences on real tasks, aligning more closely with production-grade evaluation criteria.

The model’s ranking is based on over 17,500 real long-horizon agent sessions from global users, with a net improvement rate of +5.1%, up 4.2 percentage points from its predecessor’s +0.9%. In agent scenarios, that swing is substantial—it means the model’s net “payoff” on real tasks has multiplied several fold.

Breaking Down the Gains: Where Did Improvement Happen?

The aggregate net improvement rate is useful, but engineering teams care more about which sub-capabilities are the weak spots and the strong suits. Public data reveals a clear structural pattern:

Metric	Meaning	Previous Generation	Current Generation
Bash Reliability	External command execution reliability	-4.5% (13th place)	+7.9% (1st among open models)
Praise vs Complaint	Net user satisfaction	+1.3%	+7.7%
Steerability	Ability to adjust direction per feedback	—	+6.5% (1st among open models)
Confirmed Success	User-confirmed task completion	—	+3.9% (4th among open models)

The most engineering-relevant swing is the command execution reliability reversal: from negative territory—where failures generated more negative than positive signals—to +7.9%, the top open-model score. For agents that call external tools, run scripts, or manipulate file systems, this metric directly determines whether a single execution failure cascades into a broken task chain.

Steerability at +6.5% also leads open models. Steerability is essentially “can the model listen to corrections?” When a user flags a wrong direction, does the model abandon the current path and re-plan, or does it keep burning context and tool-call budget down the wrong trail? On long-horizon tasks, this ability often matters more than single-step accuracy.

Architecture and Training: Engineering Trade-Offs Behind 2.8 Trillion Parameters

The model uses a Mixture-of-Experts (MoE) architecture: 2.8 trillion total parameters, but only about 104 billion activated per inference pass. This “large total, small active” design is a classic trade-off between model capacity and inference cost. A larger expert pool broadens knowledge coverage and specialization, while sparse activation keeps per-pass compute manageable.

At the architectural level, two key changes distinguish this generation from the base version:

1. Large-scale joint reinforcement learning post-training: Starting from the base model, joint RL is applied across hundreds of downstream domains. The focus here is not on injecting new knowledge, but on aligning to task-completion standards and strengthening multi-step decision stability and error correction. The steerability and execution reliability gains trace largely to this stage.
2. Gated sparse attention replacing some full-attention layers: The dominant cost in long-context modeling is the quadratic growth of attention computation. Swapping certain full-attention layers for gated sparse attention preserves critical information pathways while reducing compute and memory for long sequences. For a 1-million-token context, this kind of modification is what makes long context “usable” rather than merely “claimable.”

Crucially, the architectural changes and post-training strategy work in tandem: sparse attention lowers the compute barrier for long context, and RL post-training teaches the model to maintain task coherence even within that compressed attention horizon.

Open-Weight Does Not Mean Run-It-Locally: The Real Deployment Bill

“Open-weight” is the headline, but engineering teams must keep in mind the gap between downloadable and locally runnable.

Storing 2.8 trillion parameters at BF16 precision yields roughly 5.6 TB of weight files; even with INT4 quantization, you still need about 1.4 TB. The implications are straightforward:

• Consumer-grade single-GPU deployment is essentially out of the question.
• Multi-GPU, multi-node inference becomes mandatory, bringing complexity in memory bandwidth, interconnect topology, and parallelism strategy.
• Quantization compresses size but introduces precision loss that must be validated against your target tasks.

For teams planning on-premises deployment, a pragmatic path looks like this:

Recommended Evaluation Path:
1. Define task boundaries: Do you truly need a 1-million-token context? Most business cases can first validate ROI with shorter contexts.
2. Prioritize quantization validation: Run your target task suite under INT4/INT8 and record the accuracy and execution-success decay curves.
3. Select a parallelism strategy: Choose a combination of tensor parallelism, pipeline parallelism, and expert parallelism based on your cluster interconnect.
4. Establish regression baselines: Use agent task success rate and tool-call error rate as core acceptance metrics, not perplexity.

In short, open-weight lowers the acquisition barrier, not the runtime barrier. For most small and mid-size teams, a hybrid approach—cloud on-demand inference combined with lightweight local validation—is the more realistic strategy, letting you invest local resources in data, evaluation, and business integration.

Benchmark Comparison on End-User Agent Tasks

On the public benchmark for end-user agent tasks, the model scores 65.7%. For reference, a leading closed-source model scores 58.2% on the same benchmark, while another major international closed-source model reaches 66.4%—a gap of less than one percentage point. This result indicates that open-weight models can now compete head-to-head with top closed-source offerings on the demanding end-user agent scenario, with differences falling into the range explainable by statistical noise and task-distribution variance.

That said, a single benchmark score is no substitute for task-specific evaluation in your own environment. End-user agent failure modes are highly environment-dependent—file-system permissions, network conditions, tool versions all matter. Public benchmarks offer directional guidance, not definitive answers.

Practical Takeaways for Developers and Enterprises

Four lessons emerge from this open-weight release:

First, align evaluation methods with task form. Agent capability assessment must rely on real task signals, not single-turn preference votes. When building internal evals, prioritize collecting behavioral data such as task-completion confirmations, correction frequency, and tool-call failure rates.

Second, weight reliability metrics above fluency metrics. Engineering indicators like command execution reliability and steerability directly determine whether an agent can enter production workflows. Give them prominent weight in model selection.

Third, long context demands both architecture and training. Claiming a context length means little without structural support. Sparse attention modifications paired with post-training strategy are prerequisites for making long context genuinely usable.

Fourth, open-weight is not low-cost deployment. Deployment cost, quantization loss, and parallelism complexity must factor into decision-making. For most teams, a hybrid model of cloud on-demand inference and lightweight local validation is the most viable path.

The emergence of ultra-large open-weight models is pushing agent capabilities from “demonstrable” toward “dependable.” For developers, the real opportunity lies not in chasing parameter records, but in building task-level evaluation and integration capabilities around reliability, steerability, and deployment economics.