Trillion-Parameter Open-Source Model: Agent Capability Evaluation and Technical Analysis

Background: Agent Evaluation Is Moving from Preference Voting to Behavioral Signals

Over the past two years, the paradigm for evaluating large language models has shifted markedly. Early evaluations relied mainly on manual pairwise blind testing: users compared two responses and voted, and the results were aggregated into win rates or Elo scores. This approach is intuitive enough for general-purpose chat, but it has exposed clear weaknesses in agent scenarios. Whether a real task succeeds is rarely determined by which answer sounds more pleasing — it is determined by whether the task is actually completed, whether commands execute successfully, and whether errors get corrected in time.

A more production-realistic evaluation approach has emerged in response. Rather than requiring users to vote explicitly, it extracts implicit signals from real agent sessions: explicit task confirmations, praise, complaints, and corrective instructions from users, as well as errors from shell command execution. A randomized inference method (often referred to in the industry as “causal tracing”) is then used to estimate each model’s net improvement rate relative to the average. The advantage is that the signals come from real, long-horizon tasks: the noise is lower, and the method can tell the difference between “sounding good” and “getting it right.”

A recent open-source model has taken first place among open-weight models in this type of evaluation, and its technical approach and metric structure are worth a detailed look.

Model Architecture: Mixture-of-Experts Plus Gated Sparse Attention

The model adopts a Mixture-of-Experts (MoE) architecture with 2.8 trillion total parameters, roughly 104 billion parameters activated per inference, and a context window of about 1 million tokens. This configuration reflects a classic engineering trade-off: a very large total parameter count to carry knowledge breadth, and sparse activation to control the cost of each inference.

Two changes stand out compared with the previous generation:

1. Large-scale joint reinforcement learning post-training

Building on the previous generation as its base, the model went through joint reinforcement learning post-training spanning hundreds of downstream domains. The goal of this stage is not to add general knowledge, but to teach the model how to act across a diverse distribution of tasks: when to call tools, how to interpret error messages, and how to stay on target across multi-turn interactions. For agent scenarios, the quality of the post-training stage often determines real-world performance more than pretraining scale does.

2. Replacing some full-attention layers with gated sparse attention layers

The cost of a 1-million-token context window is quadratic growth in attention computation. Replacing some full-attention layers with gated sparse attention layers preserves long-range dependency modeling while significantly cutting the overhead of long-context processing. The gating mechanism dynamically decides which positions need to participate in attention computation, rather than treating every token alike. This change matters especially for agents: over a long-horizon task, conversation history, tool outputs, and file contents pile up quickly, and without sparsification, inference latency and memory usage would become unmanageable.

# Conceptual illustration: selection logic of gated sparse attention (pseudocode)
def gated_sparse_attention(query, keys, values, gate):
    # gate is a learnable gate that outputs a keep probability for each key position
    scores = gate(query, keys)          # dynamic scoring
    mask = top_k_mask(scores, k)         # keep only the top-k key positions
    return softmax_attention(query, keys, values, mask)

Evaluation Results: Net Improvement Rate and What the Sub-Metrics Mean

The model’s ranking is based on more than 17,500 real long-horizon agent task sessions from users worldwide, with a net improvement rate of +5.1% — a 4.2 percentage-point increase over the previous generation’s +0.9%. It ranks 13th among all evaluated models, first among open-weight models, and first among open models in all three categories: work, coding, and chat.

The sub-metrics deserve particular attention, because they map directly to the pain points of putting agents into production:

Metric	Previous Generation	Current Model	What It Means in Practice
Bash Reliability	-4.5% (13th)	+7.9% (1st among open models)	Reliability of shell command execution; directly affects automated scripts and ops tasks
Praise vs Complaint	+1.3%	+7.7%	User satisfaction; reflects response quality and how accurately the model understands the task
Steerability	—	+6.5% (1st among open models)	Ability to adjust execution direction based on user feedback; determines multi-turn collaboration efficiency
Confirmed Success	—	+3.9% (4th among open models)	Share of tasks that users explicitly confirm as completed

Bash Reliability turning from negative to positive is the most telling improvement in this release. Command execution errors are a frequent cause of agent task failures: -4.5% means the previous generation performed below average on this dimension, while +7.9% means the current model is now well above it. For scenarios where the model needs to run build, deployment, or data-processing scripts on its own, this improvement translates directly into less manual intervention.

Steerability reaching +6.5% concerns the interaction experience. In real development workflows, users rarely specify a complete requirement up front; they work in the pattern of “do this step first, and we will adjust after seeing the result.” Whether the model can correctly interpret corrective instructions and change course determines whether the collaboration runs smoothly. Ranking first among open models on this metric suggests that the post-training stage made good use of feedback signals.

Practical Takeaways for Developers

1. A long context window is not the same as effective context

A 1-million-token window is a capacity ceiling, not a guarantee that the model can make effective use of everything inside it. The introduction of gated sparse attention shows that vendors themselves are using architectural means to control the real cost of long contexts. When building long-horizon agents, developers should still combine retrieval, summarization, and state-compression strategies rather than dumping the entire history into the context indiscriminately.

2. Match evaluation metrics to your business scenario

If your workload is dominated by automated script execution, Bash Reliability should carry more weight than chat-oriented metrics; if it is dominated by multi-turn requirement iteration, Steerability deserves more attention. General leaderboard rankings serve as an initial screen — the sub-metrics are what should drive model selection.

3. Open weights bring deployment flexibility

Open weights mean a team can deploy, fine-tune, and quantize the model in its own environment, which matters most for scenarios with strict data-compliance requirements or a need for deep customization. That said, 2.8 trillion total parameters mean the deployment bar stays high even with sparse activation; a realistic plan needs to combine quantization, inference optimization, and hardware provisioning.

4. Post-training quality is becoming the key differentiator

As base-model capabilities converge, the breadth and quality of the domains covered by joint RL post-training are becoming the dividing line for agent performance. When choosing a model, the domain coverage and task distribution of the post-training data deserve more scrutiny than raw parameter count.

Summary

This release reflects a clear trend: agent evaluation is shifting from subjective preference to behavioral signals, model architecture is shifting from dense to sparse attention to support long contexts, and post-training domain coverage is becoming a core variable in agent capability. For developers, understanding what metrics such as net improvement rate, command execution reliability, and steerability actually mean delivers more long-term value than memorizing leaderboard positions. As agents move from demo to production, these metrics speak to the fundamental question: can the model reliably finish the job?