A Trillion-Parameter Open-Weight Model: Agent Evaluation and Deployment in Practice

Recently, a mixture-of-experts (MoE) model with a total of 2.8 trillion parameters officially released its model weights as promised at launch, and now ranks first in the open-weight model category on a third-party agent evaluation leaderboard. Setting aside the ranking itself, two threads behind this event deserve closer attention from developers: first, how a very large MoE model upgrades from “chat capability” to “agent capability” through architectural design and reinforcement learning post-training; second, what engineering gap lies between open weights being “downloadable” and actually “runnable.” This article focuses on both questions.

1. MoE Architecture: Balancing 2.8 Trillion Parameters and 104 Billion Active Parameters

1.1 The Basic Logic of Mixture-of-Experts

The core idea behind the mixture-of-experts architecture is to replace the feed-forward network (FFN) in a Transformer with multiple parallel expert subnetworks. A gating router dynamically selects a small number of experts for each token to participate in computation:

# Simplified MoE forward pass illustration (not an actual model implementation)
gate = softmax(W_gate @ x)              # Router computes expert weights for the current token
topk_ids = argsort(gate)[-k:]           # Select only the k experts with the highest scores
y = sum(gate[i] * Expert_i(x) for i in topk_ids)

This design creates a key asymmetry: the model’s “capacity” is determined by its total parameter count, while the computational cost of a single inference pass depends only on the active parameters. In this model’s case, out of 2.8 trillion total parameters, only about 104 billion are activated per inference pass — an activation rate of roughly 3.7%. This is equivalent to getting the knowledge capacity of a multi-trillion-parameter model at the inference cost of a hundred-billion-parameter model. This is precisely why leading large models are moving toward MoE: under constrained compute budgets, sparse activation is the most economical way to scale parameter count.

1.2 Gated Sparse Attention: Controlling Long-Context Overhead

The model supports a context window of approximately 1 million tokens. Long contexts incur two main costs: attention computation grows quadratically with sequence length, and KV cache memory consumption scales linearly with context size. To address this, the model replaces some full attention layers with gated sparse attention layers: each query position no longer computes relevance against all historical tokens. Instead, a learnable gating mechanism selects portions of key segments to participate in attention computation. This “selective look-back” incurs minimal loss on most real-world tasks while significantly reducing long-context modeling overhead. It is one of the key techniques that makes million-token contexts practical.

1.3 Large-Scale Reinforcement Learning Post-Training

Compared with the previous generation, this model underwent large-scale joint reinforcement learning post-training using the older version as its base, covering hundreds of downstream domains. For agent scenarios, this step is particularly valuable: agent tasks are typically multi-step, long-horizon, and have sparse rewards — dozens of intermediate tool calls may have no clear right or wrong answer, and only the final task outcome counts. Reinforcement learning can propagate the delayed reward of “final success” back to intermediate decisions, systematically improving the completion quality of multi-step tasks. This is difficult to achieve with supervised fine-tuning alone.

2. Evaluation Paradigm Shift: From Pairwise Blind Tests to Real Conversation Signals

2.1 Limitations of Traditional Pairwise Blind Tests

For a long time, the mainstream approach to large model evaluation has been pairwise blind testing: let two models answer the same question and have users vote for the better response. This approach works for evaluating “single-turn response quality,” but has clear shortcomings for agent evaluation: users naturally prefer longer, better-formatted responses regardless of whether the task actually succeeded; evaluation questions tend to be static and struggle to cover multi-step tool calls; and voting happens after “reading,” whereas the value of an agent is realized after “execution.”

2.2 A Signal System Based on Real Conversations

This evaluation adopted a completely different approach: instead of deriving signals from votes, it extracts four types of behavioral signals from real users’ agent conversations:

• Explicit task confirmation: whether the task was actually accepted by the user;
• Praise and complaints: subjective feedback during execution;
• Correction instructions: times when the user interrupted and asked to change direction;
• Shell command execution errors: objective records of technical failures.

2.3 Causal Tracking and Net Improvement Rate

On top of these signals, the evaluator used a randomized inference method called “causal tracking” to estimate each model’s net improvement rate relative to the average. The significance of randomized inference is that different models face tasks and user populations that are not naturally comparable. Only through randomized controls and statistical inference can the “difference attributable to the model” be separated from the “difference attributable to the task.” This ranking is based on more than 17,500 real long-horizon agent conversations from global users — both the sample size and task realism are significantly higher than typical static benchmarks.

3. Key Metrics Explained

Metric	Meaning	Change
Net improvement rate	Overall net improvement relative to average	+0.9% → +5.1%
Command execution reliability	Correctness of Shell command execution	-4.5% → +7.9%
Praise/complaint ratio	Subjective user satisfaction	+1.3% → +7.7%
Steerability	Ability to adjust execution direction based on feedback	Reached +6.5%
Confirmed success	User explicitly confirmed task completion	+3.9%

Several noteworthy observations:

1. Command execution reliability is the first threshold for agents. Coding, operations, and data-processing tasks ultimately depend on command-line execution. A single failed command can break an entire task chain. This metric jumping from -4.5% (below average) to +7.9% indicates a qualitative change in the model’s most fundamental capability: getting things right.
2. Steerability reflects collaboration quality. In real usage, users rarely specify requirements completely in one go. Agents need to adjust based on feedback like “No, try a different approach.” The combination of +6.5% steerability and +7.9% command reliability corresponds exactly to the capability mix most needed in coding and operations scenarios.
3. Confirmed success is the hardest metric. Although the +3.9% confirmed success rate is lower than other sub-metrics, it directly corresponds to the user’s final acceptance behavior. It has the least room for interpretation and best represents real productivity.

4. Open Weights ≠ Locally Runnable: The Engineering Reality After 5.6 TB

Open weights are often understood as “anyone can run them locally,” but the numbers do not support this optimism. Stored at BF16 precision, the weight files for 2.8 trillion parameters amount to roughly 5.6 TB. Even quantized to INT4, they still require about 1.4 TB:

# Weight size estimation (illustrative)
total_params = 2.8e12
bf16_tb = total_params * 2 / 1e12     # BF16: 2 bytes per parameter
int4_tb = total_params * 0.5 / 1e12   # INT4: 0.5 bytes per parameter
print(bf16_tb, int4_tb)               # 5.6 1.4

Converted to GPU memory: assuming 80 GB per GPU, loading BF16 weights alone would theoretically require around 70 GPUs. Adding KV cache, activations, communication redundancy, and failover replicas, the actual cluster size would be even larger. Even quantized to INT4, the model still requires at least around 18 high-end GPUs as a starting point — a single consumer-grade machine (typically 24 GB of VRAM) cannot fit the model regardless of quantization. Moreover, MoE architectures have an easily overlooked characteristic: although each inference pass activates only a small number of experts, all experts must remain resident in GPU memory. Sparse activation saves compute, not memory.

Therefore, there is an order-of-magnitude gap between “downloadable” and “locally runnable.” In practice, there are roughly three deployment paths:

1. Self-hosted inference clusters: require GPU compute, object storage for weight shards, container platforms, monitoring and alerting infrastructure. These correspond to standard product categories available on major cloud platforms. Additionally, teams need to design tensor parallelism, pipeline parallelism, and expert parallelism strategies, and handle high-speed interconnect and throughput optimization.
2. Quantization compression: low-bit quantization such as INT4 can compress the model to the 1.4 TB range, but this is still far beyond single-machine capability. It should be viewed as a way to reduce cluster size, not a shortcut to consumer-grade deployment.
3. Managed services: use APIs through cloud-hosted large model services, shifting deployment and operational complexity to the service provider. This is suitable for teams that do not want to build their own clusters.

5. Benchmark Performance: The Gap Between Open and Closed Is Narrowing

On an agent benchmark that evaluates a model’s ability to complete multi-step operations in real terminal environments, this model achieved a score of 65.7%, higher than a leading closed-source model’s 58.2%, and within 1 percentage point of another flagship closed-source model’s 66.4%. This conveys two signals:

• The gap in agent capabilities between open-weight models and flagship closed-source models has narrowed to the 1-percentage-point range. The old impression that “open models lag a generation behind” is no longer valid.
• The industry’s evaluation focus is shifting from “how human-like is the conversation” to “was the task completed.” Execution-side metrics such as command execution, steerability, and confirmed success are replacing style and formatting as the core variables of model competitiveness.

6. Four Takeaways for Developers and Enterprises

1. Choose based on scenarios first, leaderboards second. Data-sensitive workloads with stable demand are better suited for private deployment or quantized smaller models. Workloads with high volatility and a need for rapid integration are better suited for API calls. Leaderboard rankings should only serve as an initial screening reference.
2. Build your own conversation signal system. Instead of relying solely on public leaderboards, collect real signals from your own business — confirmation rates, correction rates, command error rates — and use the same methodology to evaluate how models perform on your own task distribution.
3. Beware of parameter worship. Active parameter count, context length, and metric definitions are better predictors of actual experience than total parameter count. 2.8 trillion total parameters represent knowledge capacity; 104 billion active parameters represent inference cost.
4. Do the math before deployment. Use the simple formula “parameter count × bytes per parameter” to estimate weight size and GPU memory requirements before deciding whether to self-host, quantize, or use a managed service. This avoids the awkward situation of downloading the model only to discover it cannot run.

Conclusion

Open weights make cutting-edge model capabilities inspectable, reproducible, and improvable. But what ultimately determines success in production is whether evaluation closely tracks real tasks and whether deployment plans match resource constraints. These 2.8 trillion parameters demonstrate the ceiling that sparse architectures and reinforcement learning post-training can reach, while also reminding us that between weight files and production readiness lies a complete engineering chain of quantization, parallelism, clusters, and operations. Treating leaderboard rankings as a starting point rather than the destination is the rational way to approach open models.