Evaluating and Deploying Trillion-Parameter Open-Weight AI Agents: A Technical Deep Dive

I. Background: Agentic Capabilities as the New Frontier in LLM Evaluation

Over the past two years, the focus of large language model (LLM) evaluation has shifted significantly. Early benchmarks focused on the “answer quality” of single-turn Q&A, writing, and reasoning tasks. However, with the rise of AI Agent applications—where models need to sequentially execute terminal commands, read and write files, and complete long-horizon tasks across multi-turn interactions—the gap between “answering well” and “successfully executing” has widened.

Simultaneously, open-weight models (models with publicly available weights that allow downloading and self-hosting) have entered the trillion-parameter era. Recently, an open-weight Mixture-of-Experts (MoE) model with a total of 2.8 trillion parameters published its agentic evaluation data: in an agent evaluation based on real user sessions, it achieved a net improvement rate of +5.1%, significantly higher than the +0.9% of its previous generation. This article uses this case study to answer three questions valuable to developers:

1. How should agentic capabilities be measured? What makes real-world session signals and causal inference methodologies better than traditional blind tests?
2. What architectural and training techniques enable trillion-parameter models to “learn” agentic capabilities?
3. What is the engineering cost gap between an open-weight model being “downloadable” and “locally runnable”?

II. Evaluation Methodology: From Pairwise Blind Tests to Real-World Session Signal Mining

2.1 Limitations of Traditional Pairwise Blind Tests

Pairwise blind testing involves presenting users with answers from two anonymous models and asking them to vote for the better one, then aggregating scores. While effective for evaluating “conversational feel,” this approach has three structural flaws in agentic scenarios:

• Style over Preference: Votes are easily swayed by fluency and formatting rather than actual task completion.
• Single-Turn Perspective: A single vote only covers a fragment, failing to reflect long-horizon performance across dozens of turns and tool calls.
• Subjective Signals: What a user “feels is good” is not isomorphic with the task being “actually done.”

2.2 Extracting Four Types of Signals from Real Sessions

Next-generation agent evaluations no longer organize dedicated voting. Instead, they mine signals directly from real user agent sessions. Based on publicly available industry practices, extractable signals include at least four categories:

• Explicit Confirmation: The user explicitly states the task is complete and the result is usable. This is the positive signal closest to “ground truth.”
• Praise and Complaints: Positive and negative feedback texts within the session, reflecting satisfaction trends.
• Corrective Instructions: The user interrupts the model and demands a change in direction, indicating whether execution has “gone off track.”
• Shell Command Execution Errors: Objective failures such as non-zero exits, errors, or timeouts. These are hard signals for tool-calling reliability.

The common characteristic of these signals is that they are “byproducts” of the task execution process. They require no extra scoring effort from the user, allowing for large-scale, continuous collection.

2.3 Causal Tracking and Net Improvement Rate

Signals alone are not enough. Task difficulty and user skill levels vary greatly across sessions, so directly comparing averages introduces severe bias. The industry’s solution is a randomized inference method called “causal tracking”: randomized controlled trials are conducted on model versions during traffic allocation, and the net improvement rate of a specific model relative to the average level is estimated.

Using the case data as an example: the model’s score is based on over 17,500 real long-horizon agent task sessions from global users, with a net improvement rate of +5.1%, compared to +0.9% for the previous generation. Two key takeaways here:

• Net improvement rate is not absolute success rate. A +5.1% means the net effect “relative to the average level of all evaluated models,” not that “the vast majority of tasks failed.”
• Small percentages have statistical significance. On a sample size of tens of thousands of sessions, a few percentage points of net effect can distinguish generational differences. A 4.2 percentage point increase is a highly significant technical iteration.

III. Four Key Metrics: Dissecting Agent “Quality”

The net improvement rate is the overarching metric, but for engineering, it must be broken down into sub-metrics. The case data provides four dimensions, each corresponding to a core agentic capability:

• Shell Command Execution Reliability: Improved from -4.5% (below average) in the previous generation to +7.9%. Terminal commands are high-risk areas for irreversible operations—a single wrong deletion, overwrite, or permission change can destroy the entire execution environment. Therefore, this metric is the “hard bottom line” for agents.
• Praise-to-Complaint Ratio: Rose from +1.3% to +7.7%, reflecting the net change in user subjective satisfaction.
• Steerability: Reached +6.5%, measuring whether the model can adjust its execution direction based on corrective instructions given by the user mid-task. In human-machine collaboration workflows, “listening to instructions” is often more important than “getting it right the first time.”
• Explicit Confirmation Success Rate: +3.9%. Only tasks explicitly confirmed as complete by the user are counted, making it the most conservative success metric.

It is worth emphasizing that these four metrics are not fully correlated. A model that is expressive and pleasing but stubborn might excel in satisfaction metrics but fall behind in steerability. When selecting a model, determine the weights based on your business context: highly automated pipeline scenarios prioritize Shell reliability, while intensive human-machine collaboration scenarios should prioritize steerability.

IV. Architecture and Training: Where Agent Capabilities Come From

4.1 Mixture-of-Experts Architecture: Decoupling Capacity from Compute

The model adopts a Mixture-of-Experts (MoE) architecture: it has a total of 2.8 trillion parameters but only activates about 104 billion parameters per inference, an activation ratio of roughly 3.7%. The core idea is to split the feed-forward layers into multiple “experts” and have a router dynamically select a small number of experts for each token’s computation. This decouples parameter capacity from per-inference compute—total parameters determine the “knowledge reserve ceiling,” while activated parameters determine the “inference cost.”

Below is a simplified MoE forward pass code snippet (actual implementations group experts for aggregated computation to avoid full matrix multiplication):

import torch
import torch.nn.functional as F
class MoEFFN(torch.nn.Module):
    def __init__(self, d_model, d_ff, num_experts=64, top_k=6):
        super().__init__()
        self.experts = torch.nn.ModuleList([
            torch.nn.Sequential(
                torch.nn.Linear(d_model, d_ff), torch.nn.SiLU(),
                torch.nn.Linear(d_ff, d_model))
            for _ in range(num_experts)])
        self.gate = torch.nn.Linear(d_model, num_experts)
        self.top_k = top_k
    def forward(self, x):                       # x: [batch, seq, d_model]
        prob = F.softmax(self.gate(x), dim=-1)  # Expert preference distribution per token
        w, idx = torch.topk(prob, self.top_k, dim=-1)
        w = w / w.sum(-1, keepdim=True)         # Normalize top-k weights
        out = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            hit = (idx == e)                    # Token positions hitting this expert
            if hit.any():
                coeff = (w * hit).sum(-1, keepdim=True)
                out = out + coeff * expert(x)   # Sparse activation: only a subset of tokens pass through this expert
        return out

Engineering focuses for the supporting design include the ratio of experts to top-k, load balancing loss (to prevent routing collapse to a few experts), and capacity factor (to prevent token dropping due to expert overload).

4.2 Gated Sparse Attention: Controlling the Overhead of Million-Token Contexts

The model supports a context window of approximately 1 million tokens. The compute and memory overhead of full attention grows quadratically with sequence length; at the million-token level, it is unaffordable both computationally and storage-wise. The case’s approach is a hybrid attention layer: replacing some full attention layers with gated sparse attention layers. Learnable gates determine whether information takes the sparse path or the dense path, drastically reducing long-context modeling overhead while preserving critical global dependencies. This hybrid design of “partial full attention + partial sparse attention” has become a standard industry solution for long-context models.

Another major overhead in long contexts is the KV cache. A generic estimation code snippet illustrates the pressure (layer width and head count are assumed values):

def kv_cache_gb(num_layers, kv_heads, head_dim, seq_len, batch, bytes_per=2):
    # One copy each for K and V, bytes_per=2 corresponds to BF16
    return num_layers * 2 * kv_heads * head_dim * seq_len * batch * bytes_per / 1e9
# Assumptions: 80 layers, 8 KV heads (Grouped-Query Attention), head_dim=128, 1 million tokens, batch=1
print(kv_cache_gb(80, 8, 128, 1_000_000, 1))   # Approximately 328 GB

Even with Grouped-Query Attention to reduce KV head counts, the cache for a single million-token session requires hundreds of GBs. This means long-context inference must rely on multi-GPU parallelism and paged cache management.

4.3 Large-Scale Joint Reinforcement Learning Post-Training

Beyond architecture, another source of agentic capability is post-training. The case model used its previous generation as a base and underwent large-scale joint reinforcement learning (RL) covering hundreds of downstream domains. Why do agentic scenarios rely so heavily on RL? Because capabilities like tool calling, multi-step planning, and error recovery are difficult to exhaustively enumerate with static supervised data. The same operation can have multiple correct paths in different environments, and subsequent states depend on environmental feedback. RL uses the environment’s real signals (whether a command executed successfully, whether a task was confirmed) as rewards, allowing the model to learn the closed loop of “trial and error — observation — correction” through environmental interaction.

Of course, large-scale joint RL comes with costs: cross-domain reward design, preventing reward hacking, and avoiding catastrophic forgetting of general capabilities are classic engineering challenges that must be addressed.

V. The Reality of Open Weights: From “Downloadable” to “Runnable”

5.1 Storage and VRAM Estimation

Open weights are often mistakenly equated with “locally deployable,” but the numbers don’t lie. Taking 2.8 trillion parameters as an example:

params = 2.8e12
for name, bits in [("BF16", 16), ("INT8", 8), ("INT4", 4)]:
    print(name, f"{params * bits / 8 / 1e12:.1f} TB")
# BF16: 5.6 TB | INT8: 2.8 TB | INT4: 1.4 TB

Even with INT4 quantization, the weight file requires about 1.4TB. Looking at activated parameters: 104 billion parameters in BF16 require about 208GB of resident VRAM, exceeding the capacity of a single mainstream data center accelerator. This means that even calculating only “the weights actually participating in computation per step” requires tensor parallelism. If full weights are resident (MoE routing requires random access to all experts), 5.6TB of BF16 weights plus KV cache and activation values mean an actual multi-node, multi-GPU cluster is required. Deployment on a single machine with consumer-grade GPUs is essentially impossible.

5.2 Practical Trade-offs in Quantization and Inference Strategies

Quantization is the first lever for cost compression, but note: INT4 weight quantization can reduce memory footprint to a quarter, but agentic scenarios are more sensitive to numerical precision. In long-chain tool calls, minor deviations in early steps are amplified by subsequent steps. After quantization, regression validation must be performed on agentic tasks, not just language modeling metrics. Furthermore, tiered offloading of expert parameters to memory and disk, and optimizing memory access locality for sparse experts, are active research areas in current inference systems, though throughput losses remain significant.

5.3 Three Actionable Paths

Balancing cost and benefit, teams of different sizes can refer to the following paths:

• Individual Developers and Small Businesses: Prioritize calling capabilities directly via cloud model services, or use model hosting and pay-as-you-go inference provided by cloud platforms to avoid the fixed costs of building a custom cluster.
• Mid-to-Large Enterprises: Without extreme data compliance constraints, deploy inference clusters based on cloud GPU compute instances and container services, scaling elastically according to business peaks. Cloud platforms like Baidu AI Cloud are relatively mature in providing these general capabilities (GPU compute, container orchestration, model services, monitoring, and alerting).
• Teams with Deep Customization Needs: Adopt parameter-efficient fine-tuning techniques like LoRA to inject domain knowledge by training only a small number of new parameters, avoiding the high cost of full-parameter fine-tuning. Alternatively, use a hybrid routing strategy: “local small models for high-frequency simple tasks + cloud large models for complex tasks.”

VI. Terminal Benchmarks: The Touchstone of Agentic Capabilities and Their Limitations

The case also mentioned a score on a terminal agent benchmark: 65.7%, less than 1 percentage point behind top closed-source models. Terminal environments are considered the touchstone for agentic capabilities because they possess three characteristics: Persistent State (every command changes the environment), Irreversible Operations (wrong commands carry real costs), and Mandatory Self-Recovery (reading errors and correcting them). These characteristics are highly isomorphic with real-world production DevOps and development scenarios.

However, benchmark scores cannot be directly equated with business value. Before deployment, it is recommended to do three things:

1. Build a Domain Evaluation Set: Extract scenarios from your real task logs to construct a use-case set with environment snapshots for replayable evaluation.
2. Focus on Failure Modes, Not Total Scores: Two models with similar total scores might fail for completely different reasons—one fails at environment understanding, the other at over-operation.
3. Grayscale Launch: Use a small volume of real sessions to continuously collect the four types of signals mentioned in Section II, forming your own “net improvement rate” dashboard.

VII. Conclusion: Reading Leaderboards as Engineering Problems

The most valuable takeaways for developers from this case are not specific rankings, but three methodological judgments:

• Evaluation Level: The net improvement rate derived from real session signals + causal inference is closer to the business essence of “whether the task is done” than pairwise blind tests. The four sub-metrics provide a finer lens for model selection.
• Technical Level: Trillion-parameter capacity, cost control via sparse activation, long-context feasibility brought by gated sparse attention, and large-scale RL covering hundreds of domains jointly form the technological foundation of agentic capabilities.
• Deployment Level: Open weights do not equal locally runnable. Storage and VRAM costs must be calculated in advance, and quantization and fine-tuning solutions must be validated alongside business needs.

For teams preparing to introduce LLM agent capabilities, the rational sequence is: first define your evaluation sets and signal systems, then estimate the deployment cost curve, and finally decide whether to build, host, or use a hybrid approach—rather than working backwards from leaderboard rankings to dictate technical solutions.