Trillion-Parameter Open-Weight Models: Agent Evaluation and Deployment Explained

1. Background: Open Weights Enter the Trillion-Parameter Era

Recently, a research organization released the full weights of its flagship large model to the open-source community, honoring a commitment made at launch. Using that model as an example, several key specifications are worth noting:

• Mixture-of-experts (MoE) architecture with 2.8 trillion total parameters;
• Only about 104 billion parameters activated per inference, using sparse activation to control inference cost;
• A context window of roughly 1 million tokens, aimed at long-horizon agent tasks;
• Ranked first in the open-weight group of a third-party agent evaluation, and first among open models in the work, coding, and chat subcategories; scored about 65.7% on an agent benchmark for terminal command-line tasks, within less than 1 percentage point of the top closed-source flagship on the leaderboard.

Behind these facts lie two main threads worth exploring. First, evaluation methodology is evolving: in the agent era, model assessment is shifting from “pairwise blind tests under human-constructed prompts” to “outcome signals from real user sessions.” Second, there is an engineering reality: for open weights, “downloadable” and “locally runnable” are separated by orders of magnitude. Understanding the causes of this gap and the ways around it is essential before any team makes a technical selection decision.

2. Evaluation Methodology: From Pairwise Blind Tests to Real Session Signals

2.1 Why Pairwise Blind Testing Is No Longer Enough

For the past several years, the dominant approach for model leaderboards has been pairwise blind testing: human reviewers compare two models’ answers to the same prompt and vote. This works well for short conversations, but it has structural flaws when applied to agent evaluation:

1. Distorted prompt distribution. Reviewers mostly see carefully constructed short prompts, whereas real agent tasks often involve dozens of interaction turns spanning hours or even days. The difficulty distributions of the two barely overlap.
2. It evaluates “style,” not “outcome.” Human reviewers naturally favor longer answers with a confident tone and polished formatting, but the criterion for agent tasks should be objective—whether the command executed successfully, whether the task actually closed the loop.
3. Single-turn voting cannot measure long-horizon capability. The core difficulties of agents lie in multi-step planning, error recovery, and adjusting direction based on feedback. These capabilities simply cannot be captured by single-turn preference votes.

2.2 Real Session Signals and “Causal Tracing”

The new generation of agent evaluation takes a different approach: instead of organizing reviewer votes, it extracts signals directly from real users’ agent sessions. Typical signals fall into four categories:

• Explicit confirmation: the user clearly states “done, no problems” at the end of a task—the strongest positive outcome signal;
• Praise and complaints: positive or negative sentiment feedback within the session, reflecting subjective satisfaction;
• Correction instructions: the user interrupts mid-task to give a corrective direction (“No, it should be…”), which is both a rejection signal and material for measuring “steerability”;
• Shell command execution errors: abnormal exit codes, error rollbacks, and similar failures—fully objective failure signals.

For statistical methodology, this type of evaluation adopts a “causal tracing” framework similar to online randomized experiments: real traffic is randomly assigned across models, and randomized inference is then used to estimate each model’s net improvement rate relative to the overall average. The idea can be abstracted as:

net_improvement(m) ≈ E[session_outcome | model=m] − E[session_outcome | overall average]

Randomization ensures that the task difficulty, user population, and session length distributions each model faces are broadly comparable, so the difference above can be attributed to the model itself rather than to “happening to draw easier tasks.” This is akin to difference-in-means estimation in A/B experiments—a direct application of causal inference to model evaluation. Compared with a simple “win rate,” the net improvement rate has three advantages: it anchors on task outcomes and resists style bias; it models the final outcome of long-horizon sessions rather than single-turn impressions; and it uses all models as the baseline, so new models entering or leaving the pool do not systematically distort others’ scores.

2.3 A Set of Case Data: How Metrics Characterize Agent Capability

Consider public evaluation data from two generations of one open-weight model (sample: more than 17,500 real long-horizon agent sessions from users worldwide):

Metric	What it measures	Previous generation	Current generation
Net improvement rate	Overall net improvement vs. the average across all models	+0.9%	+5.1%
Command execution reliability	Objective success rate of Shell command execution	-4.5%	+7.9%
Praise vs. complaints	Difference between positive and negative user feedback	+1.3%	+7.7%
Steerability	Adjusting execution direction based on feedback	—	+6.5%
Confirmation success rate	Users explicitly confirming task completion	—	+3.9%

Two observations stand out:

1. Command execution reliability improved the most (about 12 percentage points) and flipped from negative to positive. This suggests that verifiable environments (whether a command succeeds) provide extremely dense reward signals and are the most “cost-effective” focus during post-training.
2. Confirmation success rate (+3.9%) is actually lower than the satisfaction-type metrics. This hints at a gap between “the user says they’re satisfied” and “the user confirms the task is closed out”—the model may complete most of the work but leave residual steps. This is exactly where agent productization needs the most refinement.

The lesson for engineering teams: when building in-house evaluations, prioritize collecting objective outcome signals (command success/failure, task closure rate), then supplement with subjective feedback, rather than relying solely on human ratings.

3. Technical Breakdown: MoE, Sparse Attention, and RL Post-Training

3.1 Mixture-of-Experts: Decoupling “Capacity” from “Cost”

The combination of “2.8 trillion total parameters, 104 billion activated parameters” is a classic mixture-of-experts design: dense feed-forward networks in Transformer layers are replaced by multiple expert sub-networks, and a router selects only a small number of experts per token. The core logic can be expressed in illustrative code:

import torch
import torch.nn.functional as F
class MoERouter(torch.nn.Module):
    """Illustrative MoE routing: shows only the core logic of Top-K gating"""
    def __init__(self, hidden_dim, num_experts, top_k=8):
        super().__init__()
        self.gate = torch.nn.Linear(hidden_dim, num_experts, bias=False)
        self.top_k = top_k
    def forward(self, x):
        # x: [batch, seq_len, hidden_dim]
        logits = self.gate(x)
        probs = F.softmax(logits, dim=-1)
        topk_probs, topk_idx = torch.topk(probs, self.top_k, dim=-1)
        topk_probs = topk_probs / topk_probs.sum(-1, keepdim=True)  # normalize
        # Next, dispatch tokens to the corresponding experts by topk_idx and merge outputs with weights
        return topk_probs, topk_idx

The direct effect: total model capacity (knowledge storage) can scale roughly linearly with the number of experts, while the compute per forward pass depends only on the activated parameters. This is precisely why “trillion-scale total parameters + hundred-billion-scale activation” can coexist.

3.2 Gated Sparse Attention: Cutting Costs for Million-Token Contexts

Standard self-attention compute grows quadratically with sequence length, while KV cache memory grows linearly with sequence length. At a 1-million-token context, even with only hundred-billion-scale activated parameters, full attention layers become a significant bottleneck.

A common industry compromise is hybrid attention: retain some full-attention layers to maintain global information flow, while replacing other layers with sparse attention layers—each token attends only to a local window plus a small number of selected key positions, with a learnable gate determining the sparsity pattern. The open model discussed here follows exactly this approach: starting from the base model, some full-attention layers are replaced with gated sparse attention layers, substantially reducing long-context overhead while preserving long-range modeling capability.

Supporting engineering measures include paged management and quantization of the KV cache, fused optimization of attention operators, and continuous batching scheduling. For teams that want to support ultra-long contexts in production, a combination of these measures is almost mandatory.

3.3 Large-Scale Joint Reinforcement Learning Post-Training

This model underwent large-scale joint reinforcement learning post-training starting from the base model, covering hundreds of downstream domains. Choosing multi-domain joint training over serial single-task fine-tuning is motivated by three factors:

1. Preventing capability collapse. Serial RL in a single domain easily degrades capabilities in other domains; joint training with balanced domain sampling significantly mitigates forgetting.
2. Complementary reward signals. Code and command-line tasks have verifiable rewards (compilation passes, tests pass, commands succeed), while dialogue tasks rely on human preference rewards. Mixing the two improves both “reliability” and “communicability,” which corroborates the simultaneous rise in command execution reliability and satisfaction metrics discussed above.
3. Suppressing reward hacking. Multi-domain joint training increases reward signal diversity and reduces the probability of the model exploiting a single reward function.

Of course, this route is extremely costly: it requires large-scale interactive execution environments, stable reward models, and massive compute. This is also why such post-training is currently concentrated among leading teams—and why open weights let newcomers build on a finished product instead of starting from scratch.

4. Deployment Engineering: The Distance from “Downloadable” to “Runnable”

4.1 Start with the Storage Math

Weight size can be estimated with a minimal formula: size ≈ parameter count × bytes per parameter.

def weight_size_tb(total_params_billion, bytes_per_param):
    return total_params_billion * bytes_per_param / 1000  # decimal TB approximation
total = 2800  # 2.8 trillion parameters ≈ 2800B
print(f"BF16 weights: about {weight_size_tb(total, 2):.1f} TB")    # about 5.6 TB
print(f"INT4  weights: about {weight_size_tb(total, 0.5):.1f} TB") # about 1.4 TB

Even with the most economical INT4 quantization, the weights still total about 1.4 TB. Downloading the full BF16 weights over a 100 Mbps connection (roughly 10 MB/s) takes days; even a gigabit dedicated line takes over ten hours. And that is only the “get the weights” step.

4.2 Now the VRAM Math

Assume data-center-grade accelerators with 80 GB of VRAM each:

• BF16 weights: 5,600 GB ÷ 80 GB ≈ 70 cards, and that only “fits the weights”—KV cache, activations, and runtime overhead are not included;
• INT4 weights: 1,400 GB ÷ 80 GB ≈ 18 cards, again only a lower bound.

Accounting for the KV cache pressure from a 1-million-token context (its VRAM footprint is roughly: 2 × number of layers × number of KV heads × head dimension × sequence length × precision bytes × concurrency), the actual deployment scale rises significantly. A single 24 GB consumer GPU is nearly two orders of magnitude away from the 1.4 TB minimum weight requirement. “Downloaded but can’t run it” is the norm for such models, not the exception.

4.3 Viable Paths to Deployment

For teams that genuinely need self-hosting, there are roughly four realistic paths:

1. Multi-node, multi-GPU clusters with multi-dimensional parallelism. Combine tensor parallelism, pipeline parallelism, and expert parallelism (MoE models also require cross-GPU all-to-all communication). This demands high-bandwidth, low-latency interconnect and strong cluster scheduling capabilities.
2. Quantization and inference optimization. Use low-bit quantization such as INT4/FP8 on the weight side, combined with paged KV cache, continuous batching, and speculative decoding to improve throughput and reduce cost.
3. Leverage general cloud capabilities. If building a cluster is too expensive, choose managed compute from cloud providers—for example, GPU compute instances, container platforms, and object storage on platforms such as Baidu AI Cloud: object storage handles reliable distribution and cache warm-up of weights, the container platform handles orchestration and elastic scaling of inference services, and log services plus monitoring and alerting complete the observability loop.
4. Pragmatic alternatives. For most small and mid-sized teams, calling a managed inference service, using a distilled smaller-parameter version, or quantizing and fine-tuning open weights before deploying to a niche scenario offers a much better return on investment.

It is worth emphasizing: the value of open weights is not equivalent to “everyone runs it locally.” Its greater significance lies in reproducible research, auditable weights, and the ability to fine-tune derivatives within the license—benefits that are often underestimated in enterprise selection processes.

4.4 Evaluation and Selection Recommendations

Leaderboard scores are only a starting point. Before deployment, teams should:

• Build an evaluation set from their own real workloads, focusing on outcome-type metrics such as command execution success rate, task closure rate, number of correction rounds, and cost per task;
• Run long-horizon tasks rather than single-turn Q&A on candidate models, and observe failure modes: is it planning wrong, executing wrong, or failing to accept corrections;
• Factor deployment cost into the evaluation: a model with a slightly lower score but an order of magnitude lower inference cost is often the better engineering choice.

5. Conclusion

The value of this wave of open weights can be understood on three levels. For researchers, open trillion-parameter models provide reproducible, tangible references for architectural innovation (such as gated sparse attention) and training recipes (such as large-scale joint reinforcement learning). For the evaluation community, the methodology is converging toward “anchoring on task outcomes”—from pairwise blind tests to real session signals, from win rate to net improvement rate—and the correlation between leaderboard scores and production performance will only grow stronger. For engineering teams, the orders-of-magnitude gap between “downloadable” and “runnable” reminds us that model selection is a systems engineering problem: open weights are only the first step, and the real barriers lie in parallel inference, quantization optimization, and observability.

It is foreseeable that as open-weight models continue to approach closed-source flagships on agent benchmarks, “leaderboard leadership” will gradually shift from a selling point to the norm. At that point, the real competition will move to deployment cost, depth of domain adaptation, and real-world task closure rate—exactly where developers should be laying groundwork in advance.