Evaluating and Deploying Open-Weight Trillion-Parameter Models for Agentic Workloads

1. Background: Open Weights Enter the Trillion-Parameter Era

The bar for open-weight large models has risen sharply over the past two years. Early open-weight releases were mostly dense architectures ranging from tens of billions to hundreds of billions of parameters, which the community could fine-tune on a single machine and serve on a single GPU. As mixture-of-experts (MoE) architectures matured, open-weight models with trillions of total parameters and hundreds of billions of active parameters began to appear. Some leading open-weight models now approach closed-source flagship performance on terminal-based agent benchmarks.

However, “open weights” is easy to misread. What is released is a trained parameter file, usually accompanied by license terms, and it is not equivalent to unrestricted open-source software. More importantly, there is a wide engineering gap between “downloadable weights” and “runnable locally.” This article focuses on three questions: How are agentic capabilities in trillion-parameter models trained and evaluated? What architectural trade-offs support those capabilities? And when developers actually want to run these weights themselves, how should they calculate the cost?

2. Architecture Breakdown: Decoupling Capacity from Cost

2.1 Sparse Activation: Memory Scales with Total Parameters, Compute Scales with Active Parameters

Take a recently released open-weight trillion-scale model as a reference: roughly 2.8 trillion total parameters, about 104 billion active parameters per forward pass, a sparsity ratio of about 27:1, and support for roughly 1 million tokens of context. The exact numbers vary, but the estimation method applies to all MoE models:

• Memory and storage requirements are determined by total parameters: all experts must reside in GPU memory or fast-access storage, even though each token only uses a small fraction of them.
• Compute per forward pass is determined by active parameters: the router selects a small number of experts for each token, and unselected experts do not participate in computation.

This decoupling of capacity from cost is why MoE has scaled so successfully in the open-weight ecosystem: total parameters can grow to trillions for knowledge capacity, while active parameters stay in the hundreds of billions to preserve inference cost.

2.2 Gated Sparse Attention: Managing Overhead at Million-Token Context Lengths

Full self-attention has two unavoidable costs: computation grows quadratically with sequence length, and KV cache grows linearly with sequence length. At the million-token scale, even with grouped-query attention (GQA) compressing KV heads to single digits, the KV cache for a single request can reach hundreds of gigabytes (see the estimate below), and attention computation becomes even harder to sustain.

A common engineering compromise in the industry is “hybrid attention”: keep a subset of full-attention layers for precise global retrieval, and replace the remaining layers with sparse attention layers where each query interacts only with a selected subset of keys, compressing computation and memory access from full-sequence pairs to a manageable scale. The role of gating is to introduce learnable parameters within each layer that control which blocks information flows from, letting training decide which positions matter rather than fixed rules. The continuous nature of gating also helps training stability after sparsification.

Every developer should understand how to estimate KV cache size:

def kv_cache_gb(num_layers, kv_heads, head_dim, seq_len, batch=1, bytes_per_elem=1):
    """KV elements per token per layer = 2 (K and V) x kv_heads x head_dim"""
    elems = 2 * num_layers * kv_heads * head_dim
    return elems * seq_len * batch * bytes_per_elem / 1024**3
# Example config: 80 layers, 8 KV heads (GQA), head dim 128, FP8 storage, 1M tokens, single request
print(kv_cache_gb(80, 8, 128, 1_000_000))   # about 152.6 GB
# Without GQA, with 64 KV heads, this grows roughly 8x, exceeding 1.2 TB for a single request

As shown, the memory bottleneck for long contexts often lies not in the weights but in the KV cache. This is why sparse attention and low-bit KV quantization have become standard features of long-context models.

2.3 Joint Reinforcement Learning: Where Agentic Capability Comes From

Agentic capabilities such as multi-step planning, tool calling, and error recovery are difficult to learn through supervised fine-tuning alone. SFT teaches a model what a good answer looks like, but agentic tasks require trial and error over long trajectories: how to diagnose a failed command, how to adjust when the user corrects the direction. Leading models therefore increasingly treat reinforcement learning as the main field of post-training.

One noteworthy approach is “joint reinforcement learning”: starting from an existing base model, it does not optimize for a single task or leaderboard, but mixes tasks from hundreds of downstream domains into a unified training distribution and optimizes them simultaneously. Sequential domain-specific training tends to cause catastrophic forgetting and a see-saw effect: code capability improves while writing quality drops. Joint training uses a unified policy to share trial-and-error experience across domains, combined with verifiable rewards such as whether a command executed successfully or whether tests passed, plus human preference signals, to raise the overall floor of agentic capability.

3. Evaluation Paradigms: From Pairwise Blind Tests to Real-Session Signals

3.1 Why Pairwise Blind Testing Falls Short

Mainstream leaderboards have long relied on pairwise blind testing: human evaluators compare two anonymous responses to the same question and vote. This works for judging which answer is more likable, but it has three structural flaws for evaluating agents:

• Preference is not capability: votes are easily swayed by formatting, length, and tone, while the value anchor of an agentic task is whether the task was actually completed.
• Samples are mostly short dialogues: real agentic tasks often involve dozens of turns over a long horizon, which single-turn voting cannot cover.
• The evaluation environment is artificial: tasks constructed by evaluators do not match the workload distribution of real users.

3.2 Session Signals and Causal Tracking

A newer evaluation approach embeds measurement into real usage: extracting outcome signals from real user agent sessions. Explicit confirmations of task completion, praise and complaints within the session, instructions that correct the execution direction, and shell command errors are all natural “outcome labels” carried by the session itself.

Statistically, this resembles an online randomized controlled experiment: requests are randomly routed to different models, with “model” as the treatment variable and session outcome as the response variable. Randomization-based inference then estimates each model’s net improvement rate relative to the population average, expressed in percentage points. Randomization ensures that models face comparable task difficulty and user distributions. A positive net improvement rate means the model outperforms the average. This method is not perfect: confirmation rates are affected by user habits, and long-task samples are sparse. But it answers the question developers care about most: when real users put a model to work, do outcomes actually improve?

3.3 Four Key Metrics

Metric	Signal Source	What It Measures
Shell command reliability	Command exit codes, error messages	Objective success rate of executing operations in a real terminal environment
Praise-to-complaint ratio	Positive and negative feedback expressions in sessions	Subjective user satisfaction
Steerability	Behavior change after corrective instructions	Ability to adjust execution direction based on user feedback
Confirmed success	User explicitly confirms task completion	The outcome signal closest to real delivery quality

For developers, these metrics are more informative than preference win rates. Shell reliability is a purely objective signal with almost no stylistic interference. Steerability determines the quality of multi-turn collaboration: an agent that ignores corrections is hard to use no matter how capable it is. Confirmed success maps most directly to delivery.

4. Deployment Reality: The Distance from Downloadable to Runnable

4.1 Start with the Storage Math

def weight_size_tb(total_params: float, bits: int) -> float:
    return total_params * (bits / 8) / 1e12
P = 2.8e12
for name, bits in [("FP32", 32), ("BF16", 16), ("INT8", 8), ("INT4", 4)]:
    print(f"{name}: {weight_size_tb(P, bits):.1f} TB")
# FP32: 11.2TB | BF16: 5.6TB | INT8: 2.8TB | INT4: 1.4TB

Precision	Bytes per Parameter	Size for 2.8T Parameters
FP32	4	11.2TB
BF16	2	5.6TB
INT8	1	2.8TB
INT4	0.5	1.4TB

The conclusion is straightforward: even with the most aggressive INT4 quantization, the weights still occupy about 1.4TB. A consumer-grade single machine, where GPU memory plus system memory typically totals only a few hundred gigabytes, cannot accommodate them at all.

4.2 Then Calculate Memory and Throughput

Beyond weights, there are three more items to account for: KV cache as discussed in Section 2.2, where a single million-token request reaches hundreds of gigabytes; activation values and framework overhead, usually reserving 20% to 35% of GPU memory; and expert-parallel communication, where tokens must be routed across nodes to the corresponding experts, making interconnect bandwidth a direct determinant of throughput. Node count can be roughly estimated as follows:

import math
def node_estimate(weight_tb, gpu_mem_gb=80, gpus_per_node=8, reserve=0.35):
    usable = gpu_mem_gb * gpus_per_node * (1 - reserve) / 1024  # TB
    return math.ceil(weight_tb / usable)
print(node_estimate(5.6))   # BF16: about 14 nodes with 8 GPUs each
print(node_estimate(1.4))   # INT4: about 4 nodes with 8 GPUs each

For throughput, with 104 billion active parameters, each token requires roughly 2 x 10^11 FLOPs for the forward pass. Assuming effective single-GPU compute in the hundreds of TFLOPS, theoretical single-GPU throughput can reach thousands of tokens per second. After accounting for routing, communication, and memory access bottlenecks, real-world throughput typically lands in the hundreds of tokens per second. To support production-level concurrency, multi-node expert parallelism is almost mandatory.

4.3 Three Practical Paths

1. Cloud API access: for most teams, this is the optimal choice in terms of cost and efficiency. Major cloud providers offer model inference and managed serving. Developers can obtain an inference endpoint through a cloud model service such as Baidu AI Cloud, then build the application-side pipeline with GPU cloud servers, container platforms, and monitoring and alerting, without dealing with weight distribution or cluster operations.
2. Quantized self-hosting: when data privacy genuinely requires it, INT4 quantization plus multi-node expert parallelism is a viable route. Note three points: expert parameters dominate MoE models, so most quantization gains come from experts; calibration sets should match the real task distribution; and KV cache should be quantized to low bit widths in parallel, otherwise KV becomes the new bottleneck.
3. Offloading and tiering: placing some experts in CPU memory and loading them on demand can dramatically lower the GPU memory threshold, but at the cost of an order-of-magnitude drop in throughput. This is only suitable for latency-insensitive scenarios such as offline batch processing.

5. A Selection Checklist for Developers

• Prioritize outcome-based metrics such as shell reliability, confirmed success, and steerability over preference win rates alone.
• Distinguish nominal context length from effective context length, and test recall and consistency with long documents and long sessions.
• Before deployment, calculate the three key costs: weights, KV cache, and framework overhead.
• Read the weight license terms carefully: commercial restrictions, attribution requirements, and redistribution terms vary.
• Pay attention to the maintenance cadence of the weights: snapshot updates, bug fixes, and the maturity of the surrounding toolchain.

6. Conclusion

The emergence of open-weight trillion-parameter models marks the point where the open ecosystem can compete with closed-source flagships on agentic capability. But for developers, two things matter more than a leaderboard ranking. First, understand evaluation: outcome-based metrics predict real-world results better than preference voting. Second, calculate deployment costs clearly: total parameters determine memory, active parameters determine compute, and context length determines KV cache. Once these three items are accounted for, the distance between “downloadable” and “runnable” becomes clear. Rational assessment of both capability and deployment cost is the real sign of a maturing open-weight ecosystem.