The trillion-parameter open-weight model: architecture, evaluation, and the real cost of deployment

Introduction: When “open weights” becomes a commitment

A major model development team recently released the full weights of its flagship model on an open model community, honoring a timeline announced at launch. The model has 2.8 trillion total parameters, uses a Mixture-of-Experts (MoE) architecture, activates approximately 104 billion parameters per inference, and supports a context window of roughly 1 million tokens. In third-party agent evaluation leaderboards, it ranks first among open-weight models and 13th across all evaluated models. It also takes the top open-model position in the work, coding, and chat categories.

The noteworthy part of this result is not the “top spot” itself, but the fact that it brings three long-conflated questions into focus:

• How can trillion-parameter models be used efficiently, rather than expensively left idle?
• How should agent capabilities be evaluated, so that “machine ability” is not held hostage by “human preference”?
• How far apart are “downloadable weights” and “locally runnable models,” really?

The following sections examine each question in turn.

Architecture breakdown: A user manual for 2.8 trillion parameters

1. Mixture of Experts: Separating total parameters from activated parameters

The core idea of MoE is to replace the dense feed-forward layers of a traditional Transformer with multiple “expert” subnetworks, and use a router to dynamically select a small number of experts for each token. This creates a key decoupling: the model’s knowledge capacity is determined by total parameters, while the computational cost of a single inference is determined only by activated parameters.

For this model, the activation ratio is approximately 104B / 2.8T ≈ 3.7%. In other words, for every token generated, the vast majority of parameters remain dormant. A useful analogy: an encyclopedia with 2.8 trillion words, where each query turns only about 3.7% of the pages — the capacity is virtually unbounded, but retrieval cost stays roughly constant.

total, active = 2.8e12, 1040e9
print(f"Activation ratio: {active / total:.2%}")   # ≈ 3.71%

The trade-off, of course, is that GPU memory usage scales with total parameters, which sets the stage for the deployment challenges discussed later.

2. Gated sparse attention: Reducing the burden of million-token contexts

The computational complexity of self-attention scales quadratically with sequence length. At a 1M-token context, full attention layers quickly overwhelm inference budgets. The model’s approach: retain some full attention layers for precise global modeling, while replacing other layers with gated sparse attention layers, where a learnable gate determines which subset of key-value pairs each query attends to. This keeps the cost of long-context modeling manageable.

This “hybrid attention” design is becoming a common technical direction for long-context models: full attention handles global dependencies in a few critical layers, while sparse attention handles efficient selective dependencies in most layers, striking a balance between capability and cost.

3. Post-training: From “chatting” to “doing”

Compared to the previous generation of dialogue-focused base models, this model uses the K3 base and undergoes large-scale joint reinforcement learning post-training across hundreds of downstream domains. This is especially important for agent scenarios: code execution, terminal operations, and tool invocation are inherently suited to “verifiable rewards” — whether a command executes successfully, whether a test passes, whether a task is confirmed by the user. These make excellent signals for reinforcement learning. This also explains the notable jump in terminal-related metrics in the evaluation results below.

Evaluation methodology: From paired blind tests to causal tracing in real sessions

1. Limitations of paired blind tests

Traditional leaderboards rely on paired blind tests: users are shown two anonymous answers to the same question and vote for the better one. This approach measures “human preference” reasonably well, but it suffers from two systematic biases. First, preference is not the same as capability — an answer that “looks good” is not necessarily one that “did the right thing.” Second, user votes typically occur in short dialogues, making it difficult to cover the long-horizon tasks common in real agent work, which often span dozens of turns and interleave tool calls.

2. Extracting signals from real sessions

The new generation of agent evaluation takes a different approach: rather than asking users to vote, it extracts behavioral signals from real user-agent sessions. A typical signal system is as follows:

Signal type	Typical example	Capability metric
Task confirmation	“OK, proceed with this plan”	Task confirmation rate
Praise / complaint	“Well done” / “Wrong again, redo it”	Satisfaction comparison
Correction instruction	“Not this file, find it again”	Guidance receptivity
Shell execution error	Non-zero exit code, command not found	Shell command reliability

3. Causal tracing: Estimating net improvement rate

On top of these signals, the evaluation team applies a randomized inference approach called “causal tracing.” The treatment variable is approximated as which model handled a given user session (treated as quasi-random). Through randomized inference, the method estimates each model’s net improvement rate relative to the overall average, rather than a simple win rate. Conceptually, this can be understood as:

def net_improvement(model_scores, all_scores):
    """Net improvement rate (illustrative): offset of a model's average session signal vs. the global average"""
    mu = sum(all_scores) / len(all_scores)
    m = sum(model_scores) / len(model_scores)
    return m - mu

The real implementation is considerably more complex — it must control for confounders such as task difficulty, user demographics, and session length — but the idea is clear: move evaluation from “spectator voting” to “participant behavior.”

4. How concrete is the improvement under this methodology?

For this model, the ranking is based on more than 17,500 real long-horizon agent task sessions from global users:

• Overall net improvement rate: +5.1%, up 4.2 percentage points from the previous generation’s +0.9%.
• Shell command reliability: improved from −4.5% (mid-to-low tier) to +7.9%, ranking first among open-weight models. This is a direct result of reinforcement learning post-training applied to verifiable tasks.
• Satisfaction comparison: improved from +1.3% to +7.7%.
• Guidance receptivity: reached +6.5%, also first among open-weight models.
• Task confirmation rate: +3.9%, fourth among open-weight models.

On a public terminal-agent benchmark, the model scores approximately 65.7%, surpassing several top closed-source flagship models, and trails the leaderboard’s highest score by less than one percentage point. For the open-weight ecosystem, this marks a shift in the “open-source catching up to closed-source” narrative — from dialogue ability to agent execution capability.

“Downloadable” vs. “runnable”: A matter of arithmetic

1. Storage: Calculate the disk first

Open weights are, first and foremost, files. Their size can be computed precisely:

def weight_size_tb(params_trillion: float, bits: int) -> float:
    """Weight size in TB (decimal) = parameter count × bits per parameter / 8"""
    return params_trillion * 1e12 * bits / 8 / 1e12
print(f"BF16: {weight_size_tb(2.8, 16):.1f} TB")  # 5.6 TB
print(f"INT4: {weight_size_tb(2.8, 4):.1f} TB")   # 1.4 TB

At BF16 precision, 2.8 trillion parameters require about 5.6 TB of storage; even quantized to INT4, roughly 1.4 TB is needed. The download stage alone demands careful consideration of disk capacity and network bandwidth — this is not a size that fits comfortably on a consumer-grade drive.

2. Inference: The triple threshold of memory, bandwidth, and KV cache

Storage is only the first hurdle. During inference, weights must reside in GPU memory, and a 1M-token context adds significant KV cache pressure. Its size can generally be estimated with the following formula:

KV Cache ≈ 2 × num_layers × num_kv_heads × head_dim × sequence_length × bytes_per_element

Under very long contexts, this easily reaches hundreds of gigabytes (exact figures depend on model configuration). Combined with the weights themselves, deployment on a single consumer GPU is not feasible. Realistic deployment requires multi-node, multi-GPU clusters with tensor parallelism, pipeline parallelism, expert parallelism, and quantization. Alternatively, teams may skip self-hosting entirely and use API access.

3. Deployment path recommendations for enterprises

• Research and learning scenarios: Download the weights for architecture analysis, expert behavior studies, or small-scale distillation. The value is clear and cost remains manageable.
• Production scenarios: Prioritize managed cloud inference. GPU compute instances, container platforms, object storage, and model-serving products cover the full chain from weight hosting and elastic inference to monitoring and alerting — teams can access models at this scale without building their own clusters.
• Compromise scenarios: For teams with sensitive data and available compute, a self-hosted setup with INT4 quantization and multi-GPU inference is viable. However, memory and throughput must be pre-calculated using the formulas above to avoid the awkward situation of “downloaded but unable to run.”

Three key takeaways

1. Evaluation signals determine optimization direction. Rather than chasing leaderboard rankings, adopt the idea of “real-session signals” by instrumenting your own workflows to collect task confirmations, correction instructions, tool execution errors, and other signals. Build your own evaluation loop. Leaderboards tell you who is strong; telemetry tells you where you are weak.

2. Open weights are an option, not an immediate asset. “Open” lowers the barrier to access, not the barrier to use. The deployment cost of trillion-parameter models means that, for the vast majority of teams, their value lies in advancing the ecosystem and raising the capability baseline — not in local privatization.

3. The long-context race has shifted from “whether” to “how expensive.” Architectural innovations like sparse activation and hybrid attention are all answering the same question: how to decouple capability growth from cost growth. This trend will continue to define the design standards of next-generation foundation models.

Conclusion

From the on-time release of weights, to causal-tracing evaluation based on real sessions, to the sober arithmetic between 5.6 TB and 1.4 TB, this case clearly illustrates the shift underway in the large model industry: competition is no longer just about parameter counts and benchmark scores, but about “net improvement in real tasks” and “unit cost of capability.” For developers, understanding these three dimensions will deliver more long-term value than memorizing any leaderboard position.