Open Weights for a Trillion-Parameter Mixture-of-Experts Model: Architecture, Evaluation Paradigm, and Deployment Realities

1. Background: What Open Weights Mean for Ultra-Large Models

In recent years, the capability frontier of large models has expanded continuously, with parameter counts moving from the tens of billions toward the trillions. Along the way, “open weights” has become an important measure of a model ecosystem’s influence. Open weights means that once training is complete, the model’s weight files are published for researchers and developers to download and use, rather than being available only through an API.

For trillion-parameter Mixture-of-Experts (MoE) models, open weights carry particular significance: they allow academia and industry to study the internal mechanisms of ultra-large sparse models, and they give enterprises a foundation for further development in specialized domains. However, between open weights and practical deployment there remains a gap defined by storage, VRAM, and inference cost.

This article examines the open-weight release of an MoE model with 2.8 trillion total parameters, analyzing it across four dimensions: architecture design, training methodology, evaluation system, and deployment barriers. It is intended as a reference for readers focused on the engineering of ultra-large models.

2. Model Architecture: Co-Design of Mixture-of-Experts and Sparse Attention

2.1 Fundamentals of Mixture-of-Experts

The core idea of MoE architecture is “sparse activation”: although the model holds a massive number of parameters, only a small fraction is activated during each inference pass. Specifically, an MoE layer consists of multiple parallel “expert” sub-networks and a gating network. Based on the features of each input token, the gating network decides which experts the token is routed to for computation.

The direct benefit of this design: total parameter count can be scaled enormously, improving capacity and knowledge storage, while the compute per inference (FLOPs) is proportional only to the number of activated parameters and does not grow linearly with total parameters.

This model has 2.8 trillion total parameters and activates roughly 104 billion parameters per inference—an activation ratio of about 3.7%, a highly sparse MoE configuration. From an engineering perspective, this ratio must balance model capacity against inference efficiency: too low an activation ratio can lead to uneven routing load and reduced expert utilization; too high a ratio forfeits the cost advantage of sparsity.

2.2 Long Context and Gated Sparse Attention

The model supports a context window of roughly 1 million tokens. Under standard full attention, attention compute complexity grows quadratically with sequence length, making long-context modeling extremely expensive.

To reduce this overhead, the model makes a key architectural change: some full-attention layers are replaced with gated sparse attention layers. Sparse attention means each token attends only to a subset of historical tokens rather than interacting with all of them. The gating mechanism dynamically decides which connections to retain, significantly lowering compute while preserving modeling quality.

This change matters especially for agent scenarios. Agent tasks typically involve long-horizon, multi-turn interaction, tool calls, and accumulating context, and context length can easily exceed a conventional window. Sparse attention makes inference cost and latency more manageable for such long-horizon tasks.

2.3 Key Parameters at a Glance

Metric	Value
Total parameters	2.8 trillion
Activated parameters per inference	~104 billion
Context window	~1 million tokens
Architecture type	Mixture-of-Experts + gated sparse attention
Weight storage (BF16)	~5.6 TB
Weight storage (INT4 quantization)	~1.4 TB

3. Training Methodology: Large-Scale Joint Reinforcement Learning Post-Training

The model was not trained from scratch. Instead, it underwent large-scale joint reinforcement learning post-training on top of an existing base model. This strategy is increasingly common in ultra-large model development: the base model has already acquired general language understanding and generation through pre-training, and the post-training stage focuses on aligning with human preferences and improving performance on specific tasks.

The “joint” in “joint reinforcement learning” has two aspects:

First, coverage of hundreds of downstream domains. Post-training data is not concentrated on a single task type but spans work, coding, chat, tool calling, and many other domains. The goal of this multi-domain joint training is to avoid overfitting in one direction, improving specialized performance while preserving general capability.

Second, multi-objective joint optimization. The reinforcement learning reward signal is not single-dimensional; it combines task completion, user satisfaction, instruction-following accuracy, and other objectives. How to weight multiple objectives and avoid reward hacking is a core engineering challenge of the post-training stage.

Evaluation results show that this post-training strategy delivered clear gains: in agent task scenarios, the model’s net improvement rate rose 4.2 percentage points over the previous generation, with significant improvements in sub-metrics including Shell command execution reliability, user satisfaction, and steerability.

4. Evaluation Paradigm: From Blind Voting to Causal Tracing

4.1 Limitations of Traditional Evaluation

Traditional large-model evaluation often relies on pairwise blind tests: users are shown responses from two models and vote for the better one. This works reasonably well for chat scenarios but has clear shortcomings when assessing agent capabilities:

• Agent tasks are often long-horizon; the quality of a single response cannot reflect overall task completion;
• Real user feedback is reflected not only in “which answer is better” but also in behavioral signals such as whether the task was completed, whether correction was needed, and whether execution errors occurred;
• Voting data is sparse and hard to cover long-tail task types.

4.2 Evaluation Based on Real Session Signals

This leaderboard takes a different approach: it does not ask users for extra votes but extracts signals from real user agent sessions. These signals include:

1. Explicit confirmation: the user explicitly states the task is complete or the result is correct;
2. Praise and complaints: positive or negative feedback on model performance;
3. Correction instructions: the user asks the model to change direction or re-execute;
4. Shell command execution errors: records of failed command execution in tasks involving command-line operations.

On this basis, the evaluators use a randomized inference method called “causal tracing” to estimate each model’s net improvement rate relative to the average. The basic logic: through randomized model assignment (users cannot choose which model handles a task), differences in task outcomes are attributed to the model itself rather than to user preference or task difficulty.

4.3 Scale and Representativeness of Evaluation Data

The ranking is based on more than 17,500 real long-horizon agent task sessions from users worldwide. This data scale is substantial for agent evaluation and covers categories including work, coding, and chat.

In the results, the model ranks first among open-weight models, 13th among all evaluated models, and first among open models in the work, coding, and chat categories. On sub-metrics:

• Bash reliability: improved from -4.5% (13th place) in the previous generation to +7.9%, ranking first among open models;
• Praise-to-complaint ratio: improved from +1.3% to +7.7%;
• Steerability (the ability to adjust execution direction based on user feedback): reached +6.5%, first among open models;
• Confirmation success rate: +3.9%, fourth among open models.

These sub-metrics reveal the specific dimensions of improvement: higher Shell command execution reliability indicates more stable tool calling and system interaction; higher steerability indicates the model better understands user correction intent and adjusts behavior accordingly.

5. Deployment Realities: Open Weights Do Not Mean Local Execution

5.1 Storage and VRAM Barriers

The most common misunderstanding about open weights is that downloadable weights mean the model can run locally. For trillion-parameter models, this assumption does not hold.

Stored in BF16 precision, the weights of a 2.8-trillion-parameter model total about 5.6 TB. Even with INT4 quantization (compressing each parameter to 4 bits), the weight volume is still about 1.4 TB. By comparison, current mainstream consumer GPUs have 8 GB to 24 GB of VRAM, and even multi-GPU single-machine configurations struggle to reach 1.4 TB of total VRAM.

This means local deployment of the model requires at minimum:

• Multiple high-VRAM servers forming an inference cluster;
• High-speed inter-GPU interconnect (such as NVLink or high-speed Ethernet);
• A dedicated inference framework supporting expert parallelism and quantized inference.

For the vast majority of developers and small to mid-sized enterprises, the more realistic path is to use the model through a cloud API or hosted service rather than self-deploying.

5.2 Possible Directions for Inference Optimization

Although the local deployment barrier is very high, the industry already has several general directions for optimizing inference of ultra-large MoE models:

Expert parallelism: distributing different experts across different devices, with each device handling computation for only a subset of experts, completing token routing through All-to-All communication. This is the core technique for distributed inference of MoE models.

Quantization compression: beyond INT4, more aggressive quantization schemes (such as INT2 or mixed-precision quantization) can further compress weight volume but may introduce accuracy loss, which needs to be evaluated on specific tasks.

Offloading and caching: offloading inactive expert weights to CPU memory or NVMe storage and loading them into VRAM only when needed. This approach can reduce VRAM requirements but increases inference latency.

Speculative decoding: using a small model to generate candidate tokens, then verifying them with the large model to improve inference throughput. This method also applies to MoE models.

6. Terminal Agent Benchmark Performance

On a terminal agent benchmark, the model scores 65.7%. For reference, in the same setting another closed-source model scores 58.2%, and a leading closed-source model scores 66.4%. This means:

1. The model’s performance on terminal agent tasks already exceeds some closed-source models;
2. The gap to the top closed-source model is less than 1 percentage point;
3. Given its open-weight nature, this result is significant for the open-source ecosystem.

Terminal agent tasks typically involve executing operations in a command-line environment, reading and writing files, installing dependencies, and debugging errors. Such tasks place extremely high demands on instruction following, tool-call accuracy, and error recovery. A score of 65.7% indicates the model can complete tasks in most scenarios, but roughly one-third of scenarios still involve failure or require human intervention.

7. Practical Guidance for Developers

7.1 Model Selection Considerations

For teams considering ultra-large MoE models, evaluate along the following dimensions:

• Task type: the model performs strongly on agent, tool-calling, and long-context tasks, making it suitable for scenarios requiring multi-turn interaction and external tool integration;
• Deployment mode: without a self-built cluster, prefer cloud-hosted services;
• Cost structure: compare API call costs against the TCO of a self-built inference cluster; the latter usually has an advantage only at very high request volumes;
• Data privacy: for sensitive data, assess the cloud service’s data processing agreement.

7.2 Agent Application Development Recommendations

# Generic pseudocode structure for an agent task loop
def agent_loop(task, model, tools, max_steps=20):
    context = initialize_context(task)
    for step in range(max_steps):
        # Model generates the next action
        action = model.generate(context, tools)
        # If a tool call, execute and get the result
        if action.type == "tool_call":
            result = execute_tool(action.tool_name, action.args)
            context.append(result)
        # If a final answer, return the result
        elif action.type == "final_answer":
            return action.content
    return "Task exceeded maximum step limit"

In practice, pay particular attention to:

• Error recovery mechanisms: when a tool call fails, the model should recognize the error and try alternatives;
• Context management: in long-horizon tasks, compress and summarize historical context reasonably to avoid exceeding the window limit;
• Steerability design: provide users with interfaces to interrupt and correct, and the model should understand and respond to such feedback.

7.3 Evaluation and Iteration

When evaluating the model in your own business scenarios, build test sets that closely reflect real tasks rather than relying only on general leaderboards. Evaluation metrics should cover:

• Task completion rate;
• Average number of interaction turns;
• Tool-call accuracy;
• User correction frequency;
• End-to-end latency.

8. Summary

The open-weight release of a trillion-parameter MoE model is an important milestone in ultra-large model development. It demonstrates the ability of sparse architectures to balance parameter scale against inference efficiency, and validates the effectiveness of large-scale joint reinforcement learning post-training across multi-domain tasks.

At the same time, a weight volume of 5.6 TB and 1.4 TB after quantization clearly draw the line between open weights and local deployability. For the vast majority of developers, cloud hosting remains the primary way to use such models.

On evaluation methodology, agent evaluation based on real session signals and causal tracing offers a more realistic perspective for assessing model capability on long-horizon tasks. Its value lies in not relying on manually annotated preference data but extracting signals from real usage behavior, bringing it closer to production performance.

For enterprise technology decision-makers, the focus should shift from “how big is the model” to “how does the model perform on specific tasks, is the deployment cost acceptable, and does it meet data compliance requirements.” Open weights lower the barrier to research and secondary development, but engineering deployment still requires comprehensive consideration of cost, latency, and operational complexity.