Open-Source MoE Model with 2.8 Trillion Parameters, 1 Million Context, and High Agent Capability: A Technical Breakdown

I. What This Release Really Means: Not Just Another Large Model, but an Auditable, Self-Deployable Agent Foundation

The recent release that has drawn significant industry attention is an open-weight Mixture-of-Experts (MoE) model with approximately 2.8 trillion total parameters and around 104 billion activated parameters per inference step. Its significance lies not merely in the parameter count, but in the fact that engineering teams with sufficient infrastructure can now access the weights, inspect the architecture, run inference deployments, perform domain fine-tuning, or audit model behavior.

Unlike previous releases that offered only APIs or showcased benchmark scores, this open-weight approach moves the model from a “black-box capability” to a “verifiable system.” For enterprise users, this provides three direct benefits:

1. Controlled data boundaries: Weights can be deployed on private clusters or trusted environments, which is suitable for scenarios with strict security and compliance requirements.
2. Customizable inference pipelines: Sampling strategies, context truncation logic, and tool-call parsers can be replaced, rather than being locked into fixed interface behavior.
3. Reproducible agent capabilities: In long-horizon tasks, whether the model remains stable or drifts from its objectives can now be tested on internal evaluation sets, instead of relying only on demo videos or leaderboard rankings.

Models of this type typically adopt a Mixture-of-Experts structure to maintain a large total capacity while reducing the number of activated parameters per inference step. The design goal is explicit: to deliver more stable long-context performance in agentic tasks such as code generation, command execution, file editing, and web operations, rather than merely chasing scores on traditional Q&A benchmarks.

---

II. Model Architecture and Computational Characteristics: Why 2.8T Parameters and 104B Activated

2.1 Total Parameters vs. Activated Parameters

This model has roughly 2.78 trillion total parameters, but not all parameters are activated for each generated token. It uses a typical sparse MoE architecture where each token activates only a subset of experts, resulting in approximately 104 billion activated parameters per token.

This design is critical from an engineering perspective. A larger total parameter count increases model capacity, allowing more knowledge, reasoning patterns, and tool-use experience to be stored. A lower activated parameter count reduces per-token computation and memory pressure. In other words, the design explicitly decouples “knowledge capacity” from “inference cost.”

2.2 Layer and Attention Structure

According to the model card, the network consists of 93 layers in total:

• 69 layers use a form of attention with more compact key-value caching, which helps manage cache memory under long contexts.
• 24 layers use gated sparse attention, designed to further reduce the cost of modeling long sequences.

Compared with the previous generation, the main structural change is the replacement of some full-attention layers with sparse attention layers. This is not a simple precision trade-off; it is a structural rebalancing aimed at the 1-million-token context scenario. In tasks such as long-document analysis, large codebase comprehension, and long task trajectories, the model does not need to maintain the same level of attention strength for every historical token.

For users, this means that deployment scripts written for a “93-layer full-attention” assumption may produce inaccurate cache size estimates and GPU memory allocation. The actual key-value cache size, inference speed, and memory usage will be affected by the ratio of sparse attention layers.

2.3 Expert Layers and Vision Module

The expert layers are configured with 896 experts in total. Each token selects 16 experts for computation, along with 2 shared experts. The shared experts are designed to carry cross-task general knowledge, reducing redundant storage across different experts and improving parameter utilization.

The vision component uses a vision encoder with approximately 401 million parameters, supporting image-text input. It is important to note that although the vision encoder is far smaller than the language backbone, it plays a critical role in multimodal agent tasks. In scenarios such as screenshot understanding, page-state assessment, and chart reading, the stability of the vision encoder directly determines whether subsequent actions are accurate.

---

III. How the Weights Are Stored: Quantization-Aware Training and Low-Bit Formats

3.1 Storage Format Choice

The model weights are stored in MXFP4 format, with activations dynamically using MXFP8. The key here is not simple “compression,” but “quantization-aware training.” In other words, the model was trained with the error introduced by low-precision formats already taken into account, rather than applying post-training quantization to float weights after training is complete.

This approach is common for extremely large models. A model with close to 2.8 trillion parameters would be enormous if stored in traditional 16-bit floating-point format. With low-bit storage, the entire repository is about 1.56 TB. While this still exceeds the capacity of an average single-machine hard drive, for teams with high-speed object storage or distributed file systems, it is now within a manageable range.

3.2 Relationship Between Parameter Count and Repository Size

With roughly 2.78 trillion parameters and a 1.56 TB repository, each parameter occupies about 0.56 bytes on average. This is consistent with a fused format of “4-bit storage plus some auxiliary tensors.” During actual deployment, weights often need to be restored to a higher precision format after loading into GPU memory, so runtime memory usage will be significantly larger than 1.56 TB.

This means hardware planning cannot be based solely on repository size. A common misconception is that “downloading 1.56 TB and preparing 2 TB of GPU memory will be sufficient.” In practice, weight dequantization, activation values, key-value caches, and optimizer states or gradients during inference will all add significantly to memory requirements.

---

IV. Deployment Methods and Hardware Requirements

4.1 Inference Framework Support

Several mainstream inference engines have already provided deployment options. Common ones include:

• vLLM: Designed for high-throughput inference, supporting tensor parallelism, pipeline parallelism, and expert parallelism.
• SGLang: Suitable for scenarios requiring complex scheduling and structured generation.
• TokenSpeed: A high-speed inference framework optimized for long-context and agent workloads.

In practice, you should not simply pick a framework based on popularity. The choice depends on the nature of your workload:

• If the primary workload involves short requests with high concurrency, a throughput-oriented framework is preferable.
• If the workload involves single long tasks, frequent tool calls, and precise control over generated structure, you should prioritize scheduling flexibility and state management capabilities.

4.2 Hardware Estimation

The deployment barrier for this model is high; it is not something a single-GPU or dual-GPU server can handle. Third-party estimates suggest that one feasible configuration involves 8 next-generation high-performance compute cards, each with around 288 GB of memory, providing roughly 2.3 TB of total GPU memory per machine. Such a server could cost in the range of $550,000 to $650,000.

These are third-party estimates, not official recommendations. Actual requirements vary significantly based on concurrency and context needs:

• If you only deploy the quantized version and limit concurrency to 1, a lower configuration may suffice.
• If you need to serve dozens of long-context agent sessions simultaneously, a single 8-GPU server may not be enough.
• If you only run offline batch inference, the hardware selection will differ from online serving.

When planning budgets, it is advisable to estimate GPU memory and bandwidth requirements based on “target concurrency + average context length + acceptable latency,” rather than looking only at the parameter count.

4.3 GPU Memory Planning Considerations

For deployment planning of a 2.8-trillion-parameter model, at least the following components must be accounted for:

1. Weight memory: Stored in 4-bit format, but may be restored to higher precision during inference.
2. Activation memory: Activations use 8-bit format; larger batch sizes increase memory usage.
3. Key-value cache: Under long contexts, this becomes one of the primary bottlenecks. Sparse attention reduces some cache pressure, but cache management complexity increases.
4. Runtime temporary buffers: Includes expert routing computation, communication buffers, and sampling intermediates.

A general recommendation is that total GPU memory should be several times the repository size. For a 1-million-token context scenario, cache memory can be an order of magnitude higher than for typical chat models.

---

V. If You Don’t Want to Deploy Yourself: API Integration and Tool-Call Considerations

5.1 Model Name and API Compatibility

For teams that do not want to maintain a large-scale cluster, the model is available through an open platform API. The corresponding model name typically includes the pro identifier. The API is compatible with two common chat protocols: OpenAI-style and Anthropic-style. A high-speed variant is also available, with the official claim of approximately 2 to 3 times the speed of the standard version, suitable for latency-sensitive but simpler tasks.

5.2 Reasoning Effort Control

The model provides multiple reasoning effort levels, typically including:

• low: Lower reasoning effort, faster responses.
• high: Higher reasoning effort, suitable for more complex tasks.
• max: Maximum reasoning effort, which is the default setting.

Users should select the appropriate level based on task complexity rather than always using max. For example, simple format conversion, rewriting, or factual Q&A may be better served by low or high, while multi-step planning, code debugging, and mathematical derivation may warrant switching to max.

5.3 State Passing in Multi-Turn Tool Calls

This is one of the most error-prone areas in agent development. When a model runs through multiple rounds of tool calls, the tool results are not the only state that needs to be passed back. The reasoning content and tool-call information from the previous round must also be passed back; otherwise, the model may lose its intermediate reasoning path, causing inconsistencies in subsequent actions.

A recommended practice is to maintain a session state object on the client side that includes at least:

messages
├── user message
├── assistant message
│   ├── content
│   ├── reasoning_content
│   └── tool_calls
├── tool result
└── next assistant message

In the next round of the request, reasoning_content and tool_calls should be preserved and submitted along with the messages. This design often has a more significant impact on agent stability than adjusting sampling parameters.

---

VI. Performance in Agent Scenarios: Net Improvement in Long-Horizon Sessions

6.1 Evaluation Methodology

Traditional benchmarks use fixed test sets and score based on how closely answers match a gold standard. Agent tasks, however, are difficult to measure with a single answer. For example, “help me run the tests in this repository and fix the failing cases” may result in a series of file and command operations, not a comparable text output.

A more realistic evaluation approach is to collect real user long-horizon sessions and extract task success signals, such as:

• Whether the user explicitly confirmed task completion.
• Whether the user expressed approval or dissatisfaction with the model.
• Whether the model improved on the next attempt after user correction.
• Whether shell commands executed successfully.

By aggregating these signals, a “net improvement” metric can be calculated. This evaluation method reflects the model’s stability in real agent workflows more accurately than single-turn accuracy.

6.2 The Model’s Position in Evaluation

In an agent session evaluation published on the same day as the weight release, this model ranked 1st among open-weight models and 13th on the overall leaderboard. Its net improvement was approximately +5.1%, whereas the previous generation was around +0.91% and ranked 6th among open models.

By subcategory, the most notable improvement was in command-line reliability. In agent tasks, models frequently need to write shell commands, run them, and observe results. If a command repeatedly fails, it significantly lowers task completion rates. This model achieved +7.9% on Bash reliability, whereas the previous generation was negative. This indicates meaningful progress in the “command generation → observation → correction” loop.

In the code, chat, and work categories, this model ranked 1st among open models. It is worth noting that a gap still exists between open and closed-source models, so 13th place overall is a more objective position. For enterprises, the main value of this data is not promotional ranking, but rather using it as one of several reference dimensions for internal model selection.

---

VII. Practical Recommendations: Who Benefits from This Release

7.1 Teams That Should Consider Direct Deployment

The following types of teams are better suited for local deployment:

• Teams with multi-GPU high-performance servers or existing large-scale inference clusters.
• Teams handling sensitive data that cannot be sent to external APIs.
• Teams that need deep modifications above the model layer, such as custom tool-call protocols or domain fine-tuning.
• Teams with clear business scenarios for long-context agent tasks, such as automated operations, codebase-level development assistance, or large document analysis.

7.2 Teams That Should Use the API

The following situations are more suitable for API usage:

• The business is still in the validation phase and cannot yet justify high hardware costs.
• Request volume fluctuates significantly, making self-hosted clusters wasteful.
• The team lacks experience operating large-scale inference clusters.
• The primary need is to quickly integrate agent capabilities rather than deeply customize the model.

7.3 Pre-Deployment Checklist

Before actually starting deployment, it is recommended to confirm the following:

1. Whether sufficient GPU memory capacity and memory bandwidth are available, not just disk space.
2. Whether the inference framework supports the required parallelism strategies and quantization formats.
3. Whether logging, monitoring, and alerting systems can cover long-task failure scenarios.
4. Whether the client correctly maintains intermediate state during multi-turn tool calls.
5. Whether a baseline testing plan exists to verify that the deployed model has not significantly degraded.

---

VIII. Summary

The release of this 2.8-trillion-parameter open-weight model marks a shift in high-capability agent models from closed services to a stage where they are obtainable, deployable, and auditable. Its core value is not the parameter count itself, but the fact that it provides engineering teams with a fully controllable, strong agent foundation.

From a technical perspective, its MoE architecture, low-bit quantization-aware training, long-context sparse attention design, and stability optimizations for agent tasks form a relatively complete engineering solution. For enterprise developers, the key is not to rush into deployment but to choose between self-hosted inference or API integration based on business needs, data compliance requirements, and existing infrastructure. The immediate practical priorities should be solving tool-call state management and long-task stability issues.