2.8T-Parameter Sparse MoE Model Weights Released: Deployment Requirements and Agent Capabilities Explained

1. Background: A High-Profile Open-Weight Release

A leading foundation model research team has released the complete weights of an ultra-large sparse Mixture-of-Experts (MoE) model, right on the schedule it had promised earlier. The model totals roughly 2.8 trillion parameters, activates about 104 billion parameters per forward pass, supports 1M-token context windows, and accepts image-and-text input. Landing exactly on the committed timeline, the release reflects the team’s deliberate control over its open-source cadence.

For developers, the core value here is not simply “one more downloadable model.” The release offers a rare chance to observe and reproduce the entire pipeline behind large-scale MoE training, quantization, and deployment. For teams working on fine-tuning, inference acceleration, model auditing, and Agent systems, having the full weights means moving from black-box API calls to white-box analysis.

On release day, a third-party Agent evaluation platform also published new ranking data. The model placed first in the open-weight category and thirteenth overall — results derived from statistics on real users’ long-horizon Agent sessions, which gives them strong practical reference value. Taken together, the weight release and the evaluation results send two signals: the model’s capabilities are at the leading edge of the open-weight field, and it has a deployable, verifiable, and reproducible engineering foundation.

2. Architecture: From Attention Layers to Expert Routing

Understanding a 2.8T-parameter model starts with its structure. According to the model card, the network has 93 layers: 69 use a KDA structure and 24 use a Gated DSA structure. The most significant change from the previous generation is that full-attention layers have been replaced with a gated sparse attention mechanism, which reduces the computational cost of long-context modeling.

2.1 Two Attention Layer Designs

The KDA and Gated DSA layers serve different roles. KDA carries forward the previous generation’s efficient attention approach, while Gated DSA introduces a gating mechanism and a sparse attention pattern. In sparse attention, the model no longer computes attention scores between every pair of tokens when processing long sequences. Instead, a learnable gating strategy selects the important regions, cutting computational complexity from roughly quadratic to near-linear or lower.

This “hybrid attention layer” design is not about architectural novelty. It serves one clear goal: under the 1M-token context constraint, preserve long-range dependency modeling while keeping inference costs from spiraling. For Agent tasks that involve long documents, long conversations, or long logs, this design has direct engineering value.

2.2 Expert Scale and Routing Strategy

On the expert side, the model configures 896 experts. Each token activates 16 of them, plus 2 shared experts that always participate in computation — meaning 18 experts actually take part in computing each token during a forward pass. This is a large configuration by MoE standards: more experts mean more model capacity, but also stricter requirements for routing stability and load balancing.

Shared experts typically carry general language knowledge and cross-domain base representations, while routed experts handle finer-grained semantic patterns. The routing ratio — 16 out of 896 — is about 1.8%, which is relatively low and indicates high sparsity at inference time. Higher sparsity reduces per-token compute, but it also puts more pressure on the inference framework’s expert scheduling and memory management.

2.3 Vision Module

The vision side uses a small vision encoder with roughly 401 million parameters that works alongside the language backbone to support mixed image-text input. For common Agent scenarios such as screenshot understanding, chart parsing, and UI element localization, this module can contribute directly. Because the encoder is small, fine-tuning and deployment do not require dedicating extra resources to the vision part.

3. Quantization and Storage: The MXFP4/MXFP8 Combination

The model uses two numeric formats across storage and inference: weights are stored in MXFP4, while activations are computed in MXFP8. According to the developer, this scheme was achieved through quantization-aware training — the impact of quantization error on model quality was accounted for during training, rather than applied as a post-hoc quantization step.

3.1 Why Separate Weight and Activation Quantization

Weights and activations differ in how sensitive they are to numerical precision during inference. Weights stay relatively static and are enormous in number, which makes them suitable for lower-bit compression. Activations are recomputed at every layer and are more sensitive to precision loss — especially in deep networks, where errors accumulate layer by layer. The combination of 4-bit weights and 8-bit activations is therefore a balance between storage efficiency and computational accuracy.

MXFP itself is a floating-point format based on microscaling. Unlike traditional fixed-point quantization, it extends the dynamic range through shared scale factors while keeping the bit width small. For MoE models with uneven parameter distributions, this format fits better than simple uniform quantization.

3.2 Repository Size vs. Actual Parameter Count

According to the model hosting platform, the repository occupies about 1.56TB of storage, and the safetensors files account for roughly 2.78 trillion parameters. The correspondence between 1.56TB and 2.78T parameters confirms the storage efficiency of MXFP4: about 0.56 bytes per parameter on average.

Teams preparing to download the model should plan storage resources in advance. Multi-node parallel downloads or resumable transfers require confirming the storage system’s throughput capacity. In cloud environments, evaluate the storage volume’s IOPS ceiling first to avoid write bottlenecks during the download.

3.3 Inference Framework Support

Several mainstream high-performance inference frameworks already provide deployment configurations for this model. They cover different scenarios: some focus on large-scale distributed inference, some emphasize the ease of single-node deployment, and others target specific hardware acceleration. For developers, the choice among them should weigh the hardware environment, concurrency requirements, and operational capabilities.

4. Deployment Threshold: From Hardware Cost to Feasibility

Deploying an ultra-large MoE model is not as simple as “download the weights and start it up.” Even with 4-bit quantization, the weights alone require about 1.5TB of GPU memory or system memory. Add activations, the KV cache, and the inference framework’s overhead, and the memory needed for single-node deployment clearly exceeds the size of the weight files themselves.

4.1 Realistic Multi-GPU Options

Community estimates put one workable configuration at eight high-capacity accelerator cards with at least 288GB of memory each — about 2.3TB of memory in total — at a hardware cost of roughly $550,000 to $650,000. This estimate accounts for weight storage, activation cache, and KV cache for a certain context length. Note that this is a third-party estimate; the developer has not published official hardware recommendations.

For most small and mid-sized teams, that hardware threshold is not low. In practice, however, not every scenario requires all weights to be fully loaded. If the business only handles short or medium-length contexts, the memory allocated to the KV cache can be reduced accordingly, lowering the total requirement. In addition, some inference frameworks support tiered scheduling of weights between GPU memory and host memory, which opens up ways to reduce deployment costs.

4.2 The API Route

For teams that do not want to build their own inference cluster, the developer also offers API access. The API model names follow a clear naming convention, with a standard tier and a high-speed tier offered in parallel — the high-speed tier is claimed to run 2 to 3 times faster than the standard one. A reasoning-effort parameter provides low, high, and maximum levels, with maximum as the default.

One detail deserves special attention during API integration: in multi-turn tool-calling scenarios, the previous turn’s reasoning content must be passed back together with the tool call information. This requirement means the API, by design, distinguishes between the “thinking process” and the “final reply.” In multi-turn Agent tasks, the state of the thinking process directly affects the model’s behavioral continuity. Developers who overlook this step may find the model losing context in subsequent turns.

5. Agent Evaluation: What Real Session Data Reveals

On the day of the weight release, the third-party Arena platform published evaluation results based on real user Agent sessions. Its methodology differs fundamentally from traditional static benchmarks: rather than relying on preset question-answer pairs, it extracts signals from real interactions between users and models.

5.1 Where the Evaluation Signals Come From

The Arena platform reads several types of signals from Agent sessions to judge model performance: whether the user explicitly confirms task success, praise or complaints about the model’s output, whether the model’s responses improve after the user corrects it, and whether shell commands execute successfully. These signals are combined into a “net uplift” metric that reflects how much positive experience the model actually delivers in real tasks.

The rationale behind this approach is sound: the core of Agent work is not “answering correctly” but “solving problems through interaction.” Users’ sentiment feedback and task-completion confirmations often reflect a model’s real usability better than a single accuracy score.

5.2 Rankings and Category Results

The model achieved +5.1% on the net uplift metric, ranking first among open-weight models and thirteenth overall. The previous generation scored +0.91% and ranked sixth among open models. The jump from +0.91% to +5.1% shows a significant improvement in real Agent tasks.

The category data is more specific. On Bash reliability, the model scored +7.9%, compared with -4.5% for the previous generation — evidence that it is more robust when executing shell commands, interpreting command output, and handling execution failures. Across the Work, Code, and Chat task categories, the model ranked first among open-weight models, demonstrating comprehensive capability across task types.

5.3 What This Means for Developers

These numbers are directly useful when choosing a model. If your business involves heavy shell command execution, long-horizon task planning, or code assistance, the model’s Agent performance deserves attention. The Arena methodology also offers a lesson worth borrowing: when building your own Agent evaluation system, do not rely solely on preset test sets — collect feedback signals from real user sessions wherever possible.

6. Practical Recommendations: Action Paths by Role

Different technical roles will find different action paths in this release.

Deployment engineers should first assess whether existing hardware meets the minimum deployment requirements. If memory falls short, consider inference frameworks that support weight tiering, or reduce concurrent requests and KV cache length to lower resource consumption. Before deployment, focus on verifying that the MXFP4 weights load correctly under the target inference framework and that inference accuracy holds up.

Fine-tuning researchers can use the architecture parameters and quantization scheme published in the model card as a baseline for parameter-efficient fine-tuning experiments. Given the model’s scale, full-parameter fine-tuning is impractical in most scenarios; LoRA or QLoRA will be the more realistic options. The quantization-aware training experience also provides a reference for preparing a model for low-precision deployment at the training stage.

Agent application developers should pay close attention to the state-passing logic in multi-turn tool calling, as well as the model’s ability to recover after shell execution failures. It is advisable to manage the model’s reasoning process explicitly at the application layer, rather than treating the model as a stateless Q&A interface.

Model auditing and safety teams can use the full weights for interpretability analysis and safety evaluation — something that was impossible when only API access was available. In-depth analysis of the routing mechanisms, attention patterns, and expert activation distributions helps clarify the model’s behavioral boundaries under specific inputs.

7. Conclusion

The open release of this 2.8T-parameter sparse model provides a complete reference case for large-scale MoE engineering practice. From model structure to quantization, from deployment thresholds to Agent evaluation, every step shows a clear engineering orientation. For developers and enterprises focused on putting ultra-large models into production, the point is not “yet another open-source model.” It is what this case demonstrates: how to make systematic trade-offs among storage efficiency, inference cost, and model capability — and how to evaluate a model’s real value with real-world data rather than a single benchmark.