A Trillion-Parameter Open-Weight MoE Model: Quantization, Deployment, and Evaluation

1. What Happened: A Prompt Open-Weight Release

On November 4, a leading model team uploaded the full weights of a Mixture-of-Experts (MoE) large language model to an open model hosting community, following through on a commitment made during its late-October launch to release the weights “in the near term.” According to the model card, this is an MoE model with roughly 2.8 trillion total parameters and about 104B parameters activated per token. It supports a 1-million-token context window and mixed text-image input. The weight repository takes up about 1.56TB in total, corresponding to approximately 2.78 trillion parameters based on the weight files. On the same day, a third-party Agent benchmark built on real user sessions updated its results: the model ranked first among open-weight models and 13th overall.

For engineering teams, downloadable weights enable three things that were previously impossible: deploying the model in their own data center or private cloud, fine-tuning it on proprietary data, and independently auditing model weights and behavior for security and compliance. This is the core value of the open-weight ecosystem: model capability is no longer something you can only rent through an API. It becomes a technology asset that enterprises can control directly.

Below, we unpack the key technical points from four angles: architecture, quantization, deployment, and evaluation.

2. Architecture: Hybrid Attention and Massive Expert Layers

2.1 Attention: A Mix of Linear and Sparse Designs

The model has 93 layers in total. Of these, 69 use a linear attention variant and 24 use gated sparse attention. This combination points to a clear engineering goal: reducing the cost of long-context inference.

• The problem with full attention: Standard self-attention computation and KV cache overhead grow sharply with sequence length. At the million-token scale, the KV cache alone can consume hundreds of gigabytes, making inference costs hard to justify.
• Linear attention: Historical information is compressed into a fixed-size recurrent state, making per-token cost roughly constant. It is well suited for modeling tasks that involve reading through an entire document. The trade-off is weaker precise retrieval capability.
• Gated sparse attention: This keeps the precise retrieval ability of attention, but each query only reads a small number of the most relevant KV entries, trimming away the most expensive part of long-context processing.

The 69:24 mix of these two layer types balances cheap long-document reading with accurate detail lookup. For developers, this explains why such models can push context length to the million-token level: sparsity dramatically shrinks the KV cache, turning long-context inference from a memory-heavy burden into something that can be planned around.

2.2 MoE: 16 of 896 Experts Activated, Plus 2 Shared Experts

The expert layers contain 896 routed experts, with 16 activated per token. There are also 2 shared experts that are always on. Two points are worth highlighting:

• The role of shared experts: They absorb common knowledge used by all tokens, preventing every routed expert from redundantly learning universal patterns. This improves parameter efficiency and stabilizes routing behavior.
• Activation ratio: About 104B parameters are activated per token, roughly 3.7% of the total. At its core, MoE decouples model capacity from per-example compute. Capacity determines the upper bound of knowledge, while activated parameters determine inference cost. The two are no longer chained together.

The vision side uses a vision encoder with roughly 400 million parameters. Images are patch-encoded and mapped into the same token space as text, so mixed image-text input does not require retraining the entire language backbone.

3. Quantization and Storage: MXFP4 Weights with Quantization-Aware Training

The weights are stored in MXFP4 format, activations use MXFP8, and the official documentation states that quantization-aware training (QAT) was used. There are three layers of information here.

1. What the MX format is: Microscaling is an open industry low-bit floating-point standard. Its core idea is block-level shared scaling factors: every 32 values share a high-precision scale, while values inside the block use 4-bit floating-point representation. Compared with naive FP4, block-level scaling significantly alleviates dynamic range issues and preserves numerical accuracy.

2. The storage math:

Parameter count P ≈ 2.78 × 10^12
MXFP4 storage ≈ P × 0.5 bytes       ≈ 1.39 TB
Block scaling factor overhead (~1/32) ≈ +0.04 TB
Embedding/output layers and other higher-precision tensors → total ≈ 1.56 TB

The 1.56TB shown on the repository page is largely consistent with the theoretical value for 2.78 trillion parameters in FP4. The extra space comes from scaling factors and a small number of higher-precision tensors.

3. Why QAT matters: Low-bit quantization is simulated during training so that the weights naturally adapt to FP4 representation. Compared with post-training quantization (PTQ), QAT generally preserves more model quality under aggressive compression. The fact that the team emphasizes QAT means this is a natively FP4 model rather than an after-the-fact compressed artifact, which is a positive signal for inference stability.

4. Local Deployment: Hardware Requirements, Parallelism, and Practical Advice

4.1 Memory and Hardware Threshold

The 1.56TB weight payload must fit entirely into accelerator memory, with additional room for the KV cache, activations, and framework overhead. Real-world requirements are typically 1.2x to 1.5x the weight size. Some third-party developers estimate that one viable configuration is a server with 8 flagship AI accelerator cards, roughly 288GB of memory per card, totaling about 2.3TB of accelerator memory, at a hardware cost of around $550,000–$650,000. Note that this is a community estimate; the official release does not provide hardware recommendations.

4.2 Parallelism Strategies

• Tensor parallelism (TP): Splits the weight matrices of attention and expert layers to reduce per-card memory pressure.
• Expert parallelism (EP): Distributes the 896 experts across devices. Tokens are dispatched through all-to-all communication, which is the communication hotspot in MoE inference and depends heavily on high-bandwidth interconnect.
• Pipeline parallelism (PP): Splits the model by layer and is typically combined with the two strategies above.

Sparse attention again proves its value here: with the KV cache dramatically compressed, planning memory for a million-token context becomes realistic.

4.3 Practical Suggestions

Mainstream open-source inference frameworks have already published deployment recipes for this model, but before starting, confirm three things: whether the framework version supports MXFP4 compute kernels (if not, dequantization will be required and throughput will suffer); weight loading time (loading 1.56TB at a few GB per second of storage bandwidth can take hours, so parallel loading should be configured); and run small-scale stress tests before going to production. If the goal is simply to validate capabilities, using the API to prove out the workflow before committing to hardware is the safer path.

5. API Essentials: Reasoning Depth Control and Multi-Turn Tool Calling

If you do not self-host, you can use the same model through an API. The interface is compatible with widely used chat completion conventions, and there is a high-speed version that the provider claims is 2 to 3 times faster. Two usage details are worth noting.

1. Reasoning depth parameter: The reasoning_effort parameter accepts low, high, or max, with max as the default. It roughly corresponds to how much inference budget the model is allowed to spend. Lower settings on simple tasks can significantly reduce cost and latency, while keeping the default is safer for complex planning tasks.

2. Multi-turn tool calling requires passing back context: The previous turn’s reasoning_content and tool_calls must be sent back together with the tool results. The former holds the model’s chain-of-thought state; dropping it can cause the model to re-reason from scratch or even drift in its conclusions. The latter carries the ID alignment for tool calls; missing it can break call-chain validation. An example:

# Turn 1: the model initiates a tool call
resp = client.chat.create(
    model="<model-id>",
    messages=messages,
    reasoning_effort="high",
)
# Keep the assistant message with reasoning_content and tool_calls intact
messages.append(resp.message)
# Turn 2: append the tool execution result
messages.append({
    "role": "tool",
    "tool_call_id": resp.message.tool_calls[0].id,
    "content": tool_output,
})

This is an easily overlooked engineering detail. Many cases of unstable Agent behavior stem from multi-turn state not being passed back according to protocol.

6. Reading Agent Benchmarks: Net Uplift and Real-Session Signals

The benchmark data comes from more than 17,500 real long-running user Agent sessions. The evaluation team extracts several types of signals: whether users explicitly confirm task success, praise or complaints directed at the model, response quality after user corrections, and whether shell commands execute successfully. These are combined into a “net uplift” score.

• Results: This model achieved a net uplift of +5.1%, ranking first among open-weight models and 13th overall. Its predecessor scored +0.91% and ranked sixth among open-weight models. In specific categories, Bash reliability improved to +7.9%, first among open models, while its predecessor was at −4.5%, ranking 13th. It also ranked first among open models in the Work, Code, and Chat categories.
• Interpretation: A difference of 4 percentage points in net uplift between leading models is substantial. The shift in Bash reliability from negative to positive is especially noteworthy because it directly reflects whether commands issued by the model actually run successfully, one of the hardest indicators for real-world Agent deployment.
• Methodology: The value of real-session evaluation lies in distributional realism (how users actually ask questions and make corrections) and long-horizon behavior (multi-step tasks and mid-course corrections), which static benchmarks struggle to cover. That said, no single leaderboard should dominate decision-making. Workload composition and user demographics affect rankings. Model selection should always include small-scale validation in your own scenarios.

7. Self-Hosting or API: A Decision Checklist

Open weights give teams the choice, but “can you download it” and “should you deploy it” are different questions. Evaluate these items:

• Data sensitivity: If data cannot leave your boundary, lean toward self-hosting or private deployment.
• Call volume and peaks: Can daily token volume amortize hardware and operations costs that can reach millions of dollars?
• Customization needs: Do you need to fine-tune on proprietary data or distill smaller models?
• Latency and compliance: What are your requirements for response latency and audit logs?
• Operational capability: Do you have a team to maintain high-bandwidth interconnect clusters, quantization kernels, and inference frameworks?

For most teams, the pragmatic route is a layered mix: use an API for fast validation and to absorb long-tail traffic, then evaluate self-hosting or private deployment for core sensitive scenarios. If the priority is getting started at low cost, mature cloud-based model services—offering inference, fine-tuning, and managed hosting with pay-as-you-go pricing and elastic scaling—are a common middle ground between full self-hosting and pure API usage.

Conclusion

This weight release can be seen as a snapshot of the open-weight ecosystem: 2.8T parameters, native MXFP4 quantization, hybrid sparse attention, and real-session-driven evaluation. Each is a frontier practice in current LLM engineering. For developers, what matters is not memorizing a leaderboard position but understanding the technical trade-offs behind it: how capacity and compute are decoupled, how long-context cost is compressed, how low-bit quantization preserves quality, and how Agent capability can be measured credibly. Understanding these gives you a stronger foundation whether you choose to self-host or call an API.