Open Weights for a 2.8T-Parameter MoE Model: Deployment and Quantization in Practice

A Mixture of Experts (MoE) model with approximately 2.8 trillion total parameters has released its full weights to the public. According to available information, the model activates roughly 104B parameters per token and supports a 1-million-token context window along with image-text input. Evaluation results published at the same time show the model ranked first among open-weight models and thirteenth overall on an agent arena leaderboard. For teams that have long tracked large-model engineering, the core value of this release is not simply “another downloadable model” — it offers a complete sample of how an ultra-large sparse model is designed, quantized, deployed, and invoked.

This article provides a technical breakdown of the release across five dimensions: model architecture, quantization strategy, deployment barriers, inference framework adaptation, and API usage. It does not recommend any specific vendor and analyzes the implementation considerations for similar ultra-large MoE models from a general engineering perspective.

1. Understanding Model Scale: The Key Difference Between Total and Activated Parameters

When discussing any MoE model, two concepts must be distinguished first: total parameters and activated parameters per token. In this case, the model has roughly 2.78 trillion total parameters, but only about 104B parameters are actually activated per forward token. The core motivation behind this design is to use large parameter capacity for knowledge storage and expressive capability while controlling inference-time compute overhead through sparse activation.

From an engineering standpoint, total parameter count determines the disk and memory footprint of weight files, while activated parameter count directly affects per-inference compute consumption and latency. Therefore, even if a model has 2.8T total parameters, its inference cost is not simply equivalent to “loading 2.8T parameters for computation every time” — this is precisely the key value that MoE architecture offers over traditional dense models.

The released weights are stored in safetensors format, with the repository occupying approximately 1.56TB of disk space. For teams that need self-hosting or offline auditing, this means sufficient storage, bandwidth, and verification resources must be prepared in advance. The 1.56TB repository size also demands resumable downloads and integrity verification mechanisms during transfer, otherwise costs can easily spiral out of control at the download stage.

2. Architecture Analysis: Engineering Intent Behind Sparse Attention and Expert Layer Design

According to publicly available architecture information, the model has 93 layers, of which 69 use an attention structure called KDA and 24 use a gated sparse attention structure. Compared with the previous generation, this design replaces full-attention gated multi-latent attention layers with a more compute-efficient approach to long-context modeling. This change directly addresses the 1-million-token context window requirement: in ultra-long sequence scenarios, inference costs become unacceptable if attention complexity is not structurally sparsified.

On the expert layer side, the model has 896 experts in total, with 16 experts selected per token and 2 additional shared experts always activated. One engineering detail worth noting: the expert count of 896 is far larger than typical small-scale MoE models, but only 16 experts are selected per token, indicating that the routing mechanism maintains a high degree of sparsity. The introduction of shared experts ensures that all tokens can obtain a portion of stable, general-purpose feature transformation capability, preventing representation degradation for certain inputs under sparse routing.

The vision component is handled by a visual encoder with approximately 401M parameters, enabling the model to accept mixed image-text input. For teams that need to process documents, screenshots, charts, and similar tasks, this capability means multimodal input can be modeled uniformly within the same inference pipeline without additionally stitching in external vision modules.

3. Quantization Strategy: Co-Design of MXFP4 Weights and MXFP8 Activations

Whether an ultra-large model can actually run on real hardware depends heavily on the quantization scheme. In this case, weights are stored in MXFP4 format and activations use MXFP8 format, with the official release stating that Quantization-Aware Training (QAT) was employed. This combination deserves closer discussion.

MXFP4 weight storage means each weight value occupies only about 4 bits of storage space, significantly reducing memory footprint compared to FP16 or BF16. For a 2.78T-parameter model, storing weights in BF16 alone would require approximately 5.56TB of memory; with MXFP4, weight storage drops to roughly 1.4TB, already close to the repository’s 1.56TB total size. What makes the MX format special is that it retains shared exponent bits, maintaining a larger dynamic range at low bit widths and performing more stably than standard INT4 in scenarios where activation distributions are wide.

MXFP8 activations quantize activation values to 8-bit format during forward computation. Activation values are typically more sensitive to quantization error than weights, so using twice the bit width of weights achieves a trade-off between computational precision and throughput. The combination of 4-bit weight quantization and 8-bit activation quantization has become a common approach for compressing models at the hundred-billion-parameter scale and above.

The significance of Quantization-Aware Training is that the model simulates quantization error during the training phase rather than applying post-training quantization after training is complete. This can significantly reduce the accuracy loss introduced by low-bit representation, especially for extremely low-bit weight formats like MXFP4. For teams fine-tuning this model themselves, this information also implies that: if continued training or domain adaptation is needed, simply following the route of FP16 fine-tuning followed by quantization may not fully align with the officially released performance level. A more reasonable approach is to retain a similar quantization simulation mechanism within the training pipeline.

4. Deployment Barriers: Hardware Costs and Inference Framework Adaptation

Based on public discussion, one feasible local deployment configuration uses 8 high-memory compute cards with approximately 2.3TB total memory, with hardware costs estimated in the $550,000 to $650,000 range. It should be clearly noted that this is a third-party estimate rather than an official recommendation, but its order of magnitude is informative. Even after MXFP4 compression, a 2.8T-parameter model still occupies approximately 1.4TB of memory for weights alone. Adding KV cache, activation temporary space, and inference framework overhead, a single machine with less than 2TB of memory essentially cannot fully load the model.

Multi-card parallelism is not optional in ultra-large MoE deployment — it is a prerequisite. When experts are distributed across multiple cards, each token’s 16-expert routing introduces cross-card communication overhead. Therefore, deployment plans must consider inter-card interconnect bandwidth and topology. For models of this scale, common approaches include combinations of tensor parallelism, expert parallelism, and pipeline parallelism. On the inference framework side, several mainstream frameworks already provide deployment scripts or configuration templates. These scripts typically preset parallelism strategies, quantization-dequantization paths, and memory management parameters, reducing the trial-and-error cost for teams configuring from scratch.

For the vast majority of small and mid-sized teams, directly purchasing hundreds of thousands of dollars in hardware is not realistic. A more practical choice is to first validate the model’s effectiveness in their own business scenarios through API access, then decide whether local deployment is worth the investment. The model naming and parameter design under the API model also reveal some engineering information — for example, standard and high-speed versions coexist, with the high-speed version claiming 2 to 3 times the speed of the standard version. This design typically corresponds to different batching strategies and hardware resource pool allocations, allowing users to choose based on latency budgets and cost constraints.

5. Multi-Turn Tool Calling Details in API Usage

For developers integrating via API, attention must still be paid to the model’s reasoning effort settings. The interface supports three reasoning effort levels: low, high, and max, with max as the default. Reasoning effort here can be understood as the amount of thinking compute the model spends before generating. Higher effort levels typically improve performance on complex tasks but significantly increase time-to-first-token and total latency. For simple Q&A or high-throughput scenarios, choosing low or high often achieves a better balance between quality and speed.

During multi-turn tool calling, the previous round’s reasoning content and tool call results must be passed back together. The engineering implication is that this model’s tool calling is not stateless — it relies on prior reasoning processes to maintain decision consistency. When implementing a tool loop, developers must properly save and pass the complete context fields; otherwise, the model may fail to correctly understand the current execution state in subsequent rounds, leading to repeated calls, missing parameters, or task interruption.

A typical tool calling loop can be summarized as follows:

messages = user question
while tool call needed:
    response = chat(messages, tools, reasoning_effort="high")
    reasoning = response.reasoning_content
    tool_calls = response.tool_calls
    results = execute_tools(tool_calls)
    messages.append({
        "reasoning_content": reasoning,
        "tool_calls": tool_calls,
        "tool_results": results
    })
final_answer = response.content

The key point is treating both the reasoning process and tool call results as part of the message sequence, rather than retaining only the final text. For agent applications that require multi-step calls to external systems, database queries, or document library operations, this design directly affects task success rates.

6. Implications for Developers: How to Evaluate Models of Similar Scale

Taken together, this weight release provides an important reference for the technical community: a comprehensive engineering practice of ultra-large MoE models spanning sparse attention, expert routing, low-bit quantization, and multi-machine deployment. For teams that do not directly use this model, the following conclusions still have general value:

First, total parameter count is no longer the sole performance metric. Activated parameters, quantization bit width, context length, and tool calling capability together determine a model’s usable boundaries. When evaluating whether a model suits a particular scenario, practical performance on target tasks and deployment cost should take priority over chasing parameter scale.

Second, low-bit quantization is not simply about “saving memory.” The co-design of quantization strategy and training method is what enables low-bit models to maintain performance. Simply applying post-training quantization to an already-trained dense model often encounters greater accuracy loss at extremely low bit settings; quantization-aware training can fundamentally improve this.

Third, deployment plans require full-pipeline planning. From weight download, storage verification, parallel partitioning, and inference framework selection to service encapsulation, every step can become a bottleneck. The relationships among total memory, inter-card bandwidth, and inference latency need clear estimates at the design stage, rather than reactive optimization after deployment is complete.

Fourth, API integration cannot ignore state management. Context passing in multi-turn tool calling, reasoning effort selection, and error retry mechanisms directly determine the stability of agent applications. Developers should treat the tool calling loop as a complete state machine, not a simple request-response sequence.

As larger models gradually open their weights, teams that thoroughly understand sparse architectures, low-bit quantization, and distributed deployment logic will gain clear advantages in subsequent technology selection and business implementation. For enterprise users, whether to deploy locally, whether to fine-tune, and whether to use API-only access should depend on real business needs and resource constraints — not on chasing model parameter scale or leaderboard rankings. Rational evaluation and incremental validation are the reliable path for large-model engineering.