A 2.8T-Parameter Open-Weight MoE Model: Architecture, Deployment, and Engineering Practice

1. What Open Weights Mean Here

The large-model landscape is shifting. Frontier-level capabilities are no longer delivered exclusively through closed APIs; full model weights are gradually being released to the community. A team recently published the complete weights of its flagship model on a major model hosting platform, following through on an earlier commitment to open the weights within two weeks of launch. For teams that need self-hosting, fine-tuning, or security auditing, this changes the situation from “API-only access” to “full control.”

Based on public information, the model is an extremely large sparse Mixture-of-Experts (MoE) architecture with roughly 2.8 trillion total parameters and about 104 billion activated parameters per token. It supports million-token context lengths and multimodal image-text inputs. On its release day, a well-known model arena ranked it first among open-weight models and thirteenth overall. Two takeaways: open-weight models are now close to the top closed-source tier on agent-style tasks, and the deployment cost of such a large model remains prohibitive for most teams.

This article covers five dimensions: model structure, numerical formats, deployment paths, hardware estimates, and API usage. The goal is to help developers decide whether self-hosting is justified and how to use the model correctly.

2. Model Architecture: Sparse Attention and Expert Routing

2.1 Layer Composition and Attention Replacement

The model has 93 layers. Of these, 69 use a KDA structure and 24 use Gated Sparse Attention. Compared with the previous generation, the key architectural change is replacing the original full-attention gated MLA layers with gated sparse attention layers. The motivation is straightforward: controlling the cost of long-context modeling.

With full attention, compute and memory overhead grow quadratically with sequence length. Sparse attention limits the key range visible to each query, reducing effective computation to near-linear. For million-token contexts, this is not the difference between “slower” and “faster” but between “runs” and “does not run.” The gating mechanism performs dynamic weighting on top of sparse selection, preventing the information loss from sparsification from becoming too rigid.

2.2 Expert Configuration

The expert layers contain 896 routed experts. Each token activates 16 of them, with 2 additional shared experts always active. The sparsity profile is notable:

• Selecting 16 out of 896 means a single token activates only about 1.8% of routed experts. The 2 shared experts handle general capabilities.
• Shared experts reduce output instability caused by routing fluctuations, which is especially important for multi-turn dialogue and tool-calling scenarios.
• A high total expert count combined with a low activation ratio is the mainstream design direction for very large MoE models: use parameter capacity for knowledge density and sparse activation for lower inference cost.

2.3 Vision Encoder

The multimodal component is handled by a vision encoder with approximately 401M parameters. Relative to the 2.8T total, the vision side is tiny. This suggests image understanding relies mainly on the language backbone jointly modeling visual tokens, while the vision encoder only performs feature extraction. For applications with mixed image-text inputs, this means the additional overhead of visual input remains manageable.

3. Numerical Formats: MXFP4 Storage and MXFP8 Activation

Weights are stored in MXFP4 format, while activations use MXFP8. The official documentation states that quantization-aware training (QAT) was used during training. These choices need to be understood separately.

Storage side (MXFP4): A 4-bit microscaling floating-point format that uses block-level shared exponents to reduce storage overhead. The model hosting platform shows the repository occupies about 1.56 TB, while the reported parameter count is about 2.78 trillion. That works out to roughly 0.56 bytes per parameter, consistent with 4-bit storage plus a small amount of scaling factor data. Storing the same model in BF16 would require over 5.5 TB, multiplying download and loading costs.

Activation side (MXFP8): Activations during inference stay at 8-bit precision, balancing matrix multiplication throughput against numerical stability. Keeping activation precision higher than weight precision is common practice because activation dynamic range is more sensitive.

Why QAT matters: Simulating quantization error during training lets the model parameters adapt to low-precision representations. This is the precondition for 4-bit weights remaining usable. With post-training quantization (PTQ) alone, a 2.8T model at 4 bits typically shows clear quality degradation. QAT does not change the numerical format used at inference time; it only changes how robust the weight distribution is to quantization.

For self-hosting teams, the direct implication is that memory and bandwidth pressure are significantly lower than the nominal parameter count suggests. However, low-precision weights require proper operator support in the inference framework. If the framework has not adapted to the corresponding format, loading may fall back to slower paths or fail entirely.

4. Deployment Paths and Hardware Requirements

4.1 Inference Framework Support

Public information indicates that mainstream inference frameworks already provide deployment solutions, covering high-throughput serving, low-latency inference, and specialized optimization paths. Engineering recommendations:

1. Confirm numerical format support first: Loading MXFP4 weights requires corresponding dequantization operators and fused kernels. Without them, the framework may fall back to layer-by-layer dequantization, sharply reducing throughput.
2. Confirm parallelism strategy: Even in 4-bit form, a 2.8T model exceeds single-node capacity. Tensor parallelism plus pipeline parallelism is required, and the framework must support expert parallelism (EP) to distribute the 896 experts across multiple devices.
3. Validate context length: KV cache overhead for million-token contexts is not negligible. Sparse attention reduces compute, but KV storage still needs to be planned according to the actual sequence length.

A conceptual startup configuration, shown only to illustrate the parameter structure:

engine_args = {
    "model": "<local_weight_path>",
    "quantization": "mxfp4",          # Weight format; requires native framework support
    "kv_cache_dtype": "mxfp8",        # Activation/KV precision
    "tensor_parallel_size": 8,         # Single-node tensor parallelism
    "pipeline_parallel_size": 2,       # Cross-node pipeline parallelism
    "expert_parallel_size": 16,        # Expert distribution
    "max_model_len": 1_000_000,        # Context length limit
    "enable_chunked_prefill": True,    # Chunked prefill for long inputs
}

4.2 Hardware Cost Estimates

A third-party estimate suggests one viable configuration: a server with 8 high-end inference accelerators and about 2.3 TB of total memory, with hardware costs in the range of $550,000 to $650,000. Several clarifications:

• This is a third-party estimate, not an official recommendation. The actual configuration depends on the parallelism strategy, context length, and throughput targets.
• 2.3 TB of memory corresponds to “fits” rather than “runs comfortably.” Long-context scenarios require reserving KV cache space.
• For most teams, self-hosting only makes economic sense in two cases: high-frequency large-scale calls that amortize fixed costs, or data compliance requirements that mandate private deployment.

If the goal is capability validation or small-scale experiments, the API route has a far more reasonable cost structure.

5. API Usage: Reasoning Effort and Multi-Turn Tool Calls

5.1 Model Identifiers and Interface

The API side provides two model identifiers: a standard version and a high-speed version. The official statement says the high-speed version is 2 to 3 times faster than the standard one. The interface is compatible with two mainstream protocol formats, keeping migration costs low.

5.2 The reasoning_effort Parameter

Reasoning effort can be set to low, high, or max, with max as the default. This parameter directly affects latency and token consumption:

• low: Suitable for deterministic tasks such as classification, extraction, and format conversion.
• high: Suitable for multi-step reasoning that does not require exhaustive search, such as routine Q&A.
• max: Suitable for complex planning, code generation, and agent decision chains. Latency is highest.

Engineering advice: do not use the default value globally. Routing batch tasks by type can significantly reduce average cost.

5.3 State Passing in Multi-Turn Tool Calls

This is the most error-prone area. In multi-turn tool calling, both reasoning_content and tool_calls from the previous assistant turn must be passed back. The reason is that the chain of thought of a reasoning model is part of the context for subsequent decisions. Dropping reasoning_content causes the model to “forget” its earlier reasoning basis, leading to repeated calls or broken logic.

Example:

messages.append({
    "role": "assistant",
    "content": resp.content,
    "reasoning_content": resp.reasoning_content,   # Must be passed back
    "tool_calls": resp.tool_calls,                  # Must be passed back
})
messages.append({
    "role": "tool",
    "tool_call_id": resp.tool_calls[0].id,
    "content": tool_result,
})

Also note that tool results should be trimmed. In long contexts, tool output is one of the main sources of token consumption.

6. Decision Guidance

Overall, the open weights of this model are better suited to teams with existing multi-node multi-GPU inference infrastructure, clear private-deployment compliance requirements, or a need for domain fine-tuning and security auditing. For application-focused teams, the API route combined with reasoning-effort tiering is the more pragmatic choice.

Whichever path you take, three technical points are worth remembering: sparse attention is the precondition for long-context feasibility; QAT is the precondition for usable 4-bit weights; and passing back reasoning_content is the precondition for correct multi-turn agent calls. Ignoring any one of them will cost you during actual implementation.