A 2.8T-Parameter Open-Weight MoE Model: Architecture, Quantization, and Deployment

1. Open Weights: From “Usable” to “Controllable”

Large model delivery typically falls into three tiers: online service only, fully open weights, and open weights plus training and evaluation details. For enterprise adoption, fully open weights represent a critical threshold: teams can deploy the model on their own infrastructure, fine-tune it on private data, and audit model behavior without depending on external service availability or billing policies.

Recently, a team uploaded the complete weights of a 2.8T total-parameter Mixture-of-Experts (MoE) model to an open model community, as previously committed. Any team can now download the weights for deployment, fine-tuning, or research. From an engineering perspective, this release carries three notable signals:

• Verifiable delivery schedule: the promise to release weights “within a few days” was met on time, reflecting maturity in R&D and engineering management.
• Low-bit storage: the weights are published in MXFP4 format, bringing the download and deployment cost of a 2.8T-class model down to an acceptable range.
• Pre-adapted ecosystem: several mainstream open-source inference frameworks published reference deployment configurations at release time, indicating that compatibility was validated before launch.

2. Architecture Breakdown: Sparse Attention plus Mixture-of-Experts

According to the public model card, the model has 93 layers in total. Among them, 69 layers use gated linear attention, while 24 layers use gated sparse attention. The expert layer contains 896 experts, with 16 routed experts activated per token, plus 2 shared experts. The vision side includes a lightweight vision encoder with roughly 400 million parameters, supporting image-text inputs. Total parameters amount to approximately 2.78 trillion, while each token activates roughly 104 billion parameters — an activation ratio of only about 3.7%.

1. Hybrid attention design.
Linear attention uses incremental state updates to compress historical information into a fixed-size state, making long-sequence computation and memory cost roughly linear. However, pure linear attention has inherent weaknesses in tasks requiring precise lookback, such as long-document retrieval or cross-segment code navigation. The model therefore keeps gated sparse attention in about one quarter of its layers, restoring precise retrieval capability at a relatively small cost. This is key to making million-token contexts manageable in terms of memory and latency. A common KV cache estimate is:

KV memory ≈ 2 × attention layers × KV heads × head dim × sequence length × bytes per element

Fewer layers and more compressible states mean lower KV overhead for long contexts.

2. Fine-grained experts plus shared experts.
Many small experts expand the routing combination space exponentially, providing more expressive flexibility at the same activation budget. Shared experts process every token regardless of routing, carrying cross-task common knowledge and reducing redundancy among routed experts. A simplified routing implementation looks like this:

def moe_forward(x, router, experts, shared, top_k=16):
    scores = softmax(router(x))                 # score all 896 experts
    idx, w = topk(scores, k=top_k)              # activate 16 routed experts
    y = sum(w[i] * experts[idx[i]](x) for i in range(top_k))
    return y + shared(x)                        # shared experts are not routed

3. Low-Bit Quantization: MXFP4/MXFP8 and Quantization-Aware Training

This is the most engineering-significant part of the release. Weights are stored in MXFP4 and activations use MXFP8, and the official description confirms the use of quantization-aware training (QAT). A quick storage calculation:

Precision	Bytes per parameter	Theoretical size for 2.78T params
BF16	2B	≈ 5.56TB
MXFP8	1B	≈ 2.78TB
MXFP4	0.5B	≈ 1.39TB

The repository actually occupies about 1.56TB, close to the MXFP4 theoretical figure. The difference comes from sharding metadata and a small number of parameters — such as embeddings and routers — that are usually kept at higher precision.

How MXFP works: microscaling floating-point formats divide consecutive elements into blocks. Each block shares a scale factor, while individual elements are represented with 4-bit or 8-bit floating-point values. Compared with pure integer quantization, this preserves floating-point dynamic range, is more robust to outliers, and maps to the native low-precision compute units of newer accelerator cards.

The core difference between QAT and PTQ: post-training quantization (PTQ) compresses already-trained high-precision weights after the fact, and quality loss becomes significant at extremely low bit widths. Quantization-aware training, by contrast, simulates quantization noise during training so that the model learns to maintain performance in low-precision representation. For a 2.8T-class model, QAT means the released low-bit weights are the intended trained state, not a lossy-compression byproduct. This is the premise that makes publishing a trillion-parameter model directly in MXFP4 viable.

Bandwidth is throughput: decoding is memory-bandwidth-bound. Every generated token requires reading the activated parameters from memory. Halving the weight size roughly halves the bandwidth pressure and improves decoding throughput accordingly.

4. Local Deployment: Memory Estimation, Hardware Threshold, and Engineering Checklist

Memory estimation: weights take about 1.39TB in MXFP4. Add KV cache (depending on concurrency and context length), activations, and communication buffers, and reserve a 15%–25% headroom. A third-party community estimate suggests one feasible configuration: 8 flagship data-center accelerator cards at roughly 288GB of memory per card, for about 2.3TB total node memory, with an estimated system cost of 550,000–650,000 USD. This is a community estimate; the model team has not published official hardware recommendations, and actual requirements depend on concurrency targets and context length.

Deployment checklist:

• Confirm the inference framework version supports MXFP4 dequantization or native low-precision compute paths.
• Use a combination of tensor parallelism (TP) and expert parallelism (EP), and monitor expert load balance.
• For long-context workloads, enable paged KV management and prefix caching.
• For a 1.5TB-class weight download, use a resumable transfer channel; object storage can be used for staging and distribution.
• Verify shard hashes to ensure weight integrity and meet audit requirements.

5. Serving and Invocation: Reasoning Effort, Chain-of-Thought Return, and High-Speed Tier

If self-hosting is not an option, the model can be called through an online service. Such services generally follow common chat-completion API conventions and offer standard and high-speed tiers. The provider states that the high-speed tier is 2–3 times faster than the standard tier, which usually corresponds to more aggressive server-side quantization and scheduling.

Two practical parameters are worth noting:

• reasoning_effort: accepts low, high, or max, with max as the default. Lower values trade quality for latency, while higher values do the reverse. Agent-style tasks should generally use high or above.
• Context return for multi-turn tool calls: when continuing a tool-use conversation, the previous assistant message’s reasoning_content and tool_calls must be passed back together. Otherwise the model may exhibit “amnesia-like” repeated reasoning or tool argument drift. An example:

{
  "model": "<model service identifier>",
  "messages": [
    {"role": "assistant", "content": null,
     "reasoning_content": "<previous chain-of-thought>",
     "tool_calls": [{"id": "call_1", "type": "function",
       "function": {"name": "run_shell", "arguments": "{...}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "<command output>"}
  ],
  "reasoning_effort": "high"
}

6. Agent Capability Evaluation: Measuring “Net Gain” with Real Sessions

According to data published by the releasing and evaluating parties, in a public evaluation based on more than 17,500 real user Agent long-horizon sessions, this model achieved a net gain of +5.1% at maximum reasoning effort, ranking first among open-weight models and 13th overall. It also ranked first among open models in the Work, Code, and Chat subcategories. The previous-generation model scored +0.91%, ranking sixth among open models. Bash reliability improved from −4.5% to +7.9%. Evaluation signals include explicit user confirmation of task success, positive and negative feedback, response quality after correction, and actual shell command success rate. “Net gain” is the weighted difference between success and failure signals.

For enterprises, the practical takeaway is that leaderboards are only a reference. The jump from +0.91% to +5.1% between generations reflects the multiplicative effect of architecture iteration and post-training reinforcement, but teams should re-evaluate on their own task sets. A recommended self-built evaluation covers three parts: sandbox execution of tool calls, per-dimension metrics such as task completion rate, correction convergence speed, and command success rate, plus manual spot-checking.

7. Deployment Paths: A Decision Framework for Three Routes

Path	Suitable for	Key constraints
Direct API calls	Fast validation, volatile traffic	Data compliance, long-term cost
Cloud-hosted or dedicated instances	Balancing compliance and elasticity	Dependence on cloud platform operations
Fully private deployment	Strong compliance, deep fine-tuning	Hardware investment in the millions of USD, operational complexity

Cloud-based model service platforms generally provide managed model inference, dedicated compute, and supporting capabilities such as logging, monitoring, and alerting. They can serve as an intermediate path between direct API access and fully private deployment. Weight files and training data can be distributed and orchestrated through object storage and container platforms.

Conclusion

The combination of low-bit quantization-aware training, sparse attention, and fine-grained MoE is turning “open-source trillion-parameter models” from a concept into an operable reality. For most teams, the practical path is to validate business value through an online service first, then evaluate the ROI of private deployment. For teams that genuinely need self-hosting, three details will directly determine success or failure: memory estimation, operator compatibility, and chain-of-thought context return.