2.8T-Parameter MoE LLM Open-Sourced: Architecture, Deployment, and Evaluation Explained

1. Background: A Trillion-Parameter MoE Model Goes Open

In early November 2024, a leading AI research team uploaded the full weights of its flagship large language model to a mainstream model hosting platform, delivering on an earlier commitment to open the weights within a set timeframe. The model is a Mixture of Experts (MoE) architecture with roughly 2.8 trillion total parameters, of which only about 104 billion are activated per inference. It supports a million-token context window and multimodal image-and-text input. Released as a package of approximately 1.56TB, the weights attracted broad attention across the developer community.

Open weights mean that research institutions and enterprises can download the model directly, deploy it locally, fine-tune it, or run safety audits — a milestone for building private AI infrastructure. On the same day, a newly published Agent benchmark showed the model ranked first among open-weight models, and thirteenth overall, in Agent capability rankings based on real user sessions.

2. Model Architecture: A Deep Dive

2.1 Overall Topology: Combining MoE and Attention

According to the official technical documentation, the model uses a 93-layer Transformer structure:

• 69 layers use a sparse attention mechanism (a DeepSeek Sparse Attention variant), replacing the full-attention gated MLA layers of the previous generation;
• 24 layers use Gated Deep Selective Attention (Gated DSA);
• The expert module contains 896 expert networks in total, with each token routed to 16 experts for computation, plus 2 shared experts;
• The vision encoder is a multimodal ViT variant with roughly 401 million parameters.

The combined “sparse attention + MoE” design aims to sharply reduce compute costs while preserving long-context modeling capability. Sparse attention shrinks the complexity of the attention matrix by attending to only a subset of token pairs, while MoE lowers per-inference FLOPs by activating experts on demand. Together, they keep inference costs acceptable at trillion-parameter scale.

2.2 Why Sparse Attention Matters

Standard full attention has O(n²) time and space complexity, which is nearly infeasible for a million-token context. Sparse attention uses specific attention patterns (such as sliding windows, global tokens, and random sampling) to bring complexity down to O(n·log n) or even O(n), making ultra-long contexts practical.

The gated sparse attention variant adopted by this model introduces learnable gating signals that dynamically decide which token pairs require attention computation. This is more flexible than fixed-pattern sparsity and can adapt context usage to the demands of different tasks.

2.3 Expert Routing Strategy

Routing each token to 16 of 896 experts means only about 1.8% of experts are activated per inference. This “Top-K routing” strategy is the mainstream approach in current MoE architectures. The shared experts ensure that general-purpose knowledge (such as grammar and basic reasoning) is processed for every token, preventing pure-routing schemes from losing foundational capabilities.

3. Quantized Storage and Quantization-Aware Training

3.1 The MXFP4/MXFP8 Hybrid Precision Scheme

The weights are stored in MXFP4 (4-bit floating point) format, while activations are computed in MXFP8 (8-bit floating point) at inference time. This is an elegantly engineered design:

• 4-bit storage: compresses the 2.8T-parameter weight files from roughly 5.6TB in FP16 to about 1.56TB, cutting download and storage costs;
• 8-bit computation: prevents inference accuracy from degrading severely due to over-quantization; 8-bit floating point already performs close to FP16 on most tasks.

3.2 Quantization-Aware Training (QAT)

The team has confirmed that quantization-aware training was used. The core idea of QAT is to simulate quantization during training, so that model parameters are updated while “aware” that they will be quantized, minimizing the accuracy loss of the final quantized model.

Compared with post-training quantization (PTQ), QAT typically delivers a 1–3 percentage point accuracy improvement, which is especially critical for numerically sensitive architectures such as MoE. A typical QAT pipeline:

1. Pre-train the FP16/BF16 model
2. Insert fake quantization nodes
3. Compute the loss in simulated-quantization forward passes
4. Use the Straight-Through Estimator (STE) to approximate gradients during backpropagation
5. Export the quantized weights after training converges

3.3 How Quantization Affects Deployment

4-bit weight storage greatly reduces memory footprint and model load times, but it places requirements on the inference framework: support for dynamic or on-the-fly dequantization. Mainstream inference engines have already adapted to this low-precision format, and developers can refer to the official deployment documentation for configuration.

4. Local Deployment: Hardware Requirements and Options

4.1 Estimating Memory Requirements

Based on community estimates, one feasible deployment configuration is:

Item	Specification
GPU model	8 high-end datacenter GPUs (~2.3TB memory per card)
Total memory	~18.4TB
Hardware cost	~$550,000–$650,000 (estimate only)

It should be emphasized that these figures are community estimates, not an official recommended configuration. Real-world deployment must also account for:

• KV cache: a million-token context means enormous KV cache usage, requiring techniques such as paged attention or speculative decoding;
• Network bandwidth: MoE experts are distributed across GPUs, and All-to-All communication requires NVLink or high-speed interconnects;
• Quantized inference engine support: an inference framework capable of loading 4-bit weights.

4.2 Inference Framework Support

Current mainstream LLM inference engines have published deployment guides for this model, including:

• vLLM: supports tensor parallelism and pipeline parallelism, well suited for high-throughput serving;
• SGLang: optimized for structured generation and batched workloads;
• TokenSpeed: an inference framework optimized at the token level.

When deploying, pay attention to expert parallelism configuration for MoE models, maximum sequence length settings, and specifying the correct quantization format.

4.3 Cloud Deployment as an Alternative

Teams without a local multi-GPU cluster can deploy elastically on GPU instances from cloud providers. Cloud platforms typically offer on-demand high-end GPU instances, which — combined with object storage and container orchestration services — enable fast rollout. Cloud vendors such as Baidu AI Cloud also provide GPU compute and model deployment platforms that support one-click deployment and elastic scaling for large models.

5. API Usage and Multi-Turn Agent Capabilities

5.1 Interface Compatibility

The model offers a standard RESTful API compatible with two widely used conversational API protocol specifications. Developers can switch model identifiers to call different versions:

• Standard: balances quality and speed;
• High-speed: officially stated to run 2–3× faster than the standard version.

5.2 Reasoning Depth Control

The API supports adjusting the reasoning depth (reasoning_effort) across low, high, and max levels, with max as the default. This design lets developers match effort to task complexity: use the low level for simple Q&A to reduce latency, and the high level for complex reasoning to improve quality.

5.3 Considerations for Multi-Turn Tool Calling

In agent scenarios involving multi-turn tool calls, the caller must pass back the previous round’s reasoning content and tool call results to keep the context coherent. This is the common pattern in current agent frameworks, and developers need to implement state management and history assembly at the orchestration layer.

6. Agent Benchmark: Measuring Capability with Real Sessions

6.1 Methodology

Unlike traditional static benchmarks, the Agent Arena ranking is based on real user agent session data. Evaluation signals include:

• whether the user explicitly confirms the task is complete;
• positive and negative user feedback;
• the model’s response quality after user corrections;
• execution success rates of tool calls such as shell commands.

From these signals, the system computes a “net uplift” metric that comprehensively reflects model performance in real agent scenarios.

6.2 Interpreting the Results

Based on more than 17,500 real long-horizon sessions:

• The model achieved a net uplift of +5.1% — first among open-weight models, thirteenth overall;
• It ranked first among open models in all three subcategories: work, coding, and conversation;
• Its predecessor scored a net uplift of only +0.91%, ranked sixth among open models.

By individual capability, shell command reliability improved most notably among open models (+7.9%, ranked first), whereas the predecessor scored −4.5% (ranked thirteenth) on that metric — clear evidence of the architecture upgrade’s impact on agent execution capability.

6.3 Takeaways from the Evaluation System

Real-session evaluation reflects production performance better than static test questions. When selecting a large model, enterprises should pay attention to such “real-world” assessment results rather than relying solely on academic benchmark scores. In addition, agent capability assessment needs to cover multiple dimensions — code execution, file operations, web queries, and more — since a single metric is insufficient for a comprehensive judgment.

7. Summary and Outlook

The release of this 2.8T-parameter MoE model’s weights provides the developer community with a foundation model of significant research and application value. Its technical highlights include the deep combination of sparse attention and MoE, the MXFP4/MXFP8 hybrid quantization scheme, quantization-aware training in engineering practice, and leading performance in real agent scenarios.

Teams intending to build their own LLM services should carefully weigh hardware investment, inference engine adaptation, and operational complexity; teams pursuing fast time-to-market will find elastic API access via cloud platforms a more pragmatic choice. As quantization technology and sparse computing continue to evolve, the deployment barrier for trillion-parameter models is expected to keep falling, pushing large models from “cloud-only” toward a new stage of “edge-cloud collaboration.”