2.8T-Parameter MoE Model with Open Weights: Architecture, Quantization, and Deployment Explained

As large language models push toward the trillion-parameter scale, the mixture-of-experts (MoE) architecture has become the industry’s consensus approach for balancing compute cost against model performance. Recently, a well-known model team delivered on its open-release commitment, publishing the complete weights of its latest-generation MoE model with 2.8T total parameters. The model currently tops the open-model category on a real-world agent evaluation leaderboard. Beyond giving the AI community a rare look at the architecture of a top-tier closed model, the release paves the way for teams with sufficient compute to deploy, fine-tune, and audit the model on their own. This article examines the model’s technical characteristics across four dimensions: architecture design, quantization strategy, deployment requirements, and agent capability evaluation.

Architecture: Sparse Attention Combined with Mixture-of-Experts

The model follows a typical large-scale MoE design, with roughly 2.78 trillion total parameters, a context window of up to 1 million tokens, and multimodal (image-and-text) input. Its core network consists of 93 layers, reflecting an extremely deep structure.

The attention mechanism has evolved significantly. Unlike the previous generation, which relied on full attention, the new model replaces part of the layers with gated sparse attention. Specifically, 69 of the 93 layers use KDA (gated attention), while 24 use sparse attention. The goal is to break through the memory and compute bottlenecks of long-context modeling: sparsification reduces the computational complexity of long-sequence inference, making million-token inference feasible on limited hardware.

The expert layers are sparsified to an extreme degree. They contain 896 independent experts, of which each token activates only 16, plus 2 shared experts. In other words, although the model holds 2.8T parameters in total, only about 104B parameters are actually activated per token. This high sparsity keeps inference-stage compute under control. The model also integrates a 401M-parameter vision encoder, enabling image-and-text mixed input and laying the groundwork for multimodal applications.

Quantization: Low-Bit Storage with Quantization-Aware Training

For a trillion-parameter model, storing and loading weights is a major engineering challenge. At release, the model adopted a forward-looking low-bit scheme: weights are stored in MXFP4 format, and activations use MXFP8.

Compared with conventional FP16 or BF16, MXFP4 shrinks the model to roughly a quarter of its size, but usually at the cost of noticeable precision loss. To offset this, the model was trained with quantization-aware training (QAT): during training, the network simulates the noise and truncation effects of low-bit quantization, so the weights converge to a representation adapted to low precision. This training-side strategy preserves inference quality under MXFP4/MXFP8 as much as possible, striking a balance between memory footprint and model performance.

Deployment: Hardware Requirements and Integration Options

Given the model’s sheer size, the hardware barrier for local deployment is extremely high. According to community estimates, smoothly loading and running a MoE model at this scale requires a professional server with at least 8 high-end AI accelerator cards (about 288GB of memory per card, roughly 2.3TB in total), with hardware procurement costs likely falling between $550,000 and $650,000. Models at this scale are therefore aimed primarily at large enterprises and research institutions; full inference is out of reach on consumer-grade hardware.

On the software side, mainstream inference frameworks have all shipped compatible support. The following is a conceptual deployment configuration based on a mainstream inference engine:

# Conceptual inference engine deployment configuration
model_config:
  model_name: "ultra-moe-2.8t"
  tensor_parallel_size: 8  # 8-way tensor parallelism across multiple cards
  quantization: "mxfp4"   # precision for loading weights
  max_model_len: 1000000  # maximum context length
  trust_remote_code: true

For users who cannot deploy locally, cloud-native APIs are a more practical option. At the API level, the model offers standardized interface compatibility and supports different reasoning effort levels (such as low, high, and max) to match scenarios with varying latency and reasoning-depth requirements. In complex multi-turn tool-calling scenarios, developers should pay particular attention to context management: the thinking process and tool call records generated in the previous round must be passed back, so that the model’s state remains consistent and its decisions stay continuous.

Agent Capability Evaluation: Reliability on Real Conversations

A model’s true capability shows not only in static benchmarks but also in how reliably it executes complex tasks as an agent. In an agent evaluation built on real user sessions, the model ranked first in the open-weight category. Rather than relying on a single automated script, the evaluation analyzed 17,500 real long-horizon sessions and extracted multi-dimensional signals, including whether users explicitly confirmed task success, spontaneous praise or complaints, the quality of the model’s responses after being corrected, and the success rate of Shell command execution.

The results show an overall net improvement of +5.1%, with first place among open models in all three core categories: Work, Code, and Chat. Notably, on Bash command execution reliability, the model posted a significant +7.9% positive improvement, whereas the previous generation scored negative on this item. This points to a qualitative leap in understanding system instructions, generating executable code, and interacting with the environment.

Conclusion

The open-weight release of this 2.8T-parameter MoE model represents the current frontier of large-model engineering practice. Optimizing long-context handling with gated sparse attention, compressing model size through quantization-aware training, and keeping activated parameters in check through extreme MoE sparsity together offer the industry an engineering blueprint for handling ultra-large models. Its strong showing in real-world agent evaluations further confirms the potential of trillion-scale sparse architectures for complex task execution. For organizations aiming to go deep on large-model infrastructure and agent applications, understanding and adopting these techniques will be key to building the next generation of AI infrastructure.