Open Weights for Trillion-Scale Sparse MoE Models: Structure, Deployment, and Evaluation Analyzed

1. Background: The Significance of Open Weights

Recently, a leading AI model team released the complete weights for their new flagship model for public download, fulfilling their previous commitment. The model features a trillion-scale total parameter count, utilizes a sparse Mixture-of-Experts (MoE) architecture, and supports ultra-long context lengths alongside image-text inputs. For teams looking to self-host, fine-tune, or conduct security audits, this transition from API-only access to full control over the inference stack is a critical milestone. For the inference framework ecosystem, it presents an immediate challenge to adapt to new structural designs, precision formats, and parallelization strategies.

Rather than reiterating press release talking points, this article breaks down three core questions from an engineering perspective: What actually changed in the model structure? How do weight and activation precision schemes impact deployment? And how should we objectively evaluate the real-world performance of such models in Agent scenarios?

2. Model Structure: Attention and Expert Layer Design Trade-offs

2.1 Layer Count and Attention Type Ratio

According to public model card information, the model consists of 93 layers: 69 layers utilize a KDA structure, while 24 layers employ Gated Sparse Attention. Compared to its predecessor, the most notable change is replacing the original full attention gating layers with gated sparse attention layers, aiming to reduce the computational and memory overhead of ultra-long context modeling.

The technical logic here is worth expanding upon. When full attention scales to context lengths reaching a million tokens, the attention matrix grows quadratically, making KV Cache usage a deployment bottleneck. Sparse attention reduces complexity to near-linear by ensuring each token only attends to a subset of historical positions. The “gating” mechanism introduces learnable gate weights on top of sparse selection, allowing the model to dynamically decide how much historical information to retain. This combination represents an engineering compromise between long-context capability and inference cost.

2.2 Expert Layer Configuration

The expert layer contains 896 routed experts, with 16 activated per token, alongside 2 shared experts. The key implications of this configuration are:

• Decoupling of total and activated parameters: Although the total parameter count is in the trillions, only tens of billions of parameters are activated for a single token. Consequently, the compute required for a single forward pass is significantly less than that of a dense model with an equivalent total parameter count.
• High number of routed experts: This results in massive weight file sizes. However, during inference, memory bandwidth pressure is concentrated on the activated experts, placing high demands on the scheduling efficiency of Expert Parallelism.
• Shared experts: These typically carry general knowledge and stabilize routing distribution, reducing fluctuations caused by load imbalance among experts.

2.3 Vision Encoder

The visual component is a Vision Transformer variant with approximately 400 million parameters, responsible for encoding images in image-text inputs. While its parameter count is extremely small compared to the main trunk, its preprocessing and feature alignment pipeline consume additional memory and compute during multimodal inference, requiring separate planning during deployment.

3. Precision Scheme: MXFP4 Storage + MXFP8 Activation

3.1 Scheme Highlights

Weights are stored in MXFP4 format, activations use MXFP8, and the official documentation notes that Quantization-Aware Training (QAT) was employed during the training phase. The design intent behind this combination is clear:

• Storage: Using a 4-bit microscaling floating-point format compresses the weight size of trillion-scale parameters to a transferable and storable volume.
• Computation: Using 8-bit activations preserves sufficient numerical dynamic range during matrix multiplication, avoiding precision collapse caused by low-bit activations.
• QAT: This allows the model to “perceive” quantization errors during training, making it more robust to low precision during inference, rather than applying post-training quantization as an afterthought.

Public interfaces indicate the repository occupies about 1.56TB, while the statistical parameter count is approximately 2.78 trillion. These two figures are notable: 1.56TB is the compressed on-disk size, and 2.78 trillion is the decompressed logical parameter count. The ratio between the two visually demonstrates the compression benefits of 4-bit storage.

3.2 Practical Impact on Deployment

Low-bit storage does not mean low-bit computation throughout the entire process. The actual inference pipeline typically involves dequantizing 4-bit weights to compute precision, followed by a matrix multiplication with 8-bit activations. This brings two engineering considerations:

1. Dequantization overhead cannot be ignored: The throughput of layer-by-layer dequantization for trillion-scale parameters directly impacts the Time-To-First-Token (TTFT). Inference frameworks must provide fused operators (e.g., dequantization fused with matrix multiplication) to amortize this cost.
2. Memory estimation requires a holistic approach: Memory usage must be estimated based on “weights + activations + KV Cache.” Although weights are compressed, the decompressed working set, long-context KV Cache, and communication buffers for expert parallelism will stack up. Memory requirements cannot be simply extrapolated from the 1.56TB figure.

4. Deployment Pathways: Adapting Three Types of Inference Frameworks

Mainstream open-source inference frameworks have provided deployment recipes, which can be categorized into three core capability requirements:

• Hybrid Expert and Tensor Parallelism: A trillion-scale MoE cannot fit on a single machine. The 896 experts must be distributed across multiple GPUs or nodes, while attention layers utilize tensor parallelism. Frameworks must support fine-grained expert placement and all-to-all communication optimization.
• Low-bit Kernel Support: Dedicated matrix multiplication and dequantization fused kernels are required for MXFP4 weights and MXFP8 activations. Otherwise, the advantages of the precision format will be negated by the overhead of generic kernels.
• Ultra-long Context and Multimodal Pipeline: Million-token contexts demand PagedAttention, KV Cache quantization, and eviction policies. Image-text inputs require pipeline orchestration between the vision encoder and the main inference trunk.

A typical deployment configuration outline (pseudo-code, for illustrative purposes only):

# Inference service configuration example (not a real interface)
model:
  weight_format: mxfp4
  activation_format: mxfp8
  tensor_parallel: 8
  expert_parallel: 16
  max_context_tokens: 1000000
  kv_cache:
    page_size: 64
    dtype: fp8
  vision_encoder:
    enable: true

It should be emphasized that the fields above only illustrate the dimensions that need explicit planning during deployment. Specific parameter names and values should be referenced in the documentation of the framework being used.

4.1 Hardware Requirements: A Reality Check

A third-party community estimate suggests a viable configuration requires 8 high-end datacenter GPUs providing approximately 2.3TB of total VRAM, with hardware costs in the hundreds of thousands of dollars. While unconfirmed officially, this provides a valuable directional benchmark: local deployment of a trillion-scale MoE is an “enterprise-grade” investment. For small to medium teams, a more realistic approach is to access the model via API and focus engineering efforts on the application layer.

5. API Integration Essentials

Teams not building their own inference stacks can use the model via APIs compatible with mainstream interface protocols. The model name distinguishes between a standard version and a high-speed version, with the latter reportedly offering 2 to 3 times the speed of the former. Engineering considerations include:

• Inference strength parameters: These typically offer low, high, and maximum tiers. The default is usually set to the maximum, implying higher latency and cost. Explicitly lowering this parameter is recommended for simple tasks.
• Multi-turn tool calling: When making multi-turn tool calls, the reasoning content and tool call records from the previous turn must be passed back. Otherwise, the model loses its intermediate reasoning state, breaking the tool call chain. This is one of the most common pitfalls in Agent-based applications.
• Dual protocol compatibility: Compatibility with two mainstream interface protocols means a single codebase can switch via an adaptation layer, reducing migration costs.

6. Agent Capability Evaluation: Understanding Net Improvement Metrics

Concurrent with the weight release, Agent evaluation data was published based on real user long-horizon sessions, with a sample size exceeding 17,500. Its methodology deserves separate discussion because it breaks away from the limitations of traditional static benchmarks.

The signals read during evaluation include: whether the user explicitly confirmed task success, positive and negative user feedback, the model’s response quality after user corrections, and whether shell commands executed successfully. Based on these, a “net improvement” metric is calculated. The rationale behind this design is that success in Agent scenarios is often reflected in the final state after multi-step operations, rather than the fluency of a single-turn response. User confirmation and correction behaviors serve as natural quality annotations, while command execution results are verifiable objective signals.

In terms of results, the model achieved a net improvement of approximately +5.1% at the maximum inference tier, ranking first among open-weight models and 13th overall. It secured the top spot for open-weight models in the work tasks, coding, and dialogue categories. The previous generation at the same tier yielded +0.91%. In specific sub-categories, shell command reliability improved by roughly +7.9% compared to the baseline, ranking first among open-weight models, whereas the previous generation scored negatively and ranked lower.

Key takeaways for readers:

1. Net improvement is an incremental metric relative to a baseline. Absolute values should not be directly compared across different evaluation batches; focus on relative rankings within the same batch.
2. The leap in shell reliability indicates a substantial improvement in the “executability of tool calls” for this generation. This holds greater value for automated operations and code-execution Agents than general conversational scores.
3. Real-world session evaluations suffer from user demographic biases. Enterprises should conduct small-scale A/B tests tailored to their specific business scenarios during model selection, rather than blindly trusting leaderboard rankings.

7. Implementation Recommendations and Risk Alerts

• Evaluate before investing: Before deploying locally, validate your business scenarios using the API. Only assess hardware procurement once model capability compatibility is confirmed.
• End-to-end precision validation: The numerical stability of low-bit weights under long-context and multimodal inputs requires empirical testing. Focus closely on generation quality at the tail end of long sequences.
• Manage Agent states: Multi-turn tool calls must completely pass back reasoning and call records. It is recommended to implement state serialization and replay testing at the application layer.
• Cross-validate evaluation data: Leaderboard rankings serve as a reference, but final decisions should be driven by three metrics on custom task sets: success rate, latency, and cost.
• Compliance and auditing: Following the weight release, fine-tuning and secondary distribution must adhere to the corresponding license terms. Enterprises should establish a model asset registry.

Overall, the release of open weights for trillion-scale sparse MoE models pushes the “engineering threshold of ultra-large models” from theory into practice: structurally trading sparse attention for long-context efficiency, leveraging QAT and low-bit formats for storage and bandwidth optimization, utilizing expert parallelism for deployability, and relying on net improvements from real sessions for credibility. These four main threads construct the complete technical narrative of current ultra-large model implementation.