Deep Dive into Trillion-Parameter MoE Open Weights: Architecture, Quantization, and Deployment

1. Open Weight Release: A Turning Point in LLM Provisioning

Recently, a leading AI team released their next-generation flagship model and fulfilled their promise to open the full weights to the community. The model features approximately 2.8 trillion total parameters and utilizes a Mixture-of-Experts (MoE) architecture, activating only about 104B parameters per token. It supports a million-token context window and multimodal (image-text) inputs. For engineering teams, this event signifies more than just “another downloadable model”; it marks a transition in how ultra-large models are provisioned—moving from API-only consumption to self-hosted, auditable, and customizable deployments.

It is crucial to clarify a concept first: open weights do not equate to complete open source. While the weight files are downloadable, the training data, full training code, and configuration scripts may not be publicly released. Therefore, the capability boundaries brought by open weights primarily fall into three categories:

1. Autonomous Deployment: Keeping data within your own domain to meet compliance requirements.
2. Deep Auditing: Allowing security teams to inspect weight files and verify behavioral consistency.
3. Secondary Development: Validating and adapting the model on proprietary evaluation datasets.

These three capabilities are highly valued in enterprise deployments, making every release of trillion-scale open weights worthy of detailed engineering analysis.

2. Architecture Breakdown: Hybrid Orchestration of Attention Mechanisms

According to public model specifications, the architecture consists of 93 layers. Among these, 69 layers employ Gated Delta Attention (a variant within the linear attention family), while 24 layers use Gated Sparse Attention. Compared to its predecessor, this generation replaces standard full attention layers with sparse attention layers to reduce the computational overhead of long-context modeling.

The technical logic behind this design is clear and worth expanding upon:

• Linear Attention Layers act as “State Memory”: Traditional attention requires O(n²) computation for a sequence of length n, and the KV cache expands linearly with the context. Linear attention compresses historical information into a fixed-size recurrent state, reducing computation to linear complexity and keeping the state memory constant regardless of context length. Whether the context is 10,000 or 1,000,000 tokens, the memory footprint for these layers remains essentially unchanged. This is the engineering cornerstone enabling million-token contexts.
• Sparse Attention Layers act as “Key Focusing”: These layers only perform global, fine-grained computations on a small number of tokens deemed critical, replacing full pairwise interactions. By using a few sparse layers for global information integration and a majority of lightweight layers for maintaining long-term state, the model approaches the modeling quality of full attention but with significantly lower resource overhead.
• Hybrid Orchestration is an Art of Trade-off: Approximately three-quarters of the 93 layers use linear mechanisms, and one-quarter use sparse global mechanisms. This indicates the designers’ judgment: the majority of comprehension and memory tasks can be handled by inexpensive mechanisms, with only a few steps requiring an “expensive global perspective.”

The design of the expert layers is equally typical: there are 896 experts in total. For each token, the router selects the 16 best-matching experts, while 2 shared experts always participate in the computation. With a total capacity of 2.8T parameters and a per-token compute requirement of 104B, it embodies the core MoE benefit of “large capacity, small computation”—the model’s knowledge base can be massive, while the floating-point operations per inference remain controlled. The shared experts handle high-frequency, general knowledge, preventing the router from repeatedly assigning similar base computations to different experts and thereby improving parameter utilization. Furthermore, the model is equipped with a visual encoder of approximately 401M parameters, jointly supporting image-text inputs alongside the language backbone. In Agent scenarios, visual capabilities mean the model can directly read screenshots and terminal errors, which has practical value for automated operations and maintenance tasks.

3. Quantization and Storage: MXFP4 Empowered by Quantization-Aware Training

The model weights are stored in MXFP4 format, activations use MXFP8, and official documentation confirms the use of Quantization-Aware Training (QAT). These two points are worth examining separately.

What is MXFP4: It belongs to the Microscaling Floating-Point (MXFP) format standard, using 4-bit floating-point representation for a single element, with every 32 elements sharing a single scaling factor. Compared to traditional INT4 integer quantization, floating-point representation is more friendly to dynamic ranges and outliers. Large model weight distributions often have long tails; integer quantization can easily have its precision for the majority of small values “squeezed” by a few large values. Floating-point blocks with shared scaling factors can adaptively adjust the scale within the block.

QAT vs. Post-Training Quantization (PTQ): PTQ is applied after training is complete, and precision loss carries uncertainty. QAT, on the other hand, simulates quantization errors during the training process, allowing the weights to “natively grow” within the quantized domain. For trillion-scale MoE models, this is especially critical: with a massive number of experts and routing decisions highly sensitive to numerical values, errors from PTQ would accumulate and amplify across routing and expert computations. The official choice to use QAT and distribute weights directly in MXFP4 essentially front-loads the quantization process to the training phase, shifting the responsibility of precision control to the training side.

A Storage Calculation to Verify the Claims: Community platform interfaces show the repository size at approximately 1.56TB, with weight files indicating about 2.78 trillion parameters. Calculating 2.78 trillion parameters at 0.5 bytes per parameter yields about 1.39TB. Add to this the 1-byte scaling factor per 32 elements (approximately 87GB), plus non-quantized components like word embeddings, normalization layers, and the visual encoder—which typically retain higher precision—the total matches the observed 1.56TB. This demonstrates mathematical consistency between the repository size and the claimed parameter count. It also serves as a low-cost, high-value verification step when auditing open weights. On the download side, it is recommended to use sharded downloads, verify hashes individually, and ensure that the chosen inference engine version supports MXFP4 de-quantization operators.

4. Deployment Barriers: VRAM Estimation and Inference Engine Adaptation

The biggest obstacle to self-deployment is hardware resources. VRAM requirements are roughly composed of three parts:

1. Weight Resident Memory: Approximately 1.56TB, serving as the absolute baseline.
2. State and KV Cache: Linear attention layer states are fixed, and sparse attention layer caches are much smaller than full attention, but when million-token contexts are combined with high concurrency, the absolute numbers remain substantial.
3. Activations and Framework Overhead: It is generally recommended to reserve an additional 10% to 20% on top of the above.

Community developers have estimated that a viable configuration could involve a server with 8 flagship data center accelerators (each with hundreds of GBs of VRAM), totaling around 2.3TB of VRAM, representing a hardware investment in the range of hundreds of thousands of dollars. It must be emphasized that this is a third-party estimate; official hardware recommendations have not been provided. Actual sizing must be determined through stress testing based on target context lengths and concurrency levels. The realistic conclusion for most teams is that the barrier to full self-hosting is extremely high, making it suitable only for organizations with strict compliance mandates or massive traffic demands.

If self-deployment is indeed necessary, several engineering priorities should be noted:

• Expert Parallelism (EP) is the main strategy for MoE deployment, combined with Tensor Parallelism (TP) and Pipeline Parallelism (PP) for sharding. The optimization of all-to-all communication between experts directly dictates throughput.
• Linear attention and sparse attention require specialized operator support. When selecting an inference engine, verify operator coverage and numerical consistency one by one.
• Several mainstream inference engines have already released deployment recipes for this model. It is advisable to strictly align with the versions and startup parameters in these recipes to establish a baseline before attempting custom tuning.
• Before going live, three verifications are mandatory: quantization precision regression testing against a reference implementation, extreme stress testing for long contexts, and disaster recovery drills.

5. API Consumption: Tiered Configuration and Multi-Turn Tool Calling

For teams without the resources to build their own infrastructure, consuming the model via API is a more realistic path. They can also leverage model service platforms, such as Baidu AI Cloud, to obtain hosted inference and computing power, allowing them to validate business use cases before deciding on deeper investments. Based on the released information, the service-oriented design of this model offers several universally applicable patterns:

• Dual-Tier Versions: Providing both a Standard and a Fast version, with the latter reportedly being 2 to 3 times faster. Splitting services by “deep reasoning” and “fast response” is a common practice for balancing latency and quality.
• Configurable Reasoning Effort: Offering three levels—low, high, and max (defaulting to max). Higher effort means more thorough reasoning, but also increased latency and cost. Engineering teams should map these to task tiers: use low for format conversion and simple Q&A, high for code generation and multi-step reasoning, and max for high-stakes decision-making tasks.
• Multi-Turn Tool Calling Must Return Reasoning Content and Call Records: This is the most easily overlooked detail. The model operates via a “reasoning chain + action” paradigm. If the next turn only passes the final answer without returning the previous turn’s reasoning content and tool calls, the cross-turn causal chain will break, significantly reducing the success rate of complex tasks. An example structure is as follows (field names are illustrative):

{
  "messages": [
    {"role": "user", "content": "Troubleshoot the online interface 5xx issue"},
    {"role": "assistant",
     "reasoning_content": "Check gateway logs first, then locate downstream dependencies...",
     "tool_calls": [{"name": "shell", "arguments": {"cmd": "grep 5xx gateway.log"}}]},
    {"role": "tool", "content": "Located upstream service timeout"}
  ]
}

Accompanying engineering practices include setting up whitelists and sandbox isolation for tool calls—especially permission restrictions for shell-like operations—and fully logging each turn’s reasoning content and tool returns for post-hoc auditing and fault replay.

6. Evaluation Methodology: The “Net Benefit” Metric Driven by Real Sessions

Concurrent with this release, a noteworthy evaluation methodology was introduced: the “Net Benefit” metric based on real user Agent sessions. Unlike static benchmarks, this data is derived from real, long-horizon task sessions and extracts four types of signals—whether the user explicitly confirms task success, user praise or complaints, the model’s response quality after user corrections, and whether shell commands executed successfully. The “Net Benefit” is then calculated (successful signals minus failure and penalty signals). Data from a real distribution is much harder to overfit with targeted optimization, and “scoring based on complete task success or failure” aligns more closely with production evaluation criteria.

Case data also illustrates the value of fine-grained sub-metrics: based on over 17,500 real long-horizon sessions, the model achieved a Net Benefit of +5.1%, whereas the previous generation only reached +0.91%. In sub-categories, command-line execution reliability improved from -4.5% in the previous generation to +7.9%. Models with similar overall scores may have drastically different shortcomings—the previous generation’s issues were precisely concentrated in tool execution. The takeaway for practitioners is that model selection should not rely solely on comprehensive rankings. Instead, map the sub-metrics to your own task distribution (coding, office tasks, dialogue) and consider building a proprietary evaluation set for regression testing.

7. Deployment Decision Framework: Build vs. API

Finally, here is a concise decision framework:

• Highly Sensitive Data, Massive Long-Term Usage, Need for Deep Customization: Evaluate self-hosting. First, calculate the Total Cost of Ownership (TCO)—hardware depreciation, electricity, dedicated operations staff, and cluster utilization—and compare it against API pricing to avoid “building for the sake of building.”
• General Business Scenarios: Start with APIs or cloud-hosted platforms. For example, leverage the model service capabilities of platforms like Baidu AI Cloud for rapid integration and validation, and decide whether to transition to self-hosting after the value is confirmed.
• Regardless of the path chosen, three actions are indispensable: Establish a proprietary evaluation set matching your business distribution; perform precision regression testing on quantized versions; and implement auditing, sandboxing, and replay mechanisms for tool calling pipelines.

Overall, the signals released by these open weights are clear: trillion-parameter models are transitioning from “lab narratives” to “engineering objects.” Architecturally, hybrid attention is used to make long contexts feasible; in training, Quantization-Aware Training trades for storage and inference efficiency; and in evaluation, real user sessions provide credibility closer to production environments. For developers, understanding these three main threads holds far more long-term value than memorizing any leaderboard ranking.