Trillion-Parameter MoE Open Weights: A Deep Dive into Architecture, Quantization, and Deployment

I. Open Weights: The Watershed from “Usable” to “Controllable”

Recently, an AI team fulfilled its commitment to releasing open weights for a Mixture-of-Experts (MoE) model with approximately 2.8 trillion total parameters and around 104B activated parameters per token. The significance of this release includes:

• Self-deployment capability: Data remains within the user’s domain, meeting the strict compliance and privacy requirements of sectors like finance, healthcare, and government.
• Fine-tuning capability: Enterprises can perform continued training or preference alignment using proprietary business data, moving beyond mere prompt engineering.
• Auditability: With weights and model cards publicly available, third parties can verify architecture claims, quantization schemes, and evaluation metrics, reducing the information asymmetry caused by “black-box” marketing.

It is important to note that “open weights” do not equate to open training data or a fully open training process. Its degree of openness lies between closed APIs and fully open-source models. Teams should verify license scopes, commercial terms, and compliance obligations for derivative models before adoption to avoid future disputes.

II. Decoding the Model Card: Key MoE Architecture Numbers

The core information on the model card can be summarized into four sets of numbers, each driven by specific design motivations:

1. 2.8T Total Parameters, 104B Activated Parameters. The essence of MoE is splitting feed-forward layers into multiple “experts,” with a router dynamically selecting a small subset of experts per token for computation. Total parameters dictate the knowledge capacity ceiling, while activated parameters determine the compute cost per inference. A 2.8T versus 104B configuration implies an activation ratio of roughly 3.7%—an aggressive tier that balances capacity and cost, enabling massive knowledge coverage at a controllable inference expense.

2. 93-Layer Hybrid Attention Stack. This consists of 69 layers using incremental linear attention and 24 layers using gated sparse attention, replacing the traditional full attention design in Transformers. The state size of linear attention remains nearly constant regardless of sequence length, while sparse attention only computes correlations at key positions. Together, they push the context window to the 1-million-token scale while keeping memory footprint and latency growth linear.

3. Expert Configuration: 896 Routed Experts, 16 Selected per Token, and 2 Shared Experts. Shared experts carry cross-task general knowledge, while routed experts handle domain-specific differentiation. This “shared + routed” combination is a mainstream practice for large-scale MoE models, effectively mitigating knowledge redundancy and routing oscillation among experts.

4. Multimodal Integration. A vision encoder with approximately 401M parameters converts images into visual tokens, which are concatenated with text tokens before entering the main network. This enables mixed image-text inputs, providing a foundation for document understanding and chart analysis tasks.

Additionally, there is a slight discrepancy between the claimed total parameters (2.8T) and the metadata statistics of the weight files (~2.78T). This typically stems from components like word embeddings or vision modules that are not stored in the standard quantization format. This is a normal phenomenon and does not warrant over-interpretation during audits.

III. Quantization-Aware Training: How the 1.56TB Footprint is Calculated

The model weights are stored in MXFP4 format, with activations in MXFP8. Official documentation states the use of Quantization-Aware Training (QAT). Two technical points are worth examining here:

First, Microscaled Floating-Point Formats. The MXFP series partitions tensors into fixed-size blocks, sharing a single scaling factor per block, and stores values as 4-bit or 8-bit floating-point numbers:

# Core concept of MXFP-like formats (illustrative)
def mxfp_quantize(w, block=32, bits=4):
    blk   = w.view(-1, block)
    scale = blk.abs().amax(dim=1) / fp_max(bits)   # One scaling factor per block
    q = (blk / scale).round().clamp(-fp_max(bits), fp_max(bits))
    return q.to(low_precision_dtype(bits)), scale

Block-level scaling fits numerical distributions more granularly than traditional per-tensor scaling, offering higher fidelity at the same bit width. This is why it can sustain the main weight storage task for a trillion-parameter model.

Second, Quantization-Aware Training. Post-Training Quantization (PTQ) compresses the model after training, dealing with errors retrospectively. QAT, conversely, simulates quantization-induced rounding errors during the training process, allowing network parameters to reconverge under low-precision representations. For an aggressive bit width like 4-bit, relying solely on PTQ usually results in noticeable performance degradation. QAT is the critical prerequisite for making 4-bit weights a “first-class citizen.”

A practical self-check method: Divide the repository size of ~1.56TB by the ~2.78T parameters, yielding approximately 0.56 bytes per parameter (around 4.5 bits). This aligns with the assumption of “4-bit main weight storage with a few modules retaining higher precision.” Volume and bit width can mutually verify each other—a low-cost, high-value technique when auditing open weights.

IV. Local Deployment: VRAM Math and Hardware Thresholds

The first step in self-deployment is calculating the bill. VRAM requirements roughly consist of three parts:

• Weight footprint: ~1.56TB;
• KV Cache: Proportional to concurrency and context length. Even with sparsification designs, careful calculation is required for million-token scenarios;
• Activations and framework overhead: Typically reserve a 10%~20% margin.

Based on these metrics, community estimates suggest a server equipped with 8 flagship data-center accelerators, totaling about 2.3TB of VRAM. The hardware investment sits in the hundreds of thousands of USD range. It must be emphasized that these are third-party estimates; the official team has not provided hardware recommendations. Actual configurations will also depend on how well the chosen inference engine supports 4-bit de-quantization operators and sparse attention implementations.

Fortunately, mainstream open-source inference frameworks released comprehensive deployment solutions on the day the weights were made available, covering tensor parallelism, expert parallelism, and shard loading. For teams planning self-deployment, the recommended path to production is:

1. Small-scale validation first: Use the framework’s example configurations to launch the service and establish the shortest functional pipeline;
2. Stress-test key metrics: Measure Time-to-First-Token (TTFT) and throughput under varying concurrency and context lengths;
3. Scale up afterward: Confirm business ROI before evaluating full-machine procurement or rental options.

V. Skipping the Cluster: Engineering Details for the API Route

For most teams, building a trillion-parameter cluster is unrealistic, making API calls a more rational starting point. Such models typically offer both standard and fast versions (with official speeds claiming 2~3x faster) and are compatible with mainstream conversational API specifications, ensuring low migration costs. Two engineering details warrant attention during usage:

• Reasoning effort parameters: Generally provided in tiers like low/high/max, controlling the depth of reasoning to balance quality and cost. The default tier is often the deepest reasoning mode. For simple tasks, explicitly lowering the tier avoids unnecessary token consumption.
• Context passback in multi-turn tool calls: In Agentic scenarios, the reasoning content and tool call records from the previous turn must be passed back. Otherwise, the reasoning chain breaks across turns, resulting in repeated tool calls or lost intermediate conclusions.

# Key points for multi-turn tool call passback (pseudo-code)
messages.append({
    "role": "assistant",
    "content": None,
    "reasoning_content": last_reasoning,   # Reasoning content must be passed back as-is
    "tool_calls": last_tool_calls,
})
messages.append({
    "role": "tool",
    "tool_call_id": call_id,
    "content": tool_result,
})

Of course, if teams prefer to focus entirely on business logic, utilizing hosted large model services via API—such as those provided by Baidu AI Cloud—typically offers better cost-performance than building a custom cluster. It eliminates the continuous burdens of hardware procurement, operations, and version upgrades.

VI. Evaluation Interpretation: Real Sessions vs. Static Benchmarks

On the day of the weight release, a third-party agentic evaluation leaderboard published results: based on over 17,500 long-context sessions from real users, the model achieved a net improvement of +5.1% under the deepest reasoning tier. It ranked 1st among open-weight models and 13th overall, leading in the office, coding, and dialogue categories among open models. The previous generation model scored +0.91% and ranked 6th among open models. In terms of command execution reliability, it improved to +7.9%, compared to the previous model’s −4.5%.

The fundamental difference between this type of evaluation and static benchmarks lies in the data source. It reads multiple signals from real sessions—whether the user explicitly confirms task success, praises or complains about the model, the model’s response quality after user corrections, and whether shell commands actually executed successfully—to calculate “net improvement.” Its reference value lies in reflecting real-world usage, but three points should be noted during interpretation:

1. Sample bias: The demographic and task distribution of session users may not match your specific business scenario;
2. Metric calibration: Net improvement is a relative metric and should be assessed alongside absolute success rates;
3. Condition consistency: Different reasoning tiers and API configurations affect scores. Conditions must be fixed when replicating tests.

The recommendation for enterprises is to use leaderboards as an initial screening tool, then build an evaluation set tailored to their own business (covering typical tasks, failure cases, and boundary inputs), and draw conclusions through double-blind comparisons.

VII. Conclusion: Treating Open Weights as a Capability, Not a Slogan

The release of this generation of trillion-parameter MoE provides the industry with a complete sample: hybrid attention makes long-context costs controllable, QAT combined with microscaled floating-point formats makes 4-bit storage possible, and real-session evaluations make capability claims verifiable. For technical teams, the real priority is translating “open weights” into an actionable checklist:

• Define objectives: Is the goal self-deployment, fine-tuning, or merely auditing?
• Account for resources: Weights + KV cache + margins. Don’t overlook interconnect bandwidth and storage throughput.
• Validate incrementally: Use APIs to verify capability boundaries first before deciding on privatization.
• Align evaluations: Define “usable” with business data rather than relying solely on leaderboard rankings.

Open weights return the power of choice to the user. Effectively leveraging this choice fundamentally tests the team’s own engineering capabilities and evaluation systems.