Trillion-Parameter MoE Weights Open for Download: Architecture, Deployment, and API Guide

The competition in large language models is shifting from benchmark rankings to accessibility. When a Mixture-of-Experts (MoE) model with a total parameter count reaching the trillion level places its complete weights in a public model repository, the real change for developers is not just another API endpoint. Instead, it offers a technical path that can be audited, fine-tuned, and privately deployed. This article breaks down the structural design, quantization storage scheme, deployment thresholds, and invocation details of a typical trillion-scale MoE weight release, serving as an actionable technical reference for teams planning to build their own inference services.

1. What Open Weights Mean

Over the past year, frontier models have mostly been provided as APIs with closed weights, restricting developers to predefined interface boundaries. Open-sourcing weights delivers three critical capabilities to the user:

• Auditability: Structural configurations, expert counts, layer types, and precision schemes can be directly verified by reading the config files, rather than relying solely on official descriptions.
• Fine-tunability: Teams can perform continued pre-training or Parameter-Efficient Fine-Tuning (PEFT) on proprietary data to align general capabilities to vertical domains.
• Private Deployment: Data remains within the internal network, and the inference pipeline is autonomously controlled, satisfying compliance and data sovereignty requirements.

The model in focus is an MoE architecture with approximately 2.8 trillion total parameters and around 104B activated parameters per token. It supports a context length of up to one million tokens and multimodal (image-text) inputs. The model repository indicates that the weights occupy about 1.56TB, with the parameter count tallying up to roughly 2.78 trillion. For teams looking to build their own inference services, this is a highly tangible asset, but one that simultaneously demands extreme hardware and engineering capabilities.

2. Architecture: Sparse Attention and Fine-Grained Experts

2.1 Layer Composition

According to the public model configuration, the architecture consists of 93 layers: 69 layers utilize linear attention modules (KDA), while 24 layers employ Gated Dense Sparse Attention (Gated DSA). A key evolution in this generation is the replacement of the previous full-attention Gated MLA layers with Gated DSA layers. The goal is to significantly reduce attention computation and memory overhead in long-context scenarios.

The cost of long-context modeling primarily stems from the quadratic growth of the attention matrix relative to sequence length. Sparse attention addresses this by allowing each token to attend only to a subset of historical positions, compressing complexity to a more manageable scale. The “gated” mechanism allows the model to learn how much information to aggregate at different layers and positions, mitigating the information loss typically caused by sparsification. The combination of both is a mainstream engineering approach to maintain feasible inference costs at a million-token context scale.

2.2 Expert Layer Configuration

The expert layer consists of 896 experts, with each token routing to 16 selected experts, alongside 2 shared experts. These numbers are particularly noteworthy:

• High expert count, small per-expert capacity: The fine-grained segmentation of 896 experts improves parameter utilization, enabling tokens from different semantic domains to route to highly specialized sub-networks.
• 16 activated experts: Compared to the top-2 routing common in early MoE models, top-16 routing significantly increases the per-token computation cost. However, it delivers stronger expressive power and a smoother routing distribution, enhancing training stability.
• Shared experts: Always-active shared experts handle general knowledge, reducing knowledge redundancy among routed experts. This is a widely adopted practice in recent MoE designs.

The resulting parameter ledger: 2.8 trillion total parameters, with approximately 104B actually participating in computation per token. In other words, the inference compute overhead roughly corresponds to a 100B-scale dense model, while the knowledge capacity is drawn from a trillion-scale parameter pool. This is the core value proposition of MoE and the reason it can strike a balance between performance and cost.

2.3 Visual Pathway

The multimodal component is handled by a visual encoder with roughly 401M parameters, responsible for extracting features from image inputs, which are then jointly modeled with text tokens in a unified sequence. Relative to the size of the main backbone, the visual side accounts for a minimal proportion of parameters, representing a typical “light encoder + heavy backbone” combination.

3. Quantization Scheme: MXFP4 Storage and MXFP8 Activation

The weights are stored in MXFP4 format, activations use MXFP8, and the official documentation notes that Quantization-Aware Training (QAT) was introduced during the training phase. These choices directly dictate the 1.56TB repository size.

A rough calculation: if 2.78 trillion parameters are stored at 4-bit, the theoretical lower bound is approximately 1.39TB. Adding scaling factors, metadata, and some non-quantized tensors, the 1.56TB footprint is reasonable. Without quantization, storing at 16-bit would require over 5.5TB, making download and loading insurmountable for the vast majority of teams.

Several engineering details are worth expanding on:

1. Separation of storage and compute precision: Weights are stored at 4-bit to compress volume and memory footprint, and de-quantized to 8-bit activation precision during computation. This strikes a balance between throughput and numerical stability.
2. The necessity of QAT: Applying Post-Training Quantization (PTQ) directly to a trained model using 4-bit weights usually results in unacceptable accuracy loss. QAT simulates quantization errors during training, allowing the model’s parameter distribution to actively adapt to low-precision representations. This is the prerequisite for a trillion-scale model to confidently utilize 4-bit storage.
3. Practical constraints for fine-tuning: Fine-tuning on 4-bit weights typically requires Low-Rank Adaptation (LoRA) approaches, training only a small number of adapter parameters to avoid the memory explosion of full-parameter updates. Teams planning to fine-tune should first verify their inference framework’s support for adding adapters to quantized weights.

4. Local Deployment: Hardware Thresholds and Framework Selection

4.1 Hardware Estimation

Estimates from third-party technical communities suggest a viable configuration: a server equipped with 8 high-end datacenter GPUs totaling approximately 2.3TB of VRAM. The hardware cost for such a machine falls roughly in the $550,000 to $650,000 range. It must be emphasized that this is a third-party estimate; official hardware recommendations have not been released. Actual requirements depend on concurrency, context length, and whether KV Cache optimizations are enabled.

From an engineering perspective, VRAM consumption generally consists of three parts:

• Weight residency: The 1.56TB-scale weights must be distributed across multiple cards; an 8-GPU single node is a standard starting point.
• KV Cache: At a one-million-token context length, the KV Cache becomes a non-negligible overhead. The sparse attention architecture is specifically designed to suppress this cost.
• Activations and temporary buffers: These scale with batch size and sequence length.

This explains why the barrier to entry for self-hosting such models is not merely “can it run,” but rather “can it run under acceptable concurrency and latency.”

4.2 Inference Framework Support

Mainstream high-throughput inference frameworks, structured generation frameworks, and low-latency serving frameworks have all provided deployment recipes for this model. The general workflow for implementation is as follows:

# General deployment process illustration (using a containerized inference service as an example)
# 1. Pull the model repository weights (~1.56TB, internal network mirroring or high-speed channel recommended)
# 2. Verify weight file integrity and shard indices
# 3. Configure tensor parallelism, quantization backend, and KV Cache strategy according to framework documentation
# 4. Start the inference service and conduct a small-batch smoke test
# 5. Gradually increase concurrency, observing Time-To-First-Token (TTFT) and throughput curves

Engineering recommendations:

• Ensure tensor parallelism matches the GPU count.
• Explicitly enable the quantization backend to successfully load the 4-bit weights.
• In long-context scenarios, prioritize enabling paged KV Cache and prefix caching; otherwise, repeated system prompts will redundantly consume VRAM.

5. Managed Invocation: Interfaces and Parameter Details

Teams unwilling to bear the cost of self-hosting can utilize managed APIs. The model offers two identifiers: a Standard version and a Fast version. The latter is officially stated to be 2 to 3 times faster than the Standard version, making it suitable for latency-sensitive online scenarios, while the Standard version is better suited for batch offline tasks.

At the API level, it is compatible with two mainstream request protocol styles, ensuring low migration costs. Several easily overlooked details directly impact effectiveness:

• Reasoning effort parameter: reasoning_effort accepts low, high, or max, defaulting to max. Lowering this parameter for simple tasks can significantly reduce reasoning token consumption, lowering both cost and latency. Complex reasoning tasks should maintain a high value.
• Multi-turn tool calls: It is correct practice to pass back the previous round’s reasoning_content along with tool_calls. Omitting the reasoning content causes the model to lose the intermediate reasoning chain, manifesting as repeated calls or logical breaks during multi-step tool orchestration.

# Multi-turn tool call callback structure illustration
messages.append({
    "role": "assistant",
    "content": previous_content,
    "reasoning_content": previous_reasoning,   # Must be passed back
    "tool_calls": previous_tool_calls           # Must be passed back
})

6. Selection Guide: Self-Hosted vs. Managed

Here is an actionable decision framework:

Dimension	Self-Hosted Deployment	Managed Invocation
Data Compliance	Data strictly remains internal	Relies on provider compliance commitments
Upfront Investment	Hundreds of thousands of USD in hardware + ops personnel	Pay-per-token, no fixed assets
Customization	Fine-tunable, structurally modifiable, auditable	Limited to API parameters
Elasticity	Long scaling cycles	Instant elasticity
Applicable Scenarios	Long-term high-frequency, strict compliance, deep customization needs	Validation phase, fluctuating loads, rapid deployment

A pragmatic approach is to first use managed APIs for efficacy validation and cost estimation, evaluating self-hosting only after business value is confirmed. If self-hosting is chosen, conduct single-node, low-concurrency stress tests first to obtain real VRAM usage and latency data before deciding on procurement scale. For trillion-scale models, paper estimates and actual measurements often diverge significantly.

The significance of open weights ultimately lies not in the magnitude of the parameter count, but in transforming “capability” into an “engineerable component.” For developers, the question to answer remains the same: does the cost structure of this component match your business load?