Practices for Open-Weight Deployment of Ultra-Large-Scale Sparse Models

1. Background: The Shift Towards Open Ultra-Large-Scale Sparse Models

As large language models (LLMs) continue to advance in long-context comprehension, tool calling, and agentic tasks, the tension between parameter scale and inference cost becomes increasingly prominent. Fully dense models experience rapid growth in computation and memory consumption after reaching a certain scale. In contrast, Mixture of Experts (MoE) architectures control actual computational overhead during inference by maintaining multiple expert networks in each layer while activating only a subset per token, allowing total parameter counts to scale without a proportional increase in compute.

Recently, an MoE model with approximately 2.8 trillion total parameters and around 104 billion activated parameters per token was released with open weights. This provides teams looking to deploy, fine-tune, audit, or research internal model behaviors with a relatively comprehensive observation window.

From an iteration perspective, this model is not an isolated release but an upgrade following its predecessor’s success in agentic task evaluations. The previous version achieved a net improvement score of approximately +0.91% in long-horizon agentic sessions, ranking sixth among open models. The new generation, however, reaches roughly +5.1% in the same evaluation framework, ranking first among open models and thirteenth overall. This leap is driven by structural adjustments as well as updates in training strategies and quantization methods.

2. Model Architecture: Key Changes from Attention to Expert Layers

2.1 Layer Structure and Attention Mechanism

According to the model card, the architecture consists of 93 layers: 69 layers utilize a structure known as KDA, while 24 layers use Gated DSA. Compared to its predecessor, the new model replaces the fully-attended gated low-rank attention layers with gated sparse attention layers to improve long-context modeling efficiency.

KDA can be understood as an attention variant combining key-value cache compression with low-rank representation. Its core idea is to avoid storing high-dimensional key-value vectors for all historical tokens in long-sequence scenarios. Instead, it compresses them into low-dimensional latent representations, which are then restored during attention computation. This effectively reduces cache consumption during long-context inference.

Gated DSA emphasizes sparsity. Traditional attention requires every query to interact with all historical keys, whereas sparse attention selectively computes against a subset of keys—filtered by block, stride, or retrieval relevance. By introducing a gating mechanism, the model learns when to use the sparse path versus a denser path, striking a better balance between efficiency and performance.

2.2 Expert Layer Configuration

The expert layer comprises 896 experts, with each token routed to 16 of them, alongside 2 shared experts. Shared experts typically handle cross-domain foundational representations, while routed experts are selectively activated based on token characteristics. Activating approximately 104 billion parameters per token indicates that despite the massive total parameter scale, the actual computational load per forward pass remains relatively manageable.

2.3 Vision Module

The model supports multimodal image-text input, utilizing a vision encoder with roughly 401 million parameters. This encoder converts images into token sequences that enter the backbone network alongside text tokens. For agentic tasks requiring the simultaneous understanding of images and text within long documents, this architecture reduces the integration complexity typically associated with external vision models.

3. Weight Storage and Quantization Strategy

The weight release adopts a mixed-precision scheme: weights are stored in MXFP4 format, while activations use MXFP8 format during computation. Official documentation states that Quantization-Aware Training (QAT) was employed. This means the model accounted for numerical errors introduced by low-precision representations during the training or continued pre-training phase, rather than applying simple post-training quantization.

MXFP4 and MXFP8 are low-precision floating-point formats designed for hardware acceleration. Compared to traditional 8-bit or 4-bit integer quantization, these formats align better with the numerical distribution of neural network weights and activations. Storing weights in 4-bit significantly reduces disk and VRAM footprints, while 8-bit activations maintain numerical stability during inference.

Repository information indicates that the model repository occupies approximately 1.56TB, with safetensors files accounting for around 2.78 trillion parameters. This scale implies that standard single-machine environments cannot load the model directly; multi-node setups or high-VRAM servers are practically mandatory. Teams considering full-parameter fine-tuning must also account for the additional VRAM required for optimizer states, gradients, and activation checkpoints, meaning actual memory requirements will far exceed the storage size of the weights alone.

4. Deployment Methods and Engineering Thresholds

4.1 Inference Framework Support

Several mainstream high-performance inference frameworks within the community now offer deployment recipes. These frameworks typically support tensor parallelism, pipeline parallelism, and expert parallelism to shard the model across multiple accelerators. For MoE models, expert parallelism is particularly crucial: different experts can be distributed across devices, and the routing results for each token dictate which devices need to be accessed.

A standard deployment workflow generally includes the following steps:

1. Download weights and convert them to the target framework's recognized format;
2. Configure parallel dimensions, such as tensor parallelism degree, expert parallelism degree, and pipeline parallelism degree;
3. Specify quantization precision and cache strategies;
4. Start the inference service and verify context length and output quality.

Because the model supports a context length of roughly 1 million tokens, reasonable cache configuration for long sequences is essential during deployment. The benefits of sparse attention can only be realized if the caching strategy is configured correctly; if cache memory is reserved based on full attention mechanisms, the VRAM requirements will be substantially higher.

4.2 Hardware Cost Estimation

Community members have provided estimates based on VRAM requirements. One viable configuration involves a single server equipped with 8 high-performance accelerators, offering a total of about 2.3TB of VRAM, with hardware costs estimated between $550,000 and $650,000. This is a third-party estimate rather than an official recommendation, but it helps teams establish a baseline budget expectation.

For most small to medium-sized teams, purchasing dedicated hardware is impractical, making on-demand cloud GPU instances or direct API calls more viable alternatives. The API format is typically compatible with mainstream conversational interfaces, allowing configurable model names and parameters. The reasoning effort parameter can be set to low, high, or maximum, defaulting to maximum. During multi-turn tool calling, the reasoning content and tool call records from the previous turn must be passed back to maintain context consistency.

5. Agentic Evaluation: The Significance of Net Improvement

Evaluating agentic tasks differs from static question-answering. User sessions often involve multi-turn interactions, command-line executions, file edits, and result confirmations. Whether a model genuinely helps the user complete a task cannot be judged solely by semantic accuracy of a single response; it requires observing whether the task concludes successfully.

The evaluation system extracts several signals from real user sessions: whether the user explicitly confirms task success, whether the user expresses approval or complaints, whether the model improves after user corrections, and whether shell commands execute successfully. The “net improvement” metric calculated from these signals represents the degree of effective progress a model brings to real, long-horizon tasks.

The new generation model achieved a net improvement of +5.1% across approximately 17,500 real long-horizon sessions, ranking first among open models. Looking at specific categories, its command-line reliability saw significant improvement, shifting from a previous negative value to approximately +7.9%. This indicates that in real terminal operation scenarios, the model is less likely to execute destructive commands or invalid operations. Across the work, code, and conversation categories, the model secured the top spot among open models.

The value of these metrics lies in their proximity to actual user experience rather than fixed benchmarks. A model might score highly on standard test sets, but its agentic value remains limited if it cannot execute long-horizon tool calls stably. With open weights, teams can reproduce or improve these evaluations on their own task sets.

6. Practical Recommendations

Teams looking to utilize this model can choose different paths based on their specific conditions:

1. Direct API Integration: Suitable for most application developers. There is no need to manage underlying hardware or parallel configurations; simply pass in messages, tool definitions, and historical reasoning content according to the API specification.
2. Cloud GPU Instance Deployment: Ideal for teams requiring privatization, data sovereignty, or custom sampling. Refer to the community inference framework’s parallel configuration guidelines during deployment.
3. Full-Parameter Fine-Tuning or Continued Training: Only viable for teams with large-scale training experience. Beyond the 1.56TB of weights, sufficient space must be allocated for gradients, optimizer states, and activation checkpoints, alongside designing a sound expert load-balancing strategy.
4. Research and Auditing: Open weights enable researchers to analyze expert routing patterns, attention sparsity, and long-context caching behaviors, providing valuable reference points for understanding how ultra-large-scale sparse models operate.

During engineering implementation, it is advisable to start with small-scale context lengths and a limited number of requests to verify the framework version, parallel configurations, and output quality before scaling up. For long-context scenarios, focus on testing whether the model maintains stable command execution and information retrieval capabilities under inputs of hundreds of thousands of tokens.

Overall, the open-weight release of this model provides a new sample for the engineering deployment of ultra-large-scale sparse models. Its structural design, quantization strategy, and agentic evaluation results collectively demonstrate that while parameter scales continue to expand, maintaining controllable inference costs through sparse activation and low-precision training is a highly viable path for the ongoing evolution of large model technology.