Trillion-Parameter Sparse Model Weights Released: MXFP4 Quantization Deployment and Long-Context Inference in Practice

1. Background: Why Open Weights Matter

A Mixture-of-Experts model with approximately 2.8 trillion total parameters and roughly 104 billion activated parameters per token has released its full weights publicly. For teams working on large model inference, fine-tuning, auditing, or private deployment, this is more than a number on a model card—it means the ability to reproduce weights in their own environment, inspect the architecture, customize the inference stack, and build secondary development around long-context and tool-calling scenarios.

Unlike the smaller open models that have been common previously, a model of this scale has typically been offered only through closed-source APIs. With open weights, teams can bypass several constraints of hosted interfaces—such as fixed context windows, fixed inference effort levels, and fixed tool-call return formats—to build their own inference services within controllable cost and data boundaries.

But open weights do not mean “download and run.” The model’s storage format, activation strategy, expert parallelism approach, and tool-calling constraints all directly affect deployment feasibility and operating cost. The following sections address these engineering concerns.

2. Model Architecture: How 2.8T Parameters Are Organized

According to the public model description, the model uses a Mixture-of-Experts architecture with 93 layers, of which 69 use a sparse attention mechanism and the remaining 24 use gated sparse attention. Compared with the previous generation, the key change is replacing some full-attention layers with units that are more memory- and compute-efficient for long-context modeling.

The expert-side configuration is as follows:

• Total routed experts: 896;
• Experts activated per token: 16;
• Additional shared experts: 2;
• Vision module: a vision encoder with approximately 401 million parameters.

This “large total parameters, small activated parameters” design is one of the primary ways current long-context models control inference cost. With 104B activated parameters, only a small fraction of the model’s feed-forward computation is activated per token, keeping per-token latency acceptable while retaining capacity advantages.

From an engineering perspective, this produces two direct consequences:

First, the inference service must support expert parallelism. The 896 experts cannot fit entirely into a single GPU’s memory, so they must be partitioned across multiple accelerators along the expert dimension. Different requests may activate different expert combinations at the same moment, requiring efficient expert routing and communication mechanisms.

Second, caching strategy must be tied to expert selection results. Traditional dense models typically only need to cache attention key-value pairs, whereas this type of model must also consider reuse of historical expert activation paths during tool calls and multi-turn conversations—otherwise additional computation is easily incurred.

3. Quantization and Storage: MXFP4 Weights, MXFP8 Activations

The open weights are stored in MXFP4 format, with MXFP8 activations, and the official description states that quantization-aware training was incorporated during the training phase. The key term here is “quantization-aware training,” which indicates that quantization is not simply a post-processing step applied to weights after training, but rather that the model was adapted to low-bit-width representations during training itself.

MXFP4 and MXFP8 belong to Microsoft’s floating-point format family. Unlike the common INT4/INT8, they use a data layout with shared exponents and low-precision mantissas. For weight storage, MXFP4 can significantly reduce disk and GPU memory usage; on the activation side, MXFP8 maintains numerical range while reducing communication pressure on intermediate tensors.

Repository details show:

• Repository size: approximately 1.56 TB;
• Parameter count from safetensors: approximately 2.78 trillion.

This means that merely downloading the weights may exceed the available space on a single conventional NVMe drive, let alone loading them into system memory. For most teams, pulling the complete repository directly is not the best starting point. A better approach is to first confirm whether the inference framework supports sharded loading, lazy loading, or object storage mounting.

If a team uses cloud object storage and GPU instances, it is advisable to prefetch weights to a high-speed cache layer to avoid pulling shards one by one from cold storage at startup. In this case, the hierarchy between network bandwidth, disk throughput, and GPU memory bandwidth becomes the primary bottleneck for startup time.

4. Deployment Options: From Inference Frameworks to Hardware Requirements

Several mainstream inference frameworks have already provided deployment recipes. For developers, this means at least they do not have to adapt custom operators from scratch, but they still need to understand configuration at three levels:

1. The combination of tensor parallelism and expert parallelism;
2. Tiered storage for key-value caches;
3. Preemption and batching strategies for long-context requests.

A typical startup configuration example is as follows:

model:
  dtype: mxfp4
  activation_dtype: mxfp8
  tensor_parallel: 8
  expert_parallel: 8
  max_context: 1048576
  enable_vlm: true
runtime:
  prefill_chunk: 8192
  max_batch_tokens: 131072
  cache_backend: hierarchical

Note that the above is only a structural illustration and does not refer to the actual configuration items of any specific product. In actual deployment, teams should adjust parameters based on inference framework documentation and their own compute specifications.

On the hardware side, community estimates suggest that one feasible configuration is a server with 8 GPUs and approximately 2.3 TB of total GPU memory, with a hardware cost of roughly $550,000 to $650,000. It must be emphasized that this is a third-party estimate, not an official recommendation. For most teams, directly purchasing such a machine is not realistic. More common paths include:

• Using on-demand cloud GPU instances for short-term evaluation;
• Building weight cache images to shorten startup time across multiple experiments;
• Splitting long-context tasks into two pipelines: “high-latency offline batch processing” and “low-latency online serving.”

From deployment practice, although the model’s activated parameters are relatively manageable, key-value cache growth in long-context scenarios is still rapid. If a million-token context uses full caching, GPU memory pressure will far exceed that of the weights themselves. Therefore, hierarchical caching and selective recomputation are almost essential measures.

5. Tool Calling and Inference Effort: State Return Cannot Be Ignored

If self-deployment is not desired, a hosted API can be used. The current API offers two speed tiers—standard and high-speed—with the latter officially stated to be 2 to 3 times faster than standard. Inference effort can be selected as low, high, or maximum, with maximum as the default.

In multi-turn tool calling, an easy pitfall is that the reasoning content generated in the previous turn must be passed back together with the tool call results. In other words, the model’s implicit reasoning state before and after tool execution cannot be discarded. Otherwise, the model may fail to correctly understand the current task progress or repeat already-completed operations in subsequent turns.

It is recommended that clients maintain the following structure:

{
  "messages": [
    {
      "role": "user",
      "content": "..."
    },
    {
      "role": "assistant",
      "content": null,
      "reasoning_content": "...",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "run_shell",
            "arguments": "{\"cmd\":\"ls -la\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "..."
    }
  ]
}

This state-return requirement imposes two constraints on server-side development: first, session storage must preserve the complete reasoning trace; second, streaming output must save not only visible text but also invisible reasoning fields. For enterprise-grade middleware, this means adding additional state management logic at the model relay layer.

In agent tasks, tool calls are often interleaved with shell execution, file reads and writes, and code modifications. If the reasoning state is truncated, the model may appear to respond normally in long-chain tasks while actually losing context.

6. Evaluation Results: Net Improvement in Real Sessions Is More Informative Than Leaderboards

On the same day the weights were released, an evaluation targeting agent scenarios also published its results. This evaluation does not rely on a fixed question bank but instead reads several signals from real user sessions:

• Whether the user explicitly confirms task success;
• Positive or negative user feedback on the model;
• Whether the model’s responses improve after user corrections;
• Whether shell commands execute successfully.

A “net improvement” metric is calculated on this basis. The latest version of the model achieved a net improvement of +5.1% across approximately 17,500 real long-horizon sessions, ranking 1st among open-weight models and 13th among all models. All three subcategories—work, code, and chat—ranked 1st among open-weight models.

By component, it reached +7.9% on shell reliability, whereas the previous generation scored -4.5% on the same metric. This change is more noteworthy than the overall score improvement. Shell reliability represents whether the model can reduce failure counts during multi-step command execution—a key factor in whether agent applications can move from demos to production.

The same model also shows differences in performance across inference effort levels. High scores often depend on a stronger reasoning budget, but this increases time-to-first-token and per-token cost. In practice, inference effort can be used as a tiered service switch:

• Simple conversation: low effort;
• Code completion and local modifications: high effort;
• Multi-turn tool calling and long-chain tasks: maximum effort.

This controls cost while retaining sufficient reasoning capability for critical tasks.

7. Reproducible Practical Recommendations

For teams planning to work with these weights, the following starting points are recommended:

First, start small before scaling up. Do not attempt million-token contexts from the beginning. Use 64K or 128K contexts to validate the basic inference pipeline, then gradually extend sequence length.

Second, the quantization pipeline must be complete. Since weights are stored in MXFP4 and activations run in MXFP8, the inference framework must have corresponding dequantization and matrix multiplication implementations. If the framework only supports INT4/INT8, forced conversion may introduce additional precision loss.

Third, persist tool-calling state separately. In multi-turn agent tasks, store reasoning content and tool call results separately for easier tracing and replay.

Fourth, do not evaluate based solely on leaderboards. Leaderboards reflect relative performance under a fixed distribution, whereas real business scenarios differ greatly in task distribution, fault-tolerance requirements, and cost constraints. It is advisable to build a representative set of sessions, run them repeatedly, and record task success rate, average turns, and failure modes.

Fifth, estimate total cost rather than per-GPU cost. GPU memory is only part of the cost. Weight downloads, cache warm-up, long-context request queuing, and failure retries all drive up total cost of ownership. Calculating cost “per successfully completed task” is more business-meaningful than “per-token price.”

8. Conclusion

The release of trillion-parameter sparse model weights gives teams a more flexible way to use the model than APIs, but it also returns responsibility for deployment optimization, state management, and cost control to the user. MXFP4/MXFP8 quantization reduces storage and transfer pressure but does not eliminate the engineering complexity introduced by long contexts and expert parallelism.

For developers, this is a combined scenario that tests model understanding, inference framework mastery, and system debugging capability simultaneously. Whether weights are open is only the starting point. Whether the model can run stably, accurately, and at controllable cost in one’s own environment is what truly determines business value. Practical work around inference caching, tool state management, and tiered serving may be more worthwhile than simply chasing leaderboard scores.