Open Weights Are Not Enough: Calculating Deployment Costs for a Trillion-Parameter Model

    Open-weight releases make model parameters downloadable, but production deployment still depends on license terms, hardware capacity, and engineering implementation. This article explains how to recalculate the deployment budget for a trillion-parameter MoE model, covering total versus active parameters, sparse attention and long-context costs, and MXFP4 weight size. It highlights why memory planning must use total parameters, why long-context benefits require empirical testing, and how quantized weights affect download size and further development.

    I. Download Is Only the First Step

    Open weights mean that parameters can be downloaded, but that does not mean the model can go straight into production. From download to deployment and then to commercial use, a team must clear at least three thresholds: license, hardware, and engineering implementation. The license determines whether the model can be placed in a commercial product, hardware determines whether it can run, and engineering capability determines whether it can serve users reliably. These three thresholds are independent. If any one of them cannot be cleared, the earlier download has no practical value.

    This article follows these three lines and provides methods that readers can use to recalculate the numbers themselves. The parameters, benchmarks, and license figures in this article mainly come from the publisher’s public descriptions. Before a formal decision, verify each item against the official model card and the original license.

    II. MoE Architecture: Large Total Parameters, Fewer Activated per Token

    The model uses a mixture-of-experts (MoE) architecture. The main configuration in the release materials is as follows:

    • Total parameters are about 2.8T, with about 104B parameters activated per token;
    • There are 896 experts in total. Each token is routed to 16 experts, and 2 additional shared experts participate in computation for all tokens;
    • The network has 93 layers and supports a context length of 1M tokens;
    • A built-in vision encoder can process image inputs. The publisher positions the model as one designed for agent scenarios.

    MoE trades sparse activation for model capacity. Knowledge capacity is determined by the parameters of all experts, while the cost of each forward pass is borne only by the activated experts. Therefore, compute per inference is close to that of a dense model in the hundred-billion-parameter class with about 104B active parameters. This is only an estimate on the compute side. Memory, however, must still be sized against the 2.8T total parameters, because routing may call any expert and the weights of all experts need to be accessible at any time. This point is easy to overlook, and the hardware estimates later will rely on it repeatedly.

    III. Sparse Attention and Long-Context Cost

    The release materials state that, compared with the previous generation, the model changes only one structural element: full attention in 24 layers is replaced with sparse attention.

    The compute cost of full attention grows with the square of sequence length. As context increases from a few thousand tokens to the million-token level, the compute cost of the attention part and KV cache usage rise sharply. Sparse attention lets each token interact with only some key positions, thereby reducing compute and memory pressure in long-text scenarios.

    This approach is already relatively common in ultra-long context processing, but the actual benefit depends on the specific implementation and task distribution. It is advisable to test with your own long-text samples: fix the input samples, compare against a full-attention baseline, and record first-token latency and throughput at the same time. Looking only at the principle cannot determine whether it is worthwhile for your business.

    IV. MXFP4 Weights: Smaller Size, Limited Further Development

    The official weights use a format obtained through quantization-aware training: weights are MXFP4 and activations are MXFP8. As of the writing of this article, the weight page lists this one version and does not provide a BF16 or FP8 version. Readers should rely on the latest information on the release page.

    This format has two-sided implications.

    First, size drops noticeably. MXFP4 uses about 4 bits per parameter, or 0.5 bytes. Multiplying 2.8T parameters by 0.5 bytes gives about 1.4TB, which is close to the download size of 1.56TB. The difference of about 0.16TB between the two may come from scaling factors, unquantized layers, the vision encoder, and other components. This is an estimate based on the format; the exact composition and the practical constraints on further development should be verified against the official model card and the latest weight page.