2.8T Open-Weight MoE Model: A Pre-Deployment Verification Checklist

    This pre-deployment checklist examines a 2.8T-parameter open-weight MoE model with roughly 104B active parameters per token. It covers architecture and sparse-attention implications, MXFP4/MXFP8 weight-size estimates, a custom-license review list, and the starting point for GPU-memory and deployment-path planning.

    Before choosing a large open-weight model, three questions come first: Is the compute cost acceptable? Do the license terms support the intended use? Can existing hardware or the cloud budget carry it? This article follows those three lines and provides estimation methods and a verification checklist.

    The data here comes from the original report and public model pages and has not been independently reproduced. TB is calculated in decimal terms (1 TB = 10¹² bytes). Before a formal decision, the official model card and configuration files should be treated as the authoritative source.

    Architecture: Large Total Parameters, Small Activation per Pass

    The model uses a mixture-of-experts (MoE) architecture. The original report states that it has 2.8T total parameters and activates about 104B parameters per token. It contains 896 routed experts, routes each token to 16 of them, and adds 2 shared experts for general knowledge. The model has 93 layers, supports a context length of up to 1 million tokens, and includes a vision encoder for image inputs.

    Total parameters and active parameters affect different parts of the system. Weight volume determines storage requirements. If all parameters remain resident in GPU memory, memory must be estimated from the storage footprint of 2.8T parameters. With offloading, sharding, or parallelism, per-GPU memory usage changes with the configuration, so memory estimates must be evaluated together with the deployment plan. Active parameters mainly affect per-step compute, so throughput and latency are closer to those of a 104B-class model. However, inter-GPU communication and data movement overhead can still slow actual performance.

    The original report also notes that full attention in 24 layers is replaced with sparse attention. Full-attention compute grows quadratically with sequence length and becomes expensive at million-token contexts. Sparse attention can reduce the compute cost of long sequences, but its applicable boundaries need validation. In tasks that require precise location of distant information—such as a specific symbol in a large codebase or a specific clause in a long contract—sparse patterns may miss key positions. If the business relies on precise retrieval, recall tests should be run on your own data, and the sparse-attention version should be compared with the full-attention version on accuracy, first-token latency, and peak memory.

    Quantization and Size: MXFP4 and MXFP8

    The original report says that only one version is currently public: weights use MXFP4 and activations use MXFP8, with no BF16 or FP8 version. This claim should be checked against the official model card and rechecked before publication.

    Weight size can be estimated with a formula: weight size ≈ parameter count × bits per parameter / 8 + metadata. For 2.8T parameters:

    • BF16 (16 bits per parameter): about 5.6 TB.
    • MXFP4: 4-bit elements with one 8-bit scale factor shared by every 32 elements, equivalent to about 4.25 bits per parameter, or about 1.49 TB.
    • 4-bit only: about 1.4 TB.

    The actual download size is 1.56 TB, about 0.07 TB more than the 4.25-bit estimate. This gap may come from modules kept at higher precision—such as embedding layers, the vision encoder, attention layers, or output layers—or from packing-format overhead. The exact composition should be confirmed from the official configuration files and tensor list; it should not be assumed that everything is MXFP4.

    Low-bit quantization can reduce size several-fold, but the main reason the model is large remains the 2.8T parameter scale. Quantization changes the storage cost per parameter; it does not change the total parameter count.

    Regarding training, releasing only low-bit weights suggests that low-precision representation may have been considered during training, but this cannot be concluded from the release format alone. Whether quantization-aware training (QAT) was used should be confirmed from official technical documentation.

    The impact on fine-tuning also needs evaluation. If high-precision fine-tuning is planned, low-bit weights introduce constraints. Confirm whether the training framework can perform forward and backward computation on MXFP4, and define how to measure accuracy loss after fine-tuning.

    Common questions:

    Will MXFP4 affect fine-tuning? Possibly. Gradient computation and optimizer-state management are more complex with low-bit weights. Validate convergence on a small dataset before committing to full fine-tuning.

    Which inference frameworks support MXFP4? Support depends on framework version, GPU model, and operator list, and combinations can differ significantly. Check the framework release notes and operator documentation, and confirm with tests on the target hardware.

    How should quantization accuracy be validated? Use the same business data to compare the quantized model with reference results, if available, on output agreement, key task metrics, and error types. General leaderboards alone cannot show business performance after quantization.

    License: Understand the Scope First

    The code hosting page lists the license type as “license:other.” This is a custom open-weight agreement, not a standard license such as MIT or Apache 2.0.

    The following is a technical verification checklist, not legal advice. Before commercial use, legal counsel should confirm each item and assess it against the specific business. Custom licenses may change between versions, so each update requires re-review.

    These agreements often add conditions on top of permissive terms. Before launch, legal and technical leads should check the following items one by one:

    1. Commercial restrictions: Is use in for-profit products allowed? Are there revenue or industry thresholds?
    2. Output ownership: Who owns generated content, and is source attribution required?
    3. Industry restrictions: Are specific fields such as healthcare, finance, or government prohibited?
    4. User-scale thresholds: Are there monthly or daily active-user limits, and is separate authorization required above them?
    5. Redistribution obligations: When distributing the model or derivative weights, is the agreement required, must notices be retained, or must derivative versions be published?
    6. Patent terms: Is there a patent grant, and does it terminate after a patent lawsuit?
    7. Data and geography: Are certain regions restricted, and are there data-transfer implications?

    Review conclusions should be documented in writing. Conclusions from an older version cannot be carried over directly to a new version.

    Deployment: Compute Threshold and Path Selection

    For memory estimation: if the full 1.56 TB weight footprint is loaded into GPU memory, the estimate must start from that amount. Because per-GPU memory usage changes with offloading, sharding, or parallelism, the memory estimate must be evaluated together with the deployment configuration. Path selection therefore depends on whether the model is fully resident or served with offloading, sharding, or parallelism; each configuration changes per-GPU memory requirements. Final planning should rely on the official model card, configuration files, and target hardware.