Trillion-Parameter Open Weights: Architecture, Licensing, and Deployment Barriers

    A recent release publishes full weights for a 2.8T-parameter MoE model with 104B activated parameters per token, a 1M-token context, sparse attention, and MXFP4/MXFP8 QAT weights. This article reviews the architecture, performance caveats, custom licensing thresholds, and deployment math, including why the reported 8-GPU inference conclusion needs a closer check.

    1. Release Background

    Recently, a large-model vendor released the full weights of its flagship model according to a previously announced schedule. According to the original report, the model has 2.8 trillion total parameters (2.8T), and the sharded files total about 1.56 TB. Previously downloadable open-weight models were mostly in the hundreds-of-billions range. This release raises both download and deployment barriers by an order of magnitude.

    The figures cited in this article come from the original report. Before formal citation or procurement, refer to the official model card and license text.

    Three points stand out in this release. The weights are publicly downloadable without an application. The license is a custom agreement, and some companies above certain scale thresholds need separate negotiation. The original report states that even after quantization, an 8-GPU high-end inference cluster is still required. After the calculation in Section 6, this article believes that conclusion needs to be re-confirmed.

    2. Architecture: Decoupling Parameter Count from Compute

    The core structural parameters given in the original report are as follows:

    Dimension Value
    Total parameters 2.8T
    Activated parameters per token 104B
    Total experts 896
    Routed experts per token 16
    Shared experts 2
    Layers 93
    Context length 1M tokens

    According to the original report, the model is a native multimodal agent model and was trained from the previous-generation model.

    For a dense model, parameter count and compute are roughly equal. A 2.8T dense model would need to compute all parameters in every forward pass, which is not practical in engineering terms. The MoE architecture lets each token pass through only 16 routed experts and 2 shared experts, so the actual compute corresponds to about 104B parameters.

    This means 2.8T parameters do not imply that inference requires 2.8T-level compute. However, VRAM still has to hold all parameters, because routing cannot know which experts will be selected before computation.

    3. Sparse Attention and Low-Precision Training

    According to the original report, the main structural change from the previous generation is replacing some full-attention layers with sparse attention (referred to in the original text as DSA). Full-attention compute grows quadratically with context length. At a 1M-token scale, attention overhead would consume a large part of the inference budget. Sparse attention lets each token attend to only some key positions, reducing long-context cost to near-linear growth. The tradeoff is possible information loss, and the actual effect needs to be validated on your own long-document tasks.

    On quantization, the official release provides only one version: MXFP4 weights, MXFP8 activations, and the result of quantization-aware training (QAT). QAT introduces quantization noise during training, so the model learns to maintain performance at low precision. The accuracy loss is usually smaller than with post-training quantization (PTQ).

    Releasing only a 4-bit weight version is a practical limitation for teams that need full-precision continued training. Such teams start from already quantized weights.

    4. How to Read the Performance Data

    The vendor-reported comparison table shows that the model performs strongly on coding and agent-type tasks. On scientific knowledge and high-difficulty reasoning benchmarks, it has a clear gap with reference models. The vendor states that the model exceeds a reference model on 16 of 37 benchmarks. The original report does not provide a complete score table, so this article does not cite specific numbers. According to the report, a third-party intelligence index ranked the model fourth globally.

    When reading this kind of data, keep three points in mind:

    1. Self-reported results lack the guarantee of a unified evaluation environment; independent third-party retesting is more credible.
    2. Focus on the benchmark subsets relevant to your own business; average scores have limited reference value.
    3. Benchmark scores are not equal to task success rates in your own codebase. Ultimately, evaluate with your own task set.

    5. License: Check the Terms Clause by Clause

    The license is a custom agreement, labeled by the platform as license: other. It is not an OSI-certified open-source license such as MIT or Apache 2.0. The main terms are close to an MIT-style grant, allowing free use, copying, modification, distribution, fine-tuning, and commercial use. According to the original report, two restrictions are layered on top:

    1. Model-as-a-service (MaaS) threshold: companies whose main business is providing model services externally and whose cumulative revenue over the past 12 months exceeds USD 20 million need a separate agreement with the publisher before commercial use.
    2. Product monthly active user threshold: commercial products with more than about 100 million monthly active users enter additional negotiation.

    The impact on different parties is as follows. Individuals and small and medium-sized teams are basically unaffected by these two restrictions. Large cloud vendors and AI companies that provide inference services based on this model and meet the revenue threshold need to enter commercial negotiation. Before adoption, enterprises should ask legal counsel to confirm the calculation basis for revenue and monthly active users, and rely on the official license text.

    6. Local Deployment: The 8-GPU Conclusion Needs Recalculation

    Based on the figures in the original report, the total weight size is about 1.56 TB, or about 1560 GB.

    First, check whether the size is reasonable. MXFP4 uses 4 bits per parameter, with one 8-bit scaling factor shared by every 32 elements, averaging about 4.25 bits. At this rate, 2.8T parameters are estimated at about 1.49 TB. After adding unquantized parts, the total is close to 1.56 TB. This suggests that 1.56 TB corresponds to 4-bit quantized weights. If these were BF16 original weights, the size would be about 5.6 TB.

    Next, consider the number of GPUs. Based on total weight size, the results are as follows:

    Configuration Total VRAM Can it hold 1.56 TB of weights?
    8 × 141 GB GPUs 1128 GB No, short by about 432 GB
    12 × 141 GB GPUs 1692 GB Yes, with about 132 GB headroom
    20 × 80 GB GPUs 1600 GB Yes, with about 40 GB headroom

    To hold all weights on 8 GPUs, each GPU would need at least 195 GB of VRAM (1560 ÷ 8). The figures in the table do not yet include KV cache, activations, framework overhead, or sharding redundancy. Therefore, whether an 8-GPU configuration can run depends on whether it uses GPU types with larger VRAM or an undisclosed compression method. The official deployment documentation should be consulted.