Open-Weight 2.8T Parameter Model: Architecture, Licensing, and Deployment Costs Explained

    This article analyzes a newly released open-weight 2.8T multimodal MoE model. It covers the 112-file, approximately 1.56TB weight package; the sparse MoE architecture with 104B activated parameters; MXFP4 weights and MXFP8 activations; vendor-reported benchmarks; the custom license; and the hardware, memory, networking, and operational requirements needed to deploy the model.

    1. What Actually Happened: The Weights Are Available, Not Just Announced

    On November 4, a model development team released the full weights of a trillion-parameter multimodal model as previously promised. The team first opened online API and web-chat access in late October, with a commitment to publish the weights by November 4. On the due date, the weights appeared on major model-hosting platforms. The repository is owned by the team and is publicly downloadable without application or approval.

    The repository was created on November 2. It remained closed on the morning of November 4 and opened that evening. The release timing itself suggests that the team had prepared the publication window in advance.

    From the file listing:

    • There are 112 files in total, including 96 sharded weight files, with a combined size of about 1.56TB.
    • The package is the largest publicly downloadable weight release so far, and the size gap compared with earlier open-weight models is substantial.

    It is important to distinguish “announcing open source” from “releasing the weights.” The former is a statement. The latter means that anyone can retrieve all parameters, verify them byte by byte, and deploy the model independently.

    This article focuses on three things that matter after the weights are available:

    1. What is inside the model?
    2. How should the license be understood?
    3. Can an ordinary team actually run it?

    2. How the 2.8T Parameters Are Constructed: A Reusable MoE Breakdown

    2.1 Total parameters and activated parameters

    According to the official model card, the model is a natively multimodal agent model continued from the previous-generation base model. It has 2.8T total parameters, but only 104B parameters are activated per token in a single forward pass.

    This means the model uses a Mixture-of-Experts, or MoE, architecture:

    • 896 routed experts
    • 16 routed experts selected per token by the router
    • 2 shared experts that always participate in computation

    These two numbers drive the rest of the analysis:

    • 2.8T is the storage cost. It directly affects GPU memory, download bandwidth, and loading time.
    • 104B is the compute cost. It determines the compute demand and latency of the decoding stage.

    The difference between the two numbers is roughly 27x. Therefore, the intuition that “parameter count equals compute demand” does not hold for this model.

    2.2 Other key specifications

    • 93 layers deep
    • 1 million-token context window
    • A self-developed vision encoder, making it natively multimodal rather than a text model with an external vision module attached
    • Compared with the previous generation, only one structural change is made: 24 layers of full attention are replaced with sparse attention, referred to as DSA. The purpose is to save compute for ultra-long contexts.

    Sparse attention saves compute because full attention has O(n²) complexity per attention head. In the sparse design, each query interacts only with a fixed number of historical positions, typically a sliding window plus attention sink tokens. The complexity drops to O(n·k).

    When n grows from 100,000 to 1 million and k remains in the thousands, the compute growth of the attention part changes from quadratic to linear.

    The tradeoff is also clear: skipped tokens are remote tokens. Strict long-range “needle-in-a-haystack” style tasks may benefit unevenly.

    A simplified text illustration:

    1. Full attention: q attends to all n tokens -> O(n^2)
    2. Sparse attention: q attends to window W + sink S -> O(n * (W + S))

    2.3 Weight precision: MXFP4 weights and MXFP8 activations

    The weights are stored in MXFP4, a 4-bit floating-point format, while activations use MXFP8, an 8-bit floating-point format. This is the result of quantization-aware training, or QAT. The model simulates quantization error during training or post-training according to the target precision, instead of being compressed only after training.

    The team publishes only this precision version. There is no BF16 version and no FP8 version.

    This is not merely a matter of convenience. A 1.56TB file size for 2.8T parameters corresponds to about 4.46 bits per parameter on average, including the overhead of a small number of higher-precision layers and routing tensors. In other words, the weights are truly stored at roughly 4 bits per parameter.

    If a BF16 version were produced, the file size would be roughly four times larger, exceeding 6TB. Therefore, the fact that the model can run on eight GPUs today is directly enabled by quantization. The model itself has not become smaller.

    The routing logic of the MoE can be expressed in simplified pseudocode:

    1. # Expert selection for each token (simplified)
    2. logits = router(x) # Shape: [896]
    3. top_idx, top_w = topk(logits, k=16) # Select 16 routed experts
    4. out = sum(top_w[i] * expert[top_idx[i]](x) for i in range(16))
    5. out = out + sum(shared_expert(x) for shared_expert in shared_experts) # 2 shared experts always active

    In a real implementation, this step becomes all-to-all communication plus grouped GEMM. Different experts reside on different devices, and because each token selects 16 experts, the activation path of each token is not fixed. This is the root reason why MoE inference is harder to operate than dense-model inference.

    3. How to Read the Scores: Using Vendor-Reported Benchmarks Correctly

    The official comparison table below is fully vendor-reported:

    Benchmark This model International flagship model A International flagship model B
    Coding-agent benchmark, Terminal-like 65.7% 66.4% 58.2%
    Long-horizon software engineering benchmark, SWE-like 51.7% 46.3% —
    Scientific reasoning benchmark, Terminal-Science-like 38.1% 63.3% —
    Hard academic QA benchmark, HLE-Diamond-like 39.4% 60.9% —

    The vendor says that the model surpasses flagship model A on 16 out of 37 benchmarks.

    The table tells a clear story:

    • Coding and agent-style tasks are close to or better than the flagship models.
    • Scientific reasoning and some high-difficulty reasoning tasks are clearly behind.

    It is not a case of winning every category.

    There are three practical rules for reading this kind of table:

    1. Distinguish self-reported results from third-party evaluations.
      The table is vendor-reported and may be affected by selective presentation, prompt differences, or sampling parameter differences. One third-party evaluation index gives the model a score of 55 and ranks it fourth globally. Independent leaderboards are closer to a comparable standard, though each still has its own biases.

    2. Look at task type rather than raw score alone.
      Coding-agent and long-horizon software-engineering tasks are essentially “multi-turn tool calling + long context + state maintenance.” This matches the architecture of an open-weight MoE model with a 1 million-token context very well. Scientific reasoning and some high-difficulty QA tasks depend more on parametric knowledge and rigorous chain reasoning, and are more likely to lose accuracy after 4-bit quantization. The architecture determines the strength distribution, not marketing.

    3. For any quantized model, benchmarks must be rerun on your own data.
      4-bit weights usually lose limited accuracy in short contexts, but the loss is amplified in long-chain reasoning, mathematics, and symbolic tasks. Running the model on your own business evaluation set is more reliable than reading ten comparison tables.

    4. License: Open Weights Are Not the Same as Open Source

    This is one of the most important points to examine separately.

    The repository carries a team-customized license, marked as license:other on the hosting platform. It is not a standard open-source license such as MIT or Apache-2.0. The license is word-for-word the same as the previous generation’s license, except for the name.

    The main permissions are MIT-like: free use, copying, modification, distribution, fine-tuning, and commercial use, without field-of-use restrictions. This is friendly for commercial adoption.

    However, there are still conditions outside the main text. These usually focus on three areas:

    • Large-scale users, such as those whose monthly active users exceed a certain threshold, may need to prominently attribute the model’s source in the product interface.
    • Use of its name or logo must comply with additional statements.
    • There may be usage restrictions, such as not using the model to train a competitor, and an “as-is” disclaimer with no warranties.

    For users, the practical judgment is:

    • Compliance level: a custom license is not equivalent to an OSI-defined open-source license. Legal and compliance teams must read the original text carefully, especially the monthly-active-user threshold and attribution requirements. Do not assume “open source means anything goes.”
    • Engineering level: the license does not change technical facts. The weights can be downloaded, modified, and used commercially. The risk lies in the uncertainty of clause interpretation, not in whether the model can be used.
    • Process level: it is advisable to archive the original license text, download source, and hashes together for later auditing and version tracking.

    5. Can You Run It Locally? Calculating the 1.56TB Model as GPU Memory

    The direct answer to the title question is: ordinary users cannot run it. Eight GPUs are the threshold.

    First, consider the weights themselves. Spreading 1.56TB across eight accelerators requires about 195GB per card. This cannot fit into a conventional 8x80GB node. Eighty GB times eight is only 640GB, which is not enough to hold 1.56TB.

    What can carry it are high-end accelerators with 192GB or even 288GB of memory per card. In other words, “8 GPUs” here means a single machine with eight high-memory accelerators, not eight arbitrary GPUs.

    Second, runtime overhead must also be counted. This part is often missed.

    • KV cache: the 1 million-token context is the key variable. KV cache is approximately:

      2 × number of layers × number of KV heads × head dimension × precision bytes × sequence length

      With 93 layers and a million-token context, even if grouped-query attention reduces the number of KV heads, the cache can still reach tens of GB.

      The 24 sparse-attention layers are intended to pull this part back from quadratic growth to linear growth. They save compute and also indirectly restrain the growth curve of the cache.

    • Activations and temporary buffers: MoE all-to-all communication buffers and peak memory caused by routing imbalance usually make actual memory usage 10%–20% higher than the theoretical value.

    • Framework overhead: the inference framework’s memory pool, CUDA graphs or compiled artifacts, and multi-GPU NCCL communication buffers can add up to several GB.

    The conclusion is that total machine memory should reserve more than 1.3x the weight size. Therefore, a 1.56TB model roughly requires a 2TB-level cluster memory footprint.

    Parallel strategies further clarify the tradeoff:

    Parallel strategy Memory demand Communication volume Typical scenario
    Tensor parallelism only, TP All experts resident, highest demand Two all-reduce operations per layer Small models
    Expert parallelism plus tensor parallelism, EP + TP Experts distributed, lowest per-card pressure All-to-all plus all-reduce Mainstream choice for large MoE models
    Expert parallelism plus redundant experts Slightly higher than pure EP Increased all-to-all Severe load imbalance or higher throughput

    The all-to-all communication volume of MoE grows with the number of activated experts and sequence length. Completing most communication inside a single node through high-speed interconnect is a hard requirement. Cross-node expansion can quickly become limited by network bandwidth.

    This is why the official threshold is “a single machine with eight GPUs,” not “multiple machines with several GPUs.”

    Executable deployment checklist

    1. Hardware: total machine memory should be at least 1.3x the weight size. Prefer a single-node, eight-GPU high-memory system and confirm inter-GPU interconnect bandwidth.
    2. Network: downloading and switching versions of a 1.56TB weight package requires a stable high-bandwidth channel. Internal mirrors or object storage as a distribution layer can significantly shorten cold-start time.
    3. Inference framework: confirm that the selected framework supports MXFP4 weights, MXFP8 activations, sparse attention, and MoE expert parallelism. A framework that does not support FP4 will fail outright rather than merely run slower.
    4. Precision validation: run short-, medium-, and long-context tests on your own evaluation set. Pay particular attention to accuracy drops in long-chain reasoning and numerical tasks.
    5. License compliance: archive the original license text and hashes, and confirm whether the monthly-active-user threshold and attribution obligations apply.
    6. Operational observability: monitor throughput, time to first token, expert load distribution, and GPU memory usage. Expert skew is a hidden cost in MoE systems, and if left unmanaged over time it can consume a large portion of throughput.

    6. If You Do Not Build Your Own Infrastructure

    An eight-GPU entry threshold means that most teams will not actually purchase a cluster. There are three practical options:

    • Use a cloud inference service. Pay per token, with zero operations, suitable for the validation stage.
    • Rent full-machine instances. Launch an eight-GPU high-memory node on demand, suitable for periodic stress testing and fine-tuning.
    • Combine self-hosting with the cloud. Use private deployment when core data must not leave your domain, and spill peak traffic to the cloud.

    Whichever option is chosen, logging, monitoring and alerting, object storage, and container orchestration should be treated as standard components. Logging and monitoring help identify latency jitter and expert skew, while object storage and orchestration support multi-version canary releases and fast rollback.

    Managed cloud services can provide similar general-purpose baseline capabilities. They can serve as a reference, but the final choice should still be based on your compliance boundaries and cost model.

    7. Conclusion: The Real Significance of This Generation of Open Weights

    Combining the three accounts:

    • Architecture: 2.8T total parameters plus 104B activated parameters, sparse MoE, and sparse attention form an engineering combination tailored for ultra-long contexts and agent scenarios.
    • Precision: MXFP4 weights and MXFP8 activations are the reason the model can fit onto eight GPUs, and they are also a direct source of accuracy loss in science and reasoning tasks.
    • License: MIT-style permissions plus custom additional terms mean the model can be used, modified, and deployed commercially, but this does not make it an unconditional standard open-source release.

    The practical advice for developers is:

    1. First validate the capability boundary with your own business evaluation set.
    2. Then calculate total machine memory and review the license terms.
    3. Finally discuss procurement.

    The real significance of trillion-parameter open weights is not that the model is the largest. It is that the complete recipe of this technical route has been handed to the community: the architecture, quantization strategy, and sparse-attention approach are all reusable public knowledge. That is the part worth spending time to study.