2.8T-Parameter Open-Weight MoE Model: Architecture, Licensing, and Deployment Requirements

    A newly released 2.8-trillion-parameter MoE model combines sparse attention, low-precision weights, and multimodal support in a roughly 1.56 TB download. This article examines its architecture, vendor-reported benchmarks, commercial licensing restrictions, and deployment requirements, including why eight GPUs cannot hold all its weights in GPU memory.

    1. Model Overview

    An AI company has released the weights of a 2.8-trillion-parameter mixture-of-experts (MoE) model, following an earlier commitment. At approximately 1.56 TB, the complete release is among the largest publicly available open-weight models.

    The technical details below are based on information published by the vendor. The reported figures have not yet been independently reproduced.

    2. Architecture Highlights

    The model uses an MoE architecture that activates approximately 104 billion parameters per token during inference. It contains 896 expert networks, with a routing mechanism that selects 16 experts for each token. Two additional shared experts process every token.

    The network has 93 layers, supports a context window of approximately one million tokens, and includes a vision encoder for image inputs.

    Compared with the previous generation, 24 full-attention layers have been replaced with sparse attention, referred to as DSA, reducing computation for long-context workloads.

    The released weights are the product of quantization-aware training and use MXFP4, while activations use MXFP8. The vendor does not provide BF16 or FP8 versions.

    For deployment teams, this makes native support for these low-precision formats essential. Older inference engines may fail to load the weights correctly or fall back to slower execution paths. Before choosing an engine, verify that the specific version supports MXFP4 and MXFP8 operations.

    3. Understanding the 1.56 TB Weight Size

    Storing 2.8 trillion parameters at an average of 4.25 bits per parameter would require approximately 1.49 TB. This estimate assumes about four bits per parameter for MXFP4, plus block-level scaling factors.

    The reported 1.56 TB is slightly larger. The difference may reflect unquantized or higher-precision components, such as the vision encoder and embedding layers.

    The exact breakdown should be checked against the official model card and weight manifest. These aggregate figures alone are not enough to establish how every component is stored.

    4. How to Interpret the Benchmark Results

    According to the vendor, the model leads on approximately 16 of the 37 benchmarks it reports. On coding and agent tasks, including Terminal-Bench and SWE Marathon, its scores exceed those of some leading competitors.

    However, its scores trail leading competitors substantially on scientific reasoning and difficult logical reasoning tests. Overall, the results are mixed: the reported strengths are concentrated in coding and agent workloads, while advanced scientific reasoning is comparatively weaker.

    A third-party intelligence index reportedly also places the model in the top tier. However, the source material does not identify the index, its version, or the evaluation date, so readers should verify that claim independently.

    For model selection, evaluations using your own business data should carry more weight than vendor leaderboard positions.

    5. Licensing: Open Weights Do Not Mean Open Source

    The model uses a custom license rather than a standard open-source license such as MIT or Apache 2.0. Its basic terms permit free use, copying, modification, distribution, fine-tuning, and commercial use, making those permissions broadly comparable to MIT.

    Two additional restrictions apply:

    1. Model-as-a-service revenue threshold: Companies operating a model-as-a-service offering whose cumulative revenue over the preceding 12 months exceeds $20 million must sign a separate agreement with the model provider before commercial use.
    2. Monthly active user threshold: Commercial products exceeding a specified monthly active user threshold also require additional authorization.

    Such provisions are commonly intended to limit direct API resale by large platforms, but their actual scope depends on the license text.

    Before launch, review the original license clause by clause and consult legal counsel. This article does not constitute legal advice.

    6. Deployment Requirements: Eight GPUs Cannot Hold All the Weights

    The vendor states that local operation starts with eight high-end GPUs. However, that configuration cannot hold the complete weights in GPU memory at the capacities considered here.

    Eight 80 GB GPUs provide 640 GB of aggregate memory, far below the approximately 1.56 TB required for the weights. Even eight 141 GB GPUs provide only about 1,128 GB.

    A rough lower-bound estimate is:

    Required GPU count ≥ weight size ÷ memory per GPU, with additional headroom for the KV cache and activations.

    Using the stated weight size:

    • 80 GB GPUs: The weights alone require approximately 20 GPUs: 1,560 ÷ 80 ≈ 19.5. Additional memory headroom increases the practical requirement.
    • 141 GB GPUs: The weights alone require approximately 12 GPUs: 1,560 ÷ 141 ≈ 11.1. Headroom is still necessary.

    An eight-GPU starting configuration therefore works only under specific conditions, such as offloading some weights to CPU memory or NVMe storage. This approach can substantially reduce inference throughput.

    The source material does not explain those conditions. Consult the vendor or inference framework documentation for the exact configuration, then stress-test it using realistic concurrency and context lengths.

    Storage and Download Considerations

    The source material states that the 1.56 TB weight release is split across more than 100 shard files. Use a download tool that supports resumable transfers, then compare file hashes with the official manifest.

    Deployment machines also need sufficient NVMe SSD capacity. Otherwise, storage I/O can become a bottleneck during model loading.

    7. Deploying the Model as a Cloud Service

    At this scale, most organizations are unlikely to deploy the entire model on a single physical server. A cloud-based serving architecture typically involves the following components.

    1. Weight storage and distribution: Store weights in object storage and distribute them to GPU clusters over high-speed private network connections, avoiding public internet bandwidth bottlenecks. Lifecycle policies can help reduce the long-term cost of storing large files.

    2. Elastic compute and container orchestration: Use a platform such as Kubernetes to manage multi-GPU nodes, with weights made available through container images or mounted distributed file systems. Adjust inference capacity to API demand to reduce idle compute.

    3. Inference engine compatibility: Confirm support for MXFP4 and MXFP8. Multi-node MoE inference generates substantial all-to-all communication, so network-topology-aware optimization is important. Otherwise, cross-node communication can limit overall throughput.

    4. Monitoring and logging: Integrate logging and alerting to track GPU utilization, GPU memory usage, inference latency, and expert routing distribution. These metrics help maintain service levels and meet SLA commitments.

    Before determining cluster size, measure time to first token and throughput on the target hardware at different concurrency levels.

    8. Conclusion

    This open-weight release offers useful engineering reference points, particularly in its use of sparse attention and low-precision quantization.

    Its practical constraints are equally important. Keeping all weights in GPU memory requires substantially more capacity than eight commonly available GPUs can provide. The custom license imposes commercial restrictions, and the benchmark evidence currently comes primarily from the vendor.

    Before committing to procurement or production deployment, review the original license, verify the supported hardware and inference stack, and run load tests on the actual target infrastructure.