Analysis of 2.8T Parameter Open Weights: Licensing, Architecture, and Local Deployment of Ultra-Large MoE Models

    A leading AI team recently released a 2.8T parameter open-weight MoE model, totaling approximately 1.56TB. This article analyzes the model's architecture, custom licensing terms, benchmark performance, and the practical hardware requirements for local deployment, providing developers and enterprises with a comprehensive guide to evaluating this massive model.

    I. Event Review: A Long-Awaited Full Release

    A leading AI team previously launched its flagship model via API and web interfaces, publicly committing to releasing the complete weights by a specified date. They fulfilled this promise: the repository was created in early November and officially opened to the public days later without any application or review process. Anyone can download it directly. The repository contains 112 files and about 96 shards, totaling approximately 1.56TB.

    It is important to emphasize that this is not a standard version iteration. With a 1.56TB download size and 2.8T total parameters, it ranks among the largest publicly downloadable open-weight models, an order of magnitude larger than previously released models. This article skips benchmark rankings to focus on three more practical questions following the weight release: what the internal structure looks like, how the licensing terms constrain usage, and whether average developers and enterprises have the capability to run it locally.

    II. Model Architecture: How 2.8T Parameters Are Composed

    Based on the official model card, this is a native multimodal Agent model, continuously trained from a previous base model. The core architectural parameters are as follows:

    • Total parameters: 2.8T, utilizing a Mixture of Experts (MoE) architecture. Each token activates only about 104B parameters, keeping actual inference compute far below the total parameter count.
    • Expert networks: 896 total, with each token routed to 16 experts. Additionally, 2 shared experts participate in all token computations.
    • Layers: 93. Context length: 1 million tokens.
    • Vision: Equipped with a visual encoder for image input, making it a native multimodal design capable of understanding images.

    Compared to its predecessor, one structural change stands out: 24 layers of full attention were replaced with Sparse Attention (DSA) to reduce memory and compute overhead in ultra-long context scenarios. Sparsification is a standard engineering approach to handle million-level contexts, where attention computation complexity grows exponentially with sequence length.

    Another notable detail: the weights are stored in MXFP4 format and activations are processed in MXFP8, resulting from Quantization-Aware Training (QAT) rather than simple post-training quantization. This means low-precision performance loss was compensated during the training phase. However, the vendor only provides this version—no BF16 or FP8 weights exist. For users, this means you cannot convert this into a higher-precision “original” version; you must accept this fixed format.

    III. Benchmarks: Evaluating Vendor-Reported Data

    Official comparisons show the model performs exceptionally well on coding benchmarks. For instance, it scored 65.7% on a terminal coding benchmark (versus 58.2% for a competitor’s flagship) and 51.7% on a long-task software engineering benchmark, also beating the baseline. However, it lags significantly in science and high-difficulty reasoning, with gaps exceeding 20 percentage points. The vendor claims it leads in 16 out of 37 benchmarks. Objectively, results are mixed: strong in coding and Agent tasks, weaker in science and certain reasoning. A third-party intelligence index gave it 55 points, ranking 4th globally.

    When interpreting this data, readers should note three points:

    1. All data is vendor-reported, allowing freedom in selecting benchmarks and evaluation configurations.
    2. The MoE characteristic of “large total parameters, small active parameters” means advantages typically lie in knowledge capacity and long context handling, not single-pass inference density.
    3. Any benchmark should be tested on a small scale within your own business scenario rather than blindly trusting the rankings.

    IV. Licensing: Open Weights Are Not Open Source

    This is the most critical aspect of the release that warrants separate discussion. The repository uses a custom license marked as “other” on hosting platforms, rather than standard open-source licenses like MIT or Apache. The main terms are indeed MIT-like: permitting free use, copying, modification, distribution, fine-tuning, and commercial use. However, two specific restrictions are layered on top, affecting large-scale external services:

    1. If the model is provided as a service and the company’s cumulative revenue over 12 months exceeds $20 million, a separate agreement must be signed with the publisher before commercial use.
    2. If a commercial product exceeds 100 million monthly active users (MAU), additional authorization is required.

    This model is known as “open weights” rather than “open source”: weights can be downloaded, researched, and used for small-to-medium commercial purposes, but ultra-large-scale commercialization reserves room for negotiation. For enterprise users, the compliance key is to assess in advance whether revenue and user volume hit these thresholds and to design architectures that allow swapping the underlying model to avoid single-license lock-in.

    V. Practical Thresholds for Local Deployment

    While the 1.56TB weights are a disk challenge, VRAM is the real bottleneck for most users. Even though MoE only activates 104B parameters, all expert weights must reside in VRAM (or rely on costly CPU offloading). In MXFP4 format, it still requires a VRAM pool of around 1.5TB. This means:

    • Individual developers: Consumer-grade hardware cannot support this; they must use cloud APIs or rent multi-GPU instances.
    • Enterprise self-hosting: 8 high-end accelerators (with 100GB+ VRAM per card) are the minimum requirement, along with high-speed interconnects, large-capacity NVMe storage, and parallel loading scripts.
    • Deployment practices: Use object storage as the foundation for weight distribution, container platforms for inference orchestration, and monitoring systems for VRAM utilization and throughput. Prioritize verifying that your quantized inference framework supports the MXFP4 format.

    A simplified resource check logic before deployment:

    1. # Pseudo-code: Resource validation before deployment
    2. total_weight_tb = 1.56
    3. per_gpu_mem_gb = 192
    4. gpu_count = 8
    5. assert total_weight_tb * 1024 < per_gpu_mem_gb * gpu_count * 0.95, "Insufficient VRAM pool"
    6. # Remaining VRAM must be reserved for KV Cache; for million-context scenarios, the reservation ratio should typically be no less than 20%

    VI. Conclusion

    This release sends two signals: First, ultra-large-scale MoE open weights are becoming a standard move for leading AI teams, expanding the boundaries of open model capabilities. Second, the semantics of “open source” are being redefined by “open weights + custom licenses.” For developers, the rational approach is to treat it as a powerful resource for research and small-to-medium commercial use, while soberly evaluating three practical gates: licensing terms, quantization format lock-in, and the 8-GPU hardware threshold.