Open-Weight Ultra-Large MoE Models: Architecture, Performance, Deployment Barriers, and License Compliance

    This article examines the recent release of a 2.8T-parameter open-weight multimodal agent model. It breaks down its MoE architecture—896 experts, 104B active parameters, 93 layers, 1M-token context, sparse attention, and MXFP4/MXFP8 quantization—then reviews benchmark strengths in coding and agent tasks and weaknesses in science and complex reasoning. It also analyzes the 1.56TB, 96-shard checkpoint and the eight-GPU minimum for local deployment, outlines a cloud deployment option, and explains the custom license's US$20 million MaaS revenue and 1 million MAU thresholds. The article closes with a staged adoption recommendation covering API validation, cloud-versus-self-hosted cost assessment, and legal review.

    The large-model open-weight ecosystem recently saw a major update. A Chinese large-model team released the full weights of its flagship multimodal agent model as scheduled. With 2.8T total parameters and a 1.56TB file size, it is the largest open-weight model released to date. The release not only gives developers a path to work with an ultra-large model but also raises new technical challenges for enterprise AI deployment. This article analyzes the practical path for ultra-large MoE models across architecture, performance, deployment barriers, and compliance.

    1. Architecture Deep Dive: How the 2.8T Parameters Are Structured and Optimized

    The model is positioned as a native multimodal agent model. It has 2.8T total parameters but activates only 104B parameters per inference pass, showing clear characteristics of a mixture-of-experts (MoE) architecture. Its core design is as follows:

    1. Expert network design: The model contains 896 expert networks. Each token dynamically selects 16 experts during inference, and 2 shared experts are included to ensure stable extraction of basic features.
    2. Network depth and multimodality: The model has 93 layers and supports an ultra-long context of up to 1 million tokens. It also integrates a dedicated visual encoder module with image understanding and processing capabilities, enabling true native multimodal interaction.
    3. Attention mechanism upgrade: Compared with the previous generation, one key structural improvement is the replacement of 24 full-attention layers with sparse attention (DSA). This optimization substantially reduces compute consumption and memory footprint in long-context scenarios, making million-token inference more feasible in engineering terms.
    4. Quantization-aware training: The weight files use MXFP4 precision, while activations use MXFP8 precision. This is the result of quantization-aware training, which compresses model size substantially while preserving model accuracy as much as possible. Notably, the official release provides only this quantized version and does not offer higher-precision original versions such as BF16 or FP8, which imposes specific precision-tolerance requirements on deployment environments.

    2. Performance Evaluation: Strengths and Gaps in Coding and Agent Capabilities

    At release, the vendor provided a self-reported benchmark comparison table. A combined analysis of the evaluation data shows a clearly uneven performance profile:

    1. Strengths: In coding tests such as Terminal-Bench and the agent marathon evaluation SWE Marathon, the model performs well, with scores even surpassing some leading proprietary models in the industry. This indicates strong engineering deployment potential in code generation, multi-step task planning, and execution scenarios in the agent category.
    2. Gaps: In science tests and some complex logical reasoning benchmarks such as HLE-Diamond, the model performs relatively weakly and shows a certain gap versus top proprietary models. Overall, of the 37 benchmarks published by the vendor, about 16 surpass the comparison models, presenting a mixed win-loss record.
    3. Third-party evaluation: Data from a third-party evaluation organization shows that the model’s composite intelligence index ranks among the top globally, confirming its technical strength as a top open-weight model.

    3. Deployment Barriers and Local Practice: The 1.56TB Checkpoint and Eight-GPU Minimum

    Although the model weights are publicly downloadable and developers can also obtain them through mainstream model hosting communities, their enormous size imposes strict requirements on local deployment.

    1. Storage and network bandwidth: The 1.56TB model file is split into 96 shards. The download process is a significant test of network bandwidth and storage-system IOPS. Enterprises need high-speed distributed storage systems to support concurrent reads of large files.
    2. Hardware configuration requirements: Because 104B parameters are activated per pass, combined with the residency requirements of the huge total parameter count, running the model locally requires at least 8 high-end GPUs (for example, 80GB memory versions) as a starting point. For most small and medium-sized enterprises, the hardware procurement cost is extremely high.
    3. Cloud deployment option: Facing high self-built compute costs, enterprises can use cloud providers’ compute clusters and container platforms for elastic deployment. For example, using GPU compute instances and model hosting services from Baidu AI Cloud, enterprises can offload underlying infrastructure operations and use object storage to quickly distribute and mount model weights, thereby accelerating business validation.
    1. # Pseudo-code example: loading logic for an ultra-large model with a distributed inference framework
    2. from inference_engine import DistributedEngine
    3. # Configure multi-GPU tensor parallelism and pipeline parallelism
    4. engine = DistributedEngine(
    5. tensor_parallel_size=8, # 8-GPU tensor parallelism
    6. pipeline_parallel_size=1,
    7. dtype="mxfp4",
    8. trust_remote_code=True
    9. )
    10. # Load the ultra-large MoE model
    11. model = engine.load_model(
    12. model_path="/mnt/distributed_storage/massive_moe_model",
    13. device_map="auto"
    14. )

    4. License Compliance: Open Weights Are Not Fully Open Source

    The most notable aspect of this release is its license. The model does not use a standard open-source license such as MIT or Apache; instead, it carries a custom license and is marked as “other” on platforms such as Hugging Face. This means it is essentially “open weights” rather than open source in the traditional sense.

    The main body of the agreement does contain MIT-like licensing terms, allowing free use, copying, modification, distribution, fine-tuning, and commercial use. However, for large-scale external-facing business, the agreement adds two key restrictions:

    1. Revenue threshold: Companies that provide model-as-a-service (MaaS) externally and have cumulative revenue exceeding US$20 million over 12 months must sign a separate agreement with the model publisher before commercial use.
    2. User-scale threshold: Commercial products with monthly active users (MAU) above a specific number (for example, 1 million) also require additional authorization.

    This kind of restricted commercial license has already become a trend in the large-model industry. For startups and small-to-mid-scale applications, the impact is minimal; however, cloud providers or large enterprises attempting to build large-scale commercial MaaS platforms on top of the model must conduct strict compliance review and business negotiations.

    5. Conclusion and Recommendations

    The release of open-weight ultra-large MoE models gives developers and enterprises an opportunity to approach the capabilities of top proprietary models, especially in agent and code-generation scenarios. However, 2.8T parameters, 1.56TB storage requirements, and an eight-GPU minimum mean that deployment is not trivial.

    For enterprises planning to adopt the model, a staged strategy is recommended: first, validate business scenarios through APIs; second, when data privacy and localization requirements genuinely exist, evaluate the rental cost of cloud GPU compute clusters against the return on investment of self-built compute; finally, before commercialization, the legal team must carefully review the custom license to avoid potential infringement and breach-of-contract risks. Only by balancing architecture, performance, cost, and compliance can ultra-large open-weight models truly become a core competitive advantage for enterprises.