Inside a 2.8T Open-Weight MoE Model: Architecture, Licensing, and Local Deployment Barriers

    A China-based LLM vendor has released the full weights of a 2.8T-parameter MoE model with roughly 104B active parameters. This article breaks down its sparse architecture, sparse attention change, quantization-aware training, custom license terms, and the real local deployment requirements—including the roughly 1.5TB VRAM footprint that puts full self-hosting in eight-GPU territory—and outlines practical alternatives for most teams.

    Event recap: a large open-weight release delivered on schedule

    A China-based LLM vendor has released the full weights of its flagship model at the time it previously promised. The model has about 2.8 trillion total parameters (2.8T), with sharded downloads totaling about 1.56TB. The vendor calls it the largest open-weight model by parameter count currently available for download. The model had already been offered through an API and a web interface. The hosting repository was created two days before the official release, and on release day the weights could be downloaded without an application. Mirrored copies were also available through community model hubs.

    The significance of this event is not that “another new model was released,” but that it puts three long-standing questions in front of the community:

    1. What exactly makes up an ultra-large MoE model? The numerical relationships among parameter count, expert count, and active parameters deserve an item-by-item breakdown.
    2. What is missing between “open weights” and “open source”? The clauses in a custom license directly affect enterprise adoption decisions.
    3. How high is the real barrier to local deployment? A 1.56TB download is only the beginning; VRAM and interconnect bandwidth are the hard constraints.

    Below, we examine each.

    Model structure: how 2.8T parameters are put to use

    2.1 MoE architecture: sparse activation is key

    According to the official model card, the model is a natively multimodal agent model, further trained from its predecessor, and uses a mixture-of-experts (MoE) architecture:

    • Total parameters: 2.8T, but only about 104B (104 billion) parameters are activated per token.
    • Expert scale: 896 experts; the router selects 16 experts per token, plus two shared experts used by all tokens.
    • Depth and context: 93 layers in total; context window up to 1 million tokens.
    • Multimodal capability: built-in vision encoder with image understanding.

    The core idea of this design is to trade sparse activation for decoupling capacity and cost. The model’s “knowledge storage capacity” is determined by total parameters, while per-inference compute and memory access are determined by active parameters. The 2.8T/104B ratio means that less than 4% of parameters participate in each forward pass, allowing inference cost to approach that of a hundred-billion-parameter dense model while retaining far greater knowledge capacity.

    2.2 The only structural change from the predecessor: sparse attention

    Compared with its predecessor, the model makes only one structural change: replacing full attention in 24 layers with sparse attention (DSA). The motivation is direct: at a 1 million token context length, full attention’s computational complexity grows quadratically with sequence length, and compute cost in long-context scenarios becomes a bottleneck. Sparse attention lets each token interact with only some key positions, greatly reducing attention computation over long sequences. This is one of the common industry technical routes for ultra-long context today.

    2.3 Quantization-aware training: MXFP4 weights, MXFP8 activations

    Notably, the official release includes only a single version with MXFP4 weights and MXFP8 activations. This is the product of quantization-aware training (QAT), not post-training quantization. This means:

    • The model simulates low-precision numerical behavior during training, so quantization accuracy loss is “absorbed” during training.
    • MXFP4 is a 4-bit floating-point format with micro-scaling factors, which covers the dynamic range of weights better than traditional integer quantization.
    • There is no BF16 or FP8 version, so the community cannot “dequantize” back to a high-precision version on its own, nor can it run freer quantization experiments based on high-precision weights. This is a limitation that must be considered when evaluating the usability of these weights.

    From a deployment perspective, MXFP4 weights directly determine the magnitude of VRAM requirements. This is the root of the eight-GPU threshold discussed in the next section.

    2.4 Benchmark performance: mixed results, read with care

    The vendor-reported benchmark results show a clear structural pattern: performance is strong on coding and agent tasks, with multiple results meeting or exceeding leading closed models; but on science and some high-difficulty reasoning benchmarks, there is still a clear gap versus the strongest closed models. A third-party evaluator’s composite intelligence index ranks it fourth globally.

    The takeaway for users is: no ultra-large model dominates every dimension, and model selection must align with the task distribution of your own business scenario. If your workload is mainly code generation, tool calling, and multi-step agent orchestration, this type of model deserves close evaluation; if your core workload is frontier scientific reasoning, more careful empirical validation is needed.

    License interpretation: open weights are not open source

    This is the point in this release that enterprise legal and technical teams should review separately.

    The weights use a custom license, marked as an other license on major hosting platforms rather than a standard open-source license such as MIT or Apache 2.0. Its main terms are indeed very permissive, close to MIT style: free use, copying, modification, distribution, fine-tuning, and commercial use are allowed.

    But on top of the permissive main terms, two targeted restrictions apply, and only to large-scale external business:

    1. Model-as-a-service (MaaS) threshold: If the model is offered as a service and the company’s cumulative revenue over the past 12 months exceeds $20 million, a separate agreement with the model vendor is required before commercial use.
    2. Product scale threshold: After a commercial product’s monthly active users exceed a certain scale, additional agreement requirements are also triggered.

    Practical impact on enterprise model selection

    • Individual developers and small-to-medium businesses: When revenue and user scale are far below the thresholds, the license can be treated as free use, including commercial use.
    • Large companies or teams with clear scaling expectations: They must include the “negotiation cost and dependency risk when future clauses are triggered” in technical selection from the start. Once core business is built on the model, bargaining room for a supplemental agreement may be very limited.
    • Compliance action: Legal teams should review the original custom license clause by clause rather than rely on the intuition that “open weights ≈ open source.” This is already common in the industry: most leading models’ open licenses carry similar scale-triggered clauses.

    Local deployment: 1.56TB is just the entry ticket

    4.1 VRAM estimate

    The weight files total about 1.56TB, most of which is the model parameters themselves. A rough estimate for 2.8T parameters stored in MXFP4:

    1. Parameter memory ≈ 2.8T × 0.5 bytes (4 bit) ≈ 1.4 TB
    2. Additional overhead:
    3. - Activations (MXFP8) and KV cache: grow linearly with context length;
    4. at million-token context, a single request may occupy tens of GB
    5. - Runtime, framework buffers, expert routing overhead: tens of GB
    6. Total: about 1.5TB or more of VRAM

    Current single-GPU memory caps are generally in the sub-100GB range. Even at the largest single-GPU memory, at least eight top-tier accelerators connected by high-speed interconnect in a single node are needed to hold the full weights. This is why “local operation starts at eight GPUs”—and even that is only “it fits.” To achieve acceptable inference throughput, KV cache and concurrency headroom must also be reserved.

    4.2 Engineering details of download and storage

    The 96 sharded weight files, totaling 1.56TB, also impose practical requirements for download and storage:

    • Sharding and resumable downloads: Large-file sharding is itself designed for resumable downloads; download tools should support verification and retry.
    • Mirrors: Community model hubs often provide synchronized copies; direct cross-region downloads can have bandwidth costs that stretch the download time to days.
    • Storage media: During loading, high-speed NVMe arrays are recommended; mechanical disk sequential read speed becomes a cold-start bottleneck.
    1. # Illustrative flow (pseudo-commands): pull sharded weights from a hosting platform and verify
    2. model-tool download <org>/<model> --local-dir ./weights
    3. checksum verify ./weights --manifest manifest.json

    4.3 Practical alternatives for most people

    For the vast majority of teams and individuals without an eight-GPU cluster, there are three realistic paths:

    1. Use the API: The vendor’s API service bears the inference cost and bills by token, suitable for validation and small-to-medium workloads.
    2. Wait for or find community quantized derivatives: The community often applies more aggressive quantization on top of the original weights, reducing VRAM requirements to consumer-grade or single-GPU workstation level—but further compression based on MXFP4 original weights requires independent accuracy evaluation.
    3. Choose smaller open models: Versions one to two orders of magnitude smaller in the same series or similar architecture can often be fully deployed in a single-machine multi-GPU environment, with far better cost-performance than forcing an ultra-large model.

    Conclusion

    This release of ultra-large weights can be summarized in three judgments:

    • Technically, the 2.8T total / 104B active MoE design, sparse attention, and quantization-aware training represent the mainstream engineering route for capacity-cost tradeoffs in ultra-large models today. The numerical composition is worth understanding for anyone doing model selection.
    • On compliance, “open weights” are restricted openness under a license. Permissive main terms coexist with scale-triggered commercial restrictions, and enterprises must complete legal assessment early in model selection.
    • On deployment, the 1.56TB download is only the first step. A VRAM requirement of about 1.5TB means full local deployment is the domain of eight-GPU-class clusters. API access, community quantized versions, and smaller models are the realistic options for most teams.

    The openness of ultra-large models is evolving from “can it be downloaded” to “can it be used affordably and compliantly.” A comprehensive assessment across these three dimensions is the professional stance to take for any large weight release.