Inside a 2.8T Open-Weights MoE Model: Architecture, Deployment, Licensing, and Evaluation Explained A newly released 2.8-trillion-parameter Mixture-of-Experts model with open weights is about 1.56TB to download and requires a minimum of 8 GPUs for local deployment. This article breaks down its architecture, why only 104B parameters are activated per token, quantization and distribution trade-offs, deployment requirements, benchmark caveats, and licensing terms that enterprises should review before adoption.

    Open weights are not open source. A recently released 2.8T-parameter MoE model makes this distinction impossible to ignore: 1.56TB of MXFP4 weights, a 1-million-token context window, an 8-GPU minimum for local deployment, and a custom license with revenue-based commercial conditions. This article analyzes the model across five dimensions—architecture, distribution, deployment, evaluation, and licensing—to help engineering teams evaluate whether it fits their use cases.

    A super-large Mixture-of-Experts (MoE) model with 2.8 trillion total parameters recently published its full weights. The download is roughly 1.56TB, and the official guidance sets an engineering floor of “8 GPUs minimum” for local deployment. The technical significance of this release goes beyond the record parameter count: it puts several issues on the table at once—the boundary between “open weights” and “open source,” the deployment cost of super-large MoE models, the engineering trade-offs of low-precision quantization, and enterprise compliance. This article analyzes the release across five dimensions: architecture, distribution, deployment, evaluation, and licensing.

    1. First, Clarify the Concept: Open Weights ≠ Open Source

    What was published is downloadable inference weights under a custom license, not a standard permissive open-source license. The core grant is close to a permissive model—you can freely use, copy, modify, distribute, fine-tune, and commercialize—but with two restrictions targeting large-scale external services:

    1. If you offer the model as a service (Model-as-a-Service) and your cumulative revenue over 12 months exceeds $20 million, you must sign a separate agreement with the rights holder before commercial use.
    2. If your commercial product exceeds 100 million monthly active users or $20 million in monthly revenue, you must display prominent attribution in your interface.

    Engineering teams therefore should not simply file this under “open-source models.” A more accurate description is “open weights with conditional commercial use.” Enterprises should complete at least three compliance checks:

    • Whether the use case counts as an external model service;
    • Whether revenue and user scale trigger the additional clauses;
    • Whether product UIs and external materials need attribution notices.

    Open weights lower the barrier to acquisition, but they do not eliminate license obligations.

    2. Architecture Breakdown: Why 2.8T Total Parameters Only Activate 104B

    The model uses an MoE architecture: 2.8 trillion total parameters, with only about 104 billion activated per token. It includes 896 experts, selecting 16 per token plus 2 shared experts, across 93 layers. The context length reaches 1 million tokens, and a built-in vision encoder gives it native multimodal and agentic capabilities.

    The key point to understand here: activated parameters are not the same as memory footprint. MoE’s routing mechanism sends each token through only a small subset of experts, reducing per-token compute—but all expert weights still need to be stored, or swapped in on demand via layering and sharding strategies. With full residency, the 1.56TB of weights is the VRAM floor; with expert swapping, per-GPU memory pressure drops, but you introduce extra I/O and latency jitter. So “104B activated” describes computational sparsity, not deployment footprint.

    Compared with its predecessor, the model replaces full attention with sparse attention in some layers, reducing compute and memory costs in long-context scenarios. Even so, the KV cache pressure from a 1-million-token context remains significant. Long-context serving cannot be planned around weights alone—you must also reserve room for caches, activations, communication buffers, and framework runtime overhead.

    3. Quantization and Distribution: The Engineering Reality Behind 1.56TB of Weights

    The published weights are a quantization-aware training artifact using MXFP4 weights and MXFP8 activations. Only one low-precision version is provided—no higher-precision weights are available. This reduces download and storage size, but it means deployment teams must verify how well their inference framework supports these numeric formats. If it doesn’t, format conversion may be required, and conversion itself introduces uncertainty in both accuracy and engineering cost.

    The weights are organized as 112 files across 96 shards, totaling about 1.56TB. Sharded distribution suits object storage, mirror repositories, or parallel downloads, but it must be paired with verification mechanisms: shard integrity checks, version consistency checks, resumable downloads, and failure retry. For on-premises deployment, the recommended approach is to land the weights in unified storage first, then mount or pull them via your container platform—avoiding repeated cross-network downloads on every node.

    A rough lower-bound estimate for VRAM:

    1. weight_tb = 1.56
    2. cards = 8
    3. weight_per_card_tb = weight_tb / cards
    4. print(weight_per_card_tb * 1024, "GiB per card, weights only")

    At 8 GPUs, weights alone average nearly 195GiB per card—before KV cache, activations, communication buffers, and runtime overhead. So “8 GPUs minimum” should be read as the lowest viable unit, not a comfortable configuration. Production deployments will need to scale up based on concurrency, context length, and the inference framework’s parallelism strategy.

    4. Deployment Threshold: 8 GPUs Is Just the Starting Point

    Local inference of a super-large MoE typically combines tensor parallelism, expert parallelism, pipeline parallelism, and layered loading. Expert parallelism introduces all-to-all communication that is sensitive to interconnect bandwidth; long contexts amplify KV cache usage; multimodal inputs add vision encoding and preprocessing overhead. In practice, teams should verify:

    1. Whether per-GPU VRAM covers the weight shard plus runtime overhead;
    2. Whether inter-GPU interconnect can support expert routing communication;
    3. Whether the inference framework supports low-precision weights and dynamic expert routing;
    4. Cache eviction and rate-limiting policies under long-context concurrency;
    5. Whether monitoring and alerting cover VRAM, bandwidth, queueing latency, and error rates.

    For cloud deployments, general-purpose capabilities such as object storage, container platforms, and monitoring/alerting can handle shard distribution, inference service orchestration, and resource observability. Specific instance sizes still need to be determined through load testing against your actual accelerators, inference framework, and business concurrency. For most teams, running small-scale offline validation first and then evaluating online serving is the safer path.

    5. Reading the Benchmarks: Take Self-Reported Scores with a Grain of Salt

    The published comparison table shows the model outperforming a certain closed-source flagship on 16 of 37 benchmarks, with particular strength in coding and agentic tasks—but clear weaknesses in scientific reasoning and some high-difficulty reasoning tasks. It’s worth emphasizing that most of these figures are vendor-reported; test environment, prompts, tool-calling strategies, and scoring criteria can all affect results. Third-party comprehensive evaluations placing it among the global leaders still cannot replace validation on your own task sets.

    When selecting a model, keep “use case first”: if your business centers on code generation, tool calling, or multi-step agent orchestration, focus your testing on task completion rates and tool stability; if it involves scientific computing, rigorous reasoning, or high-risk decisions, set stricter offline evaluation and human review. Vendor benchmarks are a reference; reproduced experiments are the evidence.

    6. Recommendations for Enterprise Adoption

    This model suits teams with strong compute and engineering capabilities that want to probe the ceiling of super-large MoE in multimodal agents, long-document processing, and complex tool calling. For teams with limited compute, low-latency requirements, or a need for standard open-source licenses, evaluate carefully: the 8-GPU floor, 1.56TB of weights, reliance on low-precision formats, and custom license terms all raise total cost of ownership.

    In summary, the core value of this open-weights release is that it provides material for studying super-large MoE and long-context engineering—while reminding the industry that open weights describe how you obtain a model; open source describes its license. Parameter scale defines the capability ceiling; deployment cost is what determines whether it lands. Enterprises should put architecture understanding, load-testing data, license review, and operations planning on the same evaluation sheet, and avoid being misled by a single parameter count or benchmark score.