A 2.8T-Parameter Open-Weight Model: Architecture Breakdown and Local Deployment Considerations

    The release of a 2.8-trillion-parameter open-weight model—roughly 1.56TB of downloadable weights—marks a milestone in publicly available model scale. This article breaks down the architecture behind the model (MoE, sparse attention, and quantization-aware training), clarifies why "open weights" is not the same as "open source" under the model's custom license, and evaluates the real hardware and engineering costs of running it locally. It also offers practical guidance for teams weighing API access against self-hosted inference.

    Background: A Record-Scale Open-Weight Release

    A model with a total parameter count of 2.8T and a download size of approximately 1.56TB has been formally released, making it the largest publicly downloadable model by parameter count to date. The model was previously available via API and web interfaces; the full weights are now published on a public model repository with no application required. Developers can download them directly, and mirror sites are available for users in regions where direct access to the primary repository is slow or restricted.

    From an engineering standpoint, two aspects deserve close attention: the architectural logic behind a model of this scale, and the difference between an “open-weight” license and traditional open-source licenses. For most developers, the 1.56TB download and the minimum 8-GPU deployment requirement also raise a practical question: can this model realistically be run locally?

    Architecture Analysis: How 2.8T Parameters Are Structured

    MoE Design: Separating Total Parameters from Active Parameters

    The model uses a Mixture-of-Experts (MoE) architecture. Total parameters reach 2.8T, but only about 104B parameters are activated per token. It is configured with 896 experts; each token is routed to 16 experts, with 2 additional shared experts always participating in computation.

    This design resolves a core tension for large-model inference: total parameters determine the model’s knowledge capacity, while active parameters determine the compute cost per forward pass. In practice, MoE replaces the FFN layers of a conventional dense Transformer with multiple parallel expert networks, dynamically dispatched by a router. The shared experts guarantee that general-purpose knowledge is always available, preventing instability when routing decisions fluctuate.

    Sparse Attention: Optimizing Compute for Long Contexts

    The model has 93 layers and supports a 1-million-token context window. Compared with the previous generation, the only structural change in the architecture is replacing full attention in 24 layers with a sparse attention mechanism (DSA). Full attention scales quadratically with sequence length, which becomes prohibitively expensive at million-token contexts. Sparse attention computes attention scores only at critical positions, significantly reducing inference cost in long-context scenarios while attempting to preserve long-range dependency modeling.

    Quantization-Aware Training: MXFP Weights

    One detail worth noting: the official release only provides MXFP4 weights with MXFP8 activations—no BF16 or FP8 versions are offered. This is not a simple post-training quantization. The weights are the result of quantization-aware training, where low-precision numerical formats are simulated during the training process so that the model maintains stable performance under low-bit representations. For deployers, this means reduced VRAM usage and lower bandwidth requirements, but it also means that dequantizing to higher precision is not possible—there is no unquantized base weight to recover.

    Multimodal Capabilities

    The model is positioned as a natively multimodal agent model. It includes a built-in vision encoder and can directly process image inputs. “Native multimodal” means that vision and text are modeled together from the training stage, rather than stitched together through an external adapter. This tends to provide better consistency on mixed image-text tasks.

    Benchmark Results: How to Read Vendor-Reported Numbers

    The vendor’s published comparison data shows strong performance on programming and agent-related benchmarks. Some agent coding tests approach or exceed the level of top-tier international models. However, there are notable gaps on scientific and certain reasoning benchmarks, with mixed results overall. Third-party evaluations place the model’s intelligence index in the global top tier.

    Three caveats apply when interpreting vendor-reported results:

    1. Self-reporting bias: Vendor comparison tables typically select benchmark combinations favorable to their own model, and evaluation configurations or prompt templates may be tuned accordingly.
    2. Uneven capability distribution: Very large MoE models show clear advantages on structurally strong tasks like code generation and tool calling, but may not lead on tasks requiring deep scientific reasoning.
    3. API scores vs. local deployment scores: The weight version’s quality depends on the local inference stack’s implementation—sampling parameters, context management, batching strategy, and so on.

    The sensible approach is to build a private evaluation set around your own use case before committing deployment resources.

    License Analysis: Open Weights Is Not Open Source

    This section deserves the most careful reading. The model repository uses a custom license, tagged as “other” rather than MIT, Apache, or another standard open-source license.

    The core of the license is indeed MIT-style: free use, copying, modification, distribution, fine-tuning, and commercial use are all permitted. But two additional restrictions apply to large-scale commercial use:

    • If the model is provided as a service and the company’s cumulative 12-month revenue exceeds a certain threshold (on the order of $20 million), a separate agreement with the publisher is required before commercial use.
    • If a commercial product’s monthly active users exceed a certain scale, the same additional agreement requirement is triggered.

    This model is what the industry calls an “open-weight” license: weights are freely downloadable, but they are not open source under the OSI definition. For individual researchers, small teams, and academic use, the impact is essentially zero. For companies planning to productize the model’s capabilities, the user scale and revenue trajectory need to be assessed at the architecture design stage, with room reserved for compliance negotiations. Read the full license text before deployment—do not assume that “open” means “unrestricted.”

    Local Deployment Feasibility

    Hardware Requirements

    A 1.56TB weight file makes local deployment a capital-intensive exercise. At MXFP4 precision, 2.8T parameters require roughly 1.4TB or more of combined VRAM and system memory. Adding KV cache, activations, and system overhead, a baseline setup would be a single server with 8 high-end accelerators (each with 80GB+ VRAM). Production deployments typically need additional headroom.

    Storage and Download Planning

    With 96 safetensors shards, interrupted downloads and checksum validation are common issues. Use a download tool that supports resuming, and verify the hash of every shard after download:

    1. # Example workflow; actual tool names depend on your environment
    2. download-cli download <model-repo> --resume
    3. sha256sum *.safetensors > checksum.txt
    4. # Compare against the manifest provided in the repo

    In regions with restricted access to the primary repository, using a regional mirror can significantly improve throughput and reduce failure rates.

    Inference Framework Selection

    MXFP4 is a relatively new quantization format, and support varies across mainstream inference frameworks. Before committing, verify the following: does the framework natively support MXFP4 dequantization or custom kernels? Does it support expert parallelism for MoE models? Are efficient kernel implementations for sparse attention already available? If the framework lacks support, you may need to wait for community adaptation or fall back to the cloud API.

    A More Realistic Alternative

    For most teams, building an 8-GPU cluster in-house is hard to justify economically. The more pragmatic paths include: calling the model through an official or third-party API; using community versions quantized to even lower bit widths (if they emerge); or spinning up multi-GPU instances on a cloud container platform, running them on demand, and releasing them when done—turning fixed capital costs into variable costs.

    Conclusion

    This release marks the entry of open-weight models into the trillion-parameter era. Technically, the combination of MoE, sparse attention, and quantization-aware training demonstrates a viable engineering path for balancing capacity and cost at extreme scale. Under licensing, the “open weights plus commercial threshold” hybrid model may become a new norm for large-model releases. For developers, the real question is not whether the model can be downloaded—it is whether deployment is worth it. Evaluating business scale, data sensitivity, and cost budgets is the right way to approach a model of this magnitude.