How Much VRAM Does a 2.8T MoE Model Need for Local Deployment?

    The weights of a 2.8T-parameter MoE multimodal model have been publicly released. This article answers three key questions developers care about: how much VRAM is required, how to load the weights correctly, and what the commercial license allows. All figures come from the publisher's original materials—always verify against the official model card and license text before making deployment decisions.

    The weights of a 2.8T-parameter MoE multimodal model have been publicly released, and developers immediately face three practical questions: how much VRAM is needed, how should the weights be loaded, and can it be used commercially? This article addresses each question based on the information published by the model’s creators. All numbers below come from the original materials—always verify against the official model card and license text before deployment.

    1. Model Architecture: Total Parameters vs. Activated Parameters

    The model has 2.8T total parameters, but uses sparse activation: each inference pass activates roughly 104B parameters. According to the published information, the model has 93 layers and 896 expert networks. For each token, a routing mechanism selects 16 routed experts, plus 2 shared experts that handle general features.

    This design means storage scales with total parameters while compute scales with activated parameters. Memory pressure remains high, but inference speed can outperform a dense model of comparable scale.

    On the multimodal side, the model integrates a vision encoder and accepts image inputs directly. The context window reportedly reaches 1 million tokens. The original materials also note that, compared to the previous generation, 24 full attention layers were replaced with sparse attention (DSA). Which specific layers were replaced, and the real-world impact on long-context throughput, should be confirmed against the technical report.

    2. VRAM Requirements: Weights First, Then Everything Else

    At roughly 4 bits per parameter, the 2.8T-parameter weights take up about 1.4TB (2.8×10^12 × 0.5 bytes). With single 80GB accelerators, the weights alone require at least 18 cards (1400GB ÷ 80GB ≈ 17.5, rounded up to 18).

    The original materials state “8×80GB to start,” which contradicts this calculation—8 cards provide only 640GB of VRAM, not even enough for the weights. So don’t plan a deployment around 8 cards.

    Beyond the weights, several overheads must be reserved:

    • KV cache: grows with context length and concurrency; a 1-million-token context consumes substantial VRAM.
    • Activations, communication buffers, and framework overhead.
    • Headroom: leave margin in your planning. Evaluate at the 32-card level at minimum, and rely on actual benchmarks.

    Real deployments typically span multiple nodes. Tensor parallelism works within a single 8-GPU machine; cross-node setups require pipeline parallelism as well. The 1.56TB download is also time-consuming—use high-bandwidth storage and verify file hashes before loading.

    3. Weight Format: MXFP4 Cannot Simply Be Loaded as FP8

    According to the original materials, the released version is quantization-aware training (QAT), with weights in MXFP4 precision and activations in MXFP8. No BF16 or FP8 variants have been announced so far.

    Key points when loading:

    • Do not force-load MXFP4 weights with torch_dtype=torch.float8_e4m3fn. The mismatched numerical formats will cause errors or failed loading. Use an inference framework with native MXFP4 support and follow the loading instructions in the model card.
    • device_map="auto" merely distributes layers across GPUs—this is model sharding, not tensor parallelism. For tensor parallelism, use the parallel configuration options in frameworks like vLLM, SGLang, or DeepSpeed.
    • The QAT version is intended for direct inference. If you need high-precision fine-tuning, first confirm your framework supports training on this format.

    4. Performance: Strong at Coding and Agentic Tasks, Weaker at Scientific Reasoning

    According to the original materials, the model performs notably well on coding and agentic tasks. In coding benchmarks such as Terminal-Bench, it scores close to some top-tier closed-source models.

    On scientific reasoning and complex logic tests, the original text notes that scores on benchmarks like Terminal-Bench-Science and HLE-Diamond fall below same-generation closed-source competitors. It also claims the model beats top competitors on roughly 16 of 37 benchmarks. This summary lacks details on testing methodology, so consult the original benchmark tables directly when evaluating.

    A third-party intelligence index gave the model a high ranking, but the specific leaderboard and date need verification—treat it as reference, not an absolute ranking.

    Overall, the model suits code assistance, long-document analysis, and multimodal information extraction. For deep scientific reasoning use cases, validate the results independently.

    5. License: Custom Terms, Not MIT or Apache 2.0

    The model uses a custom license. The core terms permit use, modification, distribution, and commercial use.

    The original materials describe two categories of additional restrictions. First, companies offering the model as a service with cumulative revenue exceeding roughly $20 million over 12 months must sign a separate agreement with the provider before commercial deployment. Second, commercial products exceeding a monthly active user threshold (1 billion is given as an example in the original text) face additional restrictions.

    These figures are paraphrased and do not constitute legal advice. Read the official license text before launch to confirm the applicable version and effective dates. For small businesses and individual developers, the impact is limited in most cases. Teams planning large-scale SaaS offerings should have legal counsel review the terms early.

    6. Practical Recommendations

    1. Audit your hardware first. Estimate GPU count based on weights, KV cache, and headroom, then decide whether self-hosting makes sense.
    2. Validate the model’s fit for your use case via API or a smaller model before committing to a large deployment.
    3. If self-hosting, use a private deployment with container orchestration and logging/monitoring, and clearly define whether your use case counts as “offering the model as a service.”
    4. Teams without local clusters can start with elastic cloud GPUs for short-term testing. When choosing a cloud provider, compare bandwidth, billing models, and data compliance terms.

    Conclusion

    A 2.8T MoE model is best suited for teams with substantial compute or cloud resources. Work out the VRAM budget first, then verify the weight format and license terms—that’s how to make a sound deployment decision.