Can a 2.8T MoE Open-Weight Model Run on 8 GPUs?

    This article evaluates the deployment feasibility of a 2.8T parameter Mixture-of-Experts (MoE) open-weight model. It analyzes the model's architecture, MXFP4 quantization, custom licensing, and hardware requirements. The analysis concludes that a single 8-GPU server is insufficient for local deployment due to the 1.56TB weight size, recommending API or cloud-based solutions for initial testing before committing to multi-node infrastructure.

    The Bottom Line

    While this model is available for download, it cannot run on a single 8-GPU server. Its weight files are approximately 1.56TB, and an 8-way GPU server with 80GB cards only provides 640GB of VRAM combined—insufficient to even hold the weights themselves. Local deployment requires at least multiple high-end servers. For most developers, it is more practical to validate the model’s performance via API or on-demand cloud compute first, and then decide whether to build a local infrastructure based on confirmed requirements.

    1. “Open Weights” vs. “Open Source”

    “Open weights” means the model parameter files can be downloaded and used. “Open source,” however, typically requires the licensing agreement to comply with the free-use terms defined by the Open Source Initiative (OSI). This model uses a custom license, marked as license: other on the platform, rather than a standard license like MIT or Apache-2.0. This distinction directly impacts commercial usage, as detailed in Section 3.

    2. Model Architecture: How the 2.8T Parameters Work

    2.1 MoE Structure

    This model is continually trained from its predecessor and is positioned as a native multimodal Agent model. The primary specifications are as follows:

    • Total parameters: ~2.8T
    • Activated parameters per token: ~104B
    • Routed experts: 896, with 16 activated per token
    • Shared experts: 2
    • Layers: 93
    • Context length: 1 million tokens
    • Visual encoder: In-house developed module supporting image understanding

    MoE expands capacity through sparse activation. All 2.8T parameters must reside in VRAM or system memory, but each forward pass only utilizes about 104B parameters. The inference compute load is comparable to a ~100B dense model, while its capability ceiling approaches that of a ~1T dense model. A smaller compute load does not equate to lower resource consumption; VRAM usage is still dictated by the total parameter count.

    2.2 Changes from the Previous Generation

    Compared to the previous generation, the architecture has only one major change: full attention in certain layers has been replaced with sparse attention to reduce the compute overhead of long contexts. Since attention compute grows quadratically with sequence length, a 1-million-token window typically relies on some form of sparsification. The specific implementation of the sparse pattern requires checking the official inference code.

    2.3 MXFP4 Weights: How Much Does the Size Shrink?

    The official weights are stored in MXFP4, with activations in MXFP8. This is a product of quantization-aware training (QAT), meaning quantization errors were accounted for during training rather than applying post-training compression.

    In MXFP4, each parameter is represented by 4 bits, with every 32 parameters sharing an 8-bit block scale, averaging about 4.25 bits per parameter. Based on this ratio:

    • Compared to BF16 (16 bits): The size is roughly a quarter. A BF16 version of a 2.8T parameter model would be ~5.6TB, while the MXFP4 version is ~1.5TB.
    • Compared to FP8 (8 bits): The size is roughly half.

    The actual file size is 1.56TB, slightly larger than the theoretical 1.49TB. This discrepancy likely stems from high-precision tensors (e.g., embedding, normalization layers) and metadata being retained. To verify this, check config.json and the safetensors headers of each shard rather than blindly trusting estimates.

    The official release only includes the MXFP4 version. If you need to perform secondary training in BF16 or FP8 precision, you must dequantize it yourself, which will result in precision loss.

    3. How to Verify the License Agreement

    The main terms resemble a permissive MIT-style license, allowing usage, copying, modification, distribution, fine-tuning, and commercial use. However, custom licenses usually append restrictions. Before commercialization, legal or technical leads should verify the following checklist:

    1. Commercial thresholds: Are there reporting requirements based on user volume, revenue, or compute scale, and what actions are triggered?
    2. Attribution and usage declarations: Must the model source be cited in the product? Are there prohibited use cases?
    3. Redistribution: Does redistributing weights require attaching the license text? Is public redistribution allowed?
    4. Derivative models: Are models derived from fine-tuning or distillation subject to the same constraints? Can they be published under a different license?
    5. Output usage: Can generated content be used to train other models?
    6. Geographic and export restrictions: Are there limitations for specific regions or organizations?
    7. Version changes: Might future versions alter the licensing terms, and do historical versions remain applicable?
    8. Tooling impact: license: other cannot be identified by automated compliance tools, requiring manual review and documentation.

    4. How to Interpret Vendor Benchmark Data

    Vendor-published comparison data shows this model performs exceptionally well on coding and Agent benchmarks, surpassing some leading international closed-source models in certain areas. However, it still lags on scientific and certain reasoning benchmarks.

    When reading such data, note three key points:

    1. Selection bias: Vendors typically showcase their strengths. Compare against the full benchmark list to see the distribution of strong and weak areas.
    2. Evaluation criteria: The evaluation framework, prompt templates, whether “thinking mode” is enabled, and tool-calling settings all affect scores. Numbers from different sources cannot be directly compared.
    3. Weights vs. API differences: The performance of quantized weights may differ from the API version. You should build an evaluation set using your own business data.

    Rankings on third-party leaderboards also require recording the leaderboard name, version, release date, and evaluation settings. Rankings lacking this information should not be cited directly.

    5. Local Deployment: Can 8 GPUs Handle It?

    5.1 How Much VRAM is Required?

    First, calculate the weights. 1.56TB is approximately 1560GB:

    • 8x 80GB GPUs: 640GB total, can only hold about 41% of the weights.
    • 8x 192GB GPUs: 1536GB total, still less than 1560GB.
    • Single 80GB GPU: You need at least 20 cards to hold the weights, and around 22 cards if reserving VRAM for overhead.

    Beyond weights, there are three additional overheads:

    • KV Cache: Continuously occupies VRAM under long contexts. The estimation formula for standard attention is: 2 × layers × KV heads × head dimension × sequence length × bytes per element. This model’s attention implementation is different, so base calculations on the actual parameters in the official config.
    • Activations: Varies with batch size and sequence length.
    • Framework overhead: Communication buffers, CUDA contexts, etc. Generally reserve 10% to 20%.

    A comprehensive estimate can be made using these two formulas:

    Required VRAM ≈ (Weights + KV Cache + Activations) × (1 + Framework overhead ratio)

    Number of GPUs ≥ Required VRAM ÷ Usable VRAM per GPU

    5.2 GPU Estimation

    Using common specifications:

    • 2x 8-GPU 141GB servers (2256GB total): Leaves about 700GB for KV Cache and activations after weights. This is a realistic entry-level configuration.
    • 3x 8-GPU 80GB servers (1920GB total): Leaves about 360GB beyond weights. This margin is tight and requires empirical testing.
    • A single 8-GPU server cannot meet the requirements, regardless of whether the cards are 80GB or 192GB.

    Inter-node interconnect bandwidth significantly impacts the expert routing communication in MoE models, necessitating high-speed networks.

    CPU memory or NVMe offloading can reduce VRAM requirements, but every inference requires moving parameters from memory or disk, drastically reducing throughput. This is only suitable for offline batch processing or extremely low-concurrency testing.

    Loading time is also not short. At a sustained read speed of 2GB/s, 1.56TB of weights takes about 13 minutes; at 1GB/s, about 26 minutes. Actual times are affected by storage and network conditions.

    5.3 Comparison of Four Usage Paths

    Path Applicable Scenario Hardware or Prerequisites Cost & Risk Data Compliance Ops Complexity
    A: Local multi-node cluster Hardware budget available, data cannot leave the domain At least 2x 8-GPU 141GB or 3x 8-GPU 80GB servers with high-speed interconnect High procurement & power costs, high failure rates and parallel debugging costs Data stays in internal network High
    B: Cloud on-demand compute Validation phase or low-frequency usage Rent multi-GPU instances on-demand, with object storage Billed by hour, long weight download and startup times Depends on cloud provider region and contract terms Medium
    C: Distilled small model Resource constrained, fixed tasks Runs on a single GPU or single machine Requires training data and evaluation, capability loss Data controllable Medium
    D: Direct API calls Most application scenarios No hardware required Pay-as-you-go, pricing and rate limits vary by provider Must evaluate data export and retention policies Low

    5.4 Frameworks and Parallelism

    • Parallelism strategies: Tensor parallelism, pipeline parallelism, and expert parallelism can be combined. Experts are distributed across different cards, making all-to-all communication during the routing phase the primary overhead.
    • Framework support: Engines like vLLM, SGLang, and TensorRT-LLM have varying levels of support for MXFP4, sparse attention, and 1-million-token contexts depending on the version. Check their release notes before selection and test on the target version.
    • Long context: The KV Cache for 1 million tokens requires paged management mechanisms like PagedAttention.
    • Residency and updates: Loading takes a long time, so the model should remain resident in production environments. Use rolling replacements when updating weights to avoid complete downtime.

    6. Download and Supply Chain Risks

    • Storage: 1.56TB of weights requires sufficient disk space, considering multiple replicas and backups.
    • Integrity: After downloading, verify the file hashes provided by the official repository to ensure each shard is intact.
    • Mirrors: The source and content of community mirror sites are not always trustworthy. Verify hashes before use and confirm the identity of the mirror maintainer.
    • Code execution: The safetensors format itself does not execute code, but custom model code in the repository can execute arbitrary commands. If loading requires enabling “trust remote code,” review this code first and run it in an isolated environment.
    • Data compliance: When using cloud or APIs, confirm whether data crosses borders and whether it is retained or used for training.

    7. Actionable Recommendations

    First, test your specific tasks using APIs or small-scale cloud compute. Once performance is confirmed, calculate the required VRAM and number of GPUs. Finally, complete the license review. Only after completing these three steps can you determine whether this model is worth deploying in-house.