2.8T MoE Open-Weight Model: Architecture, Quantization, VRAM, and Deployment Limits

    This article examines a 2.8T-parameter Mixture-of-Experts open-weight model with a download size of about 1.56TB. It covers the 93-layer, 896-expert architecture with roughly 104B activated parameters per forward pass, the MXFP4/MXFP8 QAT-only release, dynamic sparse attention for long context, vision capabilities, VRAM estimation, multi-GPU parallelism, and capability boundaries. The goal is to help readers judge whether their hardware is sufficient and whether the model is worth testing.

    The model has 2.8T total parameters and a download size of about 1.56TB. The vendor describes it as one of the largest open-weight models currently available for download. The larger the model, the higher the deployment barrier. This article covers architecture, quantization, VRAM estimation, and capability boundaries in turn, so readers can judge whether their hardware is sufficient and whether the model is worth testing.

    Architecture: Wide Network and Sparse Activation

    The model has 93 layers and contains 896 expert networks. For each token, the routing mechanism selects 16 routed experts to participate in computation, plus 2 shared experts used by all tokens. As a result, each forward pass activates about 104B parameters.

    The design goal is to provide high total model capacity while keeping per-inference compute tied to the activated parameters. Note, however, that the parameters of all experts still need to be loaded into VRAM. This is the main source of deployment difficulty discussed later. The vendor claims that the model’s knowledge capacity is close to or even exceeds that of trillion-parameter dense models. This is a vendor statement and has not been independently verified in this article.

    For long context, the vendor claims support for 1 million tokens of input. To achieve this, 24 layers replace full attention with dynamic sparse attention (DSA). Full attention compute grows quadratically with sequence length, while KV Cache usage grows linearly with sequence length. DSA reduces compute for long sequences, but whether it also reduces KV Cache usage must be confirmed by actual measurements in the inference framework.

    The model includes a built-in vision encoder. The vendor positions it as a native multimodal Agent model for scenarios such as image understanding, interface operation, and chart analysis.

    Quantization: MXFP4 Weights and MXFP8 Activations

    The official release only includes a quantization-aware training (QAT) version, with MXFP4 weight precision and MXFP8 activation precision. There is no BF16 or FP8 version.

    MXFP4 uses 4 bits per element, with every 32 elements sharing an 8-bit scaling factor, averaging about 4.25 bits per parameter. Compared with post-training quantization, QAT simulates low-precision computation during training, allowing the model to adapt to precision loss. As a result, deployment precision is usually more stable.

    This single-release strategy places high requirements on the inference environment. If the inference engine does not support MXFP4/MXFP8 operators, the only option is to dequantize to a higher precision for execution. VRAM usage and speed will change, and precision deviations may occur. Before deployment, two checks are recommended: confirm whether the engine version supports the relevant operators, and compare output results against vendor-published data using the same set of fixed prompts.

    Deployment: VRAM Estimation and Parallelism Options

    The formula for estimating weight VRAM is: total parameter count × bytes per parameter, plus KV Cache and runtime overhead.

    2.8T parameters × 0.5 bytes ≈ 1.4TB (about 1300GiB). This result is close to the 1.56TB download size. The difference mainly comes from scaling factors and some layers that are not quantized.

    For weights alone, assuming 80GB per GPU, at least 18 GPUs are required (1400GB ÷ 80GB ≈ 17.5). This number does not include KV Cache, activations, or framework overhead. Actual usable VRAM is usually smaller, so more GPUs will be needed.

    The official claim of an “8-GPU minimum” needs verification. Eight 80GB GPUs provide 640GB in total, which cannot hold about 1.4TB of weights. If an 8-GPU configuration relies on large per-GPU memory or CPU/SSD offload, throughput and concurrency will be significantly limited and must be confirmed through testing.

    For multi-GPU deployment, two common parallelism methods are used. Expert parallelism (EP) lets each GPU store a subset of experts, and tokens must be routed between GPUs, creating high communication overhead. Tensor parallelism (TP) splits each layer’s weights across multiple GPUs, with frequent synchronization communication. Configurations with 8 or more GPUs usually require a hybrid of both, which places high demands on the inference framework’s scheduling capability.

    For most teams, building a multi-node GPU cluster in-house requires a large investment. A more practical approach is to first rent GPU instances and container services from a cloud provider for small-scale testing, then use object storage to distribute weights and cache them locally to avoid repeatedly downloading files of 1.4TB or more. Pay-as-you-go pricing can help teams control costs during validation.

    Capability Boundaries

    Vendor-published data shows that the model performs strongly in coding and Agent-related tests. The mentioned Terminal-Bench 4.0 score exceeds some closed-source models. In 37 benchmarks, 16 exceeded the comparison baseline. The test versions, prompts, and tool-calling settings for these results have not been made public, so they need to be reproduced independently.

    In scientific reasoning tests, the model’s results are noticeably lower than some closed-source models. The mentioned third-party intelligence index ranking is high, but the evaluator and specific scores have not been verified, so it should not be cited directly.

    Selection Advice

    If you have stable resources of 8 or more GPUs or multiple nodes, and your business is mainly coding assistance or Agent automation, the model can be included in your evaluation scope.

    If tasks are mainly question answering, summarization, or lightweight classification, smaller models or on-demand API calls are usually more cost-effective.

    If the core business is scientific reasoning, run benchmark comparisons on your own data before deciding whether to deploy.

    Individual developers and small teams without GPU operations capability are not suitable for directly self-hosting this model.