How Many GPUs Are Needed to Deploy a 2.8T Open-Weight Model Locally?

    This article estimates the GPU requirements for local deployment of a 2.8T open-weight MoE model. It explains how weight precision, context length, concurrency, and VRAM offloading affect hardware needs, provides GPU-count estimates for 80GB, 141GB, and 192GB cards, compares full-VRAM residency, offloading, and evaluation-only configurations, and lists license points to verify before commercial use.

    1. Bottom Line Up Front

    How many GPUs are needed to deploy this model locally depends on four variables: weight precision, context length, concurrency, and whether VRAM offloading is used. As long as the full model weights must reside in VRAM, the GPU count is mainly determined by total weight volume, not by the number of parameters actually used in each inference run.

    Based on public information, an environment with 80GB GPUs needs at least twenty-plus GPUs to hold all weights. An environment with 141GB or 192GB GPUs also needs around a dozen or about ten GPUs. If slower speed is acceptable, placing expert parameters in host memory or on NVMe disk can greatly reduce VRAM requirements, but throughput will drop significantly. If the goal is only to validate model quality, renting temporary compute or using a hosted inference service is usually more cost-effective than building a long-term local deployment.

    The original source does not provide the publisher name, weight repository URL, or license link. This article does not infer or add that information; readers should verify it through official channels.

    2. Background

    According to the original source, a model vendor released the full weights of a new-generation flagship model on the previously promised timeline. The total parameter count is 2.8T, and the weight files are about 1.56TB. They can be downloaded directly from a public model hosting repository without an application. The original source says this is the largest open-weight model publicly released to date; that claim should be checked against the official release list.

    The meaning of open weights is that developers can download the weights, inspect the implementation, fine-tune the model, and deploy it in their own environment. Developers first need to answer two questions: what hardware is required to run it locally, and what the license actually permits.

    3. Model Architecture and Its Relationship to VRAM

    3.1 MoE Sparse Activation

    The model uses a mixture-of-experts (MoE) architecture. MoE splits the network into many expert modules, and a gating network selects some experts for each token to participate in computation. During forward computation for each token, only about 104B parameters are activated, about 3.7% of total parameters, meaning about 96% of parameters are idle in a single computation. The model contains 896 routed experts and 2 shared experts. The gating network selects 16 routed experts for each token, and the shared experts handle general knowledge.

    Sparse activation saves compute; it does not reduce VRAM usage. Expert selection varies with input, and the system cannot know in advance which experts the next token will use. Therefore, to keep all experts available at all times, the full 2.8T-parameter weights must be kept in VRAM, or placed in memory or disk through an offloading scheme.

    3.2 Sparse Attention

    The model has 93 layers and natively supports a million-token context window. In 24 layers, full attention is replaced by sparse attention. In sparse attention, each query computes attention only with a filtered subset of keys. With a fixed number of retained keys, this part of the computation no longer grows quadratically with sequence length.

    This change has two limitations. First, it holds only with a fixed number of retained keys or a specific sparse pattern; the actual benefit depends on the configuration of the sparse layers. Second, KV cache (key-value cache, used to store attention intermediate results for historical tokens) still grows linearly with context length, so long contexts still pressure VRAM. If filtered-out keys contain critical information, answer quality can suffer, especially for precise retrieval tasks. Whether it is usable requires testing with your own long-document samples.

    3.3 Multimodal and Agent Positioning

    According to the release notes, the model natively supports multimodal input, includes a built-in vision encoder, and can process images directly. The publisher positions it as an agent model that supports tool calls and multi-step task execution. If your scenario requires calling external tools to complete multi-step workflows, this capability is more worth testing than pure conversation ability.

    3.4 Quantization Scheme

    Weights are stored in MXFP4, a micro-scaling format with 4-bit floating point plus shared scaling factors. Activations use MXFP8. The publisher says this is the result of quantization-aware training (QAT), meaning quantization error was considered during training, so accuracy loss is usually smaller than with post-training quantization (PTQ).

    Volume can be roughly checked. 2.8T parameters at 4 bits have a theoretical volume of about 1.4TB. The MX format has an 8-bit scaling factor for every 32 elements, equivalent to about 4.25 bits per parameter, or about 1.49TB. There is still a gap of about 0.07TB from the 1.56TB file, possibly from unquantized layers such as embedding layers, output layers, or some attention weights. The exact composition should be checked against the official weight list.

    Note that the official release provides only this one quantized version. If your inference framework or hardware does not support MXFP4 or MXFP8, confirm whether there is a fallback or whether you need to convert the format yourself. Conversion may introduce accuracy changes and should be validated with benchmarks before launch.

    4. VRAM Estimation Method

    Single-GPU VRAM usage can be estimated as follows:

    Single-GPU VRAM ≈ weight volume ÷ number of parallel shards + per-GPU KV cache share + activation and communication buffers + framework reserve

    The lower bound for total GPU count can be calculated as follows:

    Required GPUs ≥ weight volume ÷ (single-GPU VRAM × usable ratio)

    The usable ratio is usually 0.8 to 0.9 to leave room for KV cache, activations, and framework overhead. Weight volume is 1560GB.

    KV cache can be estimated as follows:

    KV cache bytes ≈ 2 × number of layers participating in caching × number of KV heads × head dimension × bytes per element × context length × concurrency

    The number of layers, KV heads, and head dimension need to be read from the model configuration file. The original source does not provide these parameters, so this article does not give a specific KV cache number. If sparse layers store KV differently from full-attention layers, handle them separately.

    Based on the formula above, the lower bounds for weights under different single-GPU memory specs are as follows, in decimal rough estimates:

    Single-GPU VRAM Lower bound for GPUs for weights only (usable ratio 0.85) Practical recommended GPU count
    80GB About 23 24 to 32
    141GB About 14 16
    192GB About 10 16

    Actual GPU count also depends on whether the parallelism method can evenly split experts and on intra-node GPU interconnect bandwidth. The table is only a lower bound and recommendation, not a substitute for testing.

    5. Three Deployment Configurations

    5.1 Full VRAM Residency

    All weights reside permanently in VRAM, giving the fastest inference but also the greatest dependence on hardware scale. According to the table above, 80GB GPUs require more than 24 GPUs, and 141GB or 192GB GPUs require around 16 GPUs. In multi-node deployment, inter-node interconnect bandwidth directly affects communication overhead for expert parallelism. This configuration suits service scenarios that need stable throughput.

    5.2 VRAM Plus Offloading

    This configuration keeps frequently used parts such as attention layers, shared experts, and embedding layers in VRAM, and places most routed experts in host memory or on NVMe disk, loading them when needed.

    Memory or disk must hold all weights, so host memory should be at least about 2TB, or NVMe disk at least about 2TB.

    Speed is limited by data movement. Each token needs about 104B activated parameters, about 55GB at 4.25 bits. If all these parameters are moved from host memory over PCIe 5.0 x16, theoretical bandwidth is about 64GB/s and effective bandwidth is about 50GB/s, so the upper speed limit is about 1 token/s. Actual speed depends on the cache hit rate for experts. If routed experts repeat often within the same batch of requests, speed will improve noticeably, but the exact value can only be determined by testing.

    This configuration suits low-concurrency, latency-insensitive offline tasks such as batch data processing or overnight evaluation.

    5.3 Evaluation Only

    If the goal is only to judge whether model capability meets requirements, you do not need to occupy hardware long term. There are two options:

    • Use a hosted inference service and test through API calls.
    • Rent a large-VRAM node, load the weights once, run your own test set in batches, and release the node afterward.

    This approach does not pursue throughput; the focus is covering your own business samples, including long-document retrieval, tool calls, multi-step tasks, and performance in target languages.

    5.4 Comparison of the Three Configurations

    Configuration Core hardware Speed characteristics Suitable scenarios
    Full VRAM residency 16 to 32 high-VRAM GPUs, multi-node interconnect Fast, stable serving Long-term online service
    VRAM plus offloading A few high-VRAM GPUs, about 2TB memory or disk Slow, about 1 token/s theoretical upper limit Offline batch processing
    Evaluation only Hosted service or temporary rented node Not a metric Quality validation to decide whether to self-host

    6. License Verification Checklist

    Open weights do not mean unconditional use. Before making any commercial decision, review the official license text and model card item by item, focusing on the following:

    1. Commercial use: whether use in for-profit products and services is allowed.
    2. Redistribution: whether forwarding weights or packaging weights into your own product is allowed.
    3. Derivative models: whether a fine-tuned model can continue to use the original name, whether it must be renamed, or whether additional terms apply.
    4. Patent grant: whether the license includes a patent grant clause, and what the scope and termination conditions are.
    5. Acceptable use policy: which uses are prohibited and what the consequences of violation are.
    6. Usage or scale thresholds: whether there are limits based on user count, revenue, or call volume.
    7. Attribution and notices: whether you must indicate the source or retain notice files when using the model.
    8. Export and geographic restrictions: whether there are restrictions on specific countries, regions, or entities.

    The list above covers common categories only; the full license text and model card should be reviewed item by item.