2.8-Trillion-Parameter Open Model: How Many GPUs Do You Need for Local Deployment?

    A 2.8T-parameter MoE model with publicly released weights (~1.56TB) raises a practical question: how many GPUs does local deployment actually require? This article covers the model's architecture, capabilities, license terms, and a concrete VRAM calculation showing why 8×192GB GPUs won't work and 12–16 is the realistic starting point.

    1. What Happened, and What “Open” Actually Means

    In early November, a Mixture-of-Experts (MoE) model with a total of 2.8 trillion (2.8T) parameters released its full weights publicly, with a download size of roughly 1.56TB. The publisher had already made the model available via API and a web service in late October, promising to release the weights about a week later. Once published, the weights can be downloaded without any application process.

    The nature of this “open” release needs to be clarified first. It is not open source in the standard sense. The repository uses a custom license, marked as “license: other” on the hosting platform. Being able to download the weights does not mean you can use them freely — you must review the license terms line by line before any commercial use. All parameters and configurations cited in this article come from the publisher’s public materials; always refer to the official model card and configuration files before deployment.

    2. Architecture Highlights

    2.1 Mixture-of-Experts Design

    The model has 896 experts in total. Each token is routed to 16 experts, plus 2 shared experts. The network has 93 layers, supports a 1-million-token context length, and activates roughly 104B parameters per token.

    104B is on the order of a hundred-billion-parameter model. Sparse activation means each token only passes through a small fraction of the experts, so the compute per inference step is comparable to a hundred-billion-scale dense model. Note that this conclusion applies only to compute. Lower compute does not mean lower memory requirements. In typical single-node or multi-GPU inference, all weights usually need to reside in GPU memory, so VRAM usage still scales with the total parameter count. CPU or NVMe offloading can reduce VRAM pressure, but latency and throughput degrade significantly, so it should not be the default approach.

    2.2 Sparse Attention

    Compared to the previous generation, the main architectural change is replacing full attention with sparse attention in some layers, reducing compute for long contexts. Full attention scales quadratically with sequence length, so sparsification is a common way to control costs at million-token context lengths.

    Before deployment, confirm that your inference framework — and its specific version — supports this operator. To verify: after starting the inference service, check the framework logs for any fallback to a generic implementation, and compare throughput under identical inputs. If the framework falls back, speed may drop noticeably, and any performance measured in that state does not reflect the model’s true capability.

    2.3 Multimodal Input

    The model has a built-in vision encoder and can process image inputs directly, with no need to attach a separate vision module.

    2.4 Quantization Format

    Only one weight version has been released. Weights are stored in MXFP4, activations use MXFP8, and the model was trained with quantization-aware training (QAT). Because quantization error was simulated during training, performance loss at low precision is relatively contained.

    The trade-off of this design is clear. The main benefit is size: 1.56TB is far smaller than the BF16 version estimated from the parameter count (~5.6TB). The cost is a lack of options: there is no higher-precision version, so you cannot trade precision for memory yourself, and fine-tuning has limited precision flexibility.

    3. Capability Profile

    According to benchmarks disclosed by the publisher, capabilities are unevenly distributed. Coding and agent-style tasks are strengths, while science and some reasoning tasks are relatively weaker.

    This article does not cite unverified scores. A single score is only meaningful together with the benchmark name, version, and test conditions. For model selection, rely on your own evaluations, covering at least these three task types:

    • Code completion and modification: test with tasks from your team’s real repositories, and record pass rates and rework counts.
    • Agent tool calling: test the format correctness of function calls and the completion rate of multi-step tasks.
    • Long-document QA: test with documents of realistic length and check whether key information is missed.

    4. License Checklist

    The license grants free rights to use, copy, modify, distribute, fine-tune, and commercially use the model. However, it is a custom license — not a standard open-source license like MIT or Apache — and it carries over the terms of the previous-generation model under a new name. Before enterprise use, check each of the following:

    • User-scale threshold: does the license restrict monthly active users or revenue scale?
    • Use restrictions: are certain industries or application scenarios prohibited?
    • Attribution requirements: must the source be credited in product UI or documentation?
    • Redistribution limits: what conditions apply when distributing the weights or derivative models?
    • Derivative model terms: do the same terms apply to fine-tuned models?
    • Commercial liability: who bears infringement and compliance risks, and is this consistent with standard open-source practice?

    Legal review should be based on the original license text, not secondhand summaries.

    5. Local Deployment Feasibility

    5.1 How to Calculate VRAM

    Required VRAM can be broken into three parts:

    Required VRAM ≈ weights + KV cache + activations and framework overhead

    This article uses decimal units throughout: 1TB = 1000GB. The weights are about 1.56TB, or 1560GB.

    The KV cache grows linearly with context length and concurrent requests. Its size depends on the number of layers, attention head configuration, and per-token KV dimensions. Since public materials do not provide these details, an exact estimate is not possible. Be aware that at a 1M-token context, the KV cache can consume a large amount of VRAM — rule-of-thumb margins are no substitute for actual measurement.

    Using a common rough estimate, weights plus a 20%–30% margin come to about 1870–2030GB. This is only an approximation: it does not reflect real KV cache growth, nor does it include communication and partitioning overhead.

    Assuming 192GB per GPU, the results are as follows. The table accounts for VRAM only — it does not include parallel partitioning efficiency, inter-GPU communication, redundancy, or system overhead.

    GPU count Total VRAM VRAM remaining after weights Verdict
    8 1536GB Insufficient — below the weight size Not feasible
    12 2304GB ~744GB Theoretically feasible; partitioning scheme and KV cache usage must be validated
    16 3072GB ~1512GB A more common configuration; still requires real-world testing

    With 8 GPUs, total VRAM is already below the weight size — the weights alone won’t fit. With 12 or 16 GPUs, there is headroom beyond the weights, but whether it is sufficient depends on context length, concurrency, and the inference framework, and must be confirmed by testing. GPUs with less memory per card (e.g., 141GB models) require more cards and are not covered in the table.

    Conclusion: with 192GB GPUs, 12 cards is the theoretically feasible starting point, and 16 is the more common configuration. Since 12 is not a power of two, the partitioning scheme needs framework support. Both configurations require validating expert-parallel communication overhead and interconnect bandwidth.

    5.2 What Else to Evaluate Beyond VRAM

    • Inter-GPU interconnect bandwidth: when experts are spread across multiple GPUs, tokens must be forwarded between cards; insufficient bandwidth will slow down inference.
    • Expert-parallel communication overhead: tokens routed to experts on different GPUs incur additional communication costs that must be measured in practice.