Deploying a 2.8T Open-Weight MoE Model: VRAM Budget and the 8-GPU Minimum

    When a 2.8T-parameter mixture-of-experts (MoE) model that activates only ~104B parameters per token releases its full weights, the conversation shifts from benchmark scores to whether it can actually run. This article walks through what those 1.56TB of weights contain, why they force an 8-GPU high-memory starting point, how MXFP4 weights and MXFP8 activations reflect quantization-aware training, how to read vendor self-reported results without over-reading them, and which license terms to verify before deploying. It closes with a concrete deployment checklist and a decision table for building versus hosting.

    1. Start with the conclusion: after an open-weight release, three engineering questions need answers

    When a mixture-of-experts (MoE) model with 2.8T total parameters that activates only ~104B per token releases its complete weights, industry attention quickly shifts from “what’s the score” to “can it run, and can I afford it.” Releases like this usually raise three engineering questions that must be answered:

    1. What is the actual structure inside the weights, and how are the parameters distributed?
    2. What is the conversion relationship between the 1.56TB footprint and the “8 GPUs per node as a starting point” requirement?
    3. How should the license terms and the self-reported results each be interpreted?

    Note that “open weights” and “fully open source” are not the same thing. The former releases the model parameter files; the latter also includes training code, data, and a reproducible training pipeline. The vast majority of very large models fall into the former category and come with additional usage terms. So before wiring it into a production pipeline, license review and performance testing matter just as much.

    2. What makes up the 2.8T parameters: MoE and sparsification are the main sources of the footprint

    2.1 Sparse activation: total vs. activated parameters differ by ~27x

    A typical configuration for this kind of model is: 2.8T total parameters, ~104B activated per token, using a MoE structure—hundreds of routed experts, with each token hitting only a few of them (for example, 16 out of 896), plus a small number of always-on shared experts that carry general capabilities, around 90 layers, a context window that can reach the million-token level, and a native vision encoder to support image-plus-text input.

    Key conclusion: activated parameters determine inference compute, while total parameters determine VRAM usage. MoE trades VRAM for compute, keeping FLOPs within a range a dense model could handle, but at the cost that nearly all experts must stay resident in VRAM—routing is dynamic, and you cannot predict which experts the next token will hit.

    1. # Illustration: fusing routing and shared experts within a single layer (pseudocode, structure only)
    2. def moe_layer(x, router, experts, shared):
    3. logits = router(x) # [tokens, num_experts]
    4. topk_w, topk_i = topk_softmax(logits, k=16)
    5. y = zeros_like(x)
    6. for idx, weight in zip(topk_i, topk_w):
    7. y += weight * experts[idx](x) # expert compute, grouped by token
    8. return y + shared(x) # shared expert, always-on compute

    2.2 How this generation differs from the previous one: trading sparse attention for long-context compute

    What typically changes between generations is not the parameter count but the attention mechanism: replacing full attention in some layers with sparse attention (for example, switching 24 of 93 layers to sparse attention), so that compute growth in long-context scenarios no longer climbs quadratically with sequence length.

    This is directly tied to VRAM. A full-attention KV cache is a linearly growing memory sink; sparse attention lowers per-token KV overhead while also reducing bandwidth pressure on long sequences. The practical takeaway for deployers: the longer the context, the more obvious the sparse-attention benefit—but baseline concurrency still has to be estimated on the most conservative assumptions.

    3. The VRAM budget: why “1.56TB of weights” necessarily points to an 8-GPU starting point

    3.1 A three-part estimation model

    To judge “can it run,” engineers should not look at weight size alone. Use a three-part approach:

    1. VRAM needed ≈ weight footprint + KV cache + runtime overhead
    2. where weight footprint = parameters × bytes per parameter (incl. shard alignment and metadata)
    3. KV cache = 2 × layers × KV heads × head_dim × bytes × sequence length × concurrency
    4. runtime overhead ≈ 10%~20% (activation buffers, comm buffers, framework overhead, VRAM fragmentation)

    3.2 The weight part: 8 GPUs is a hard floor, not a recommendation

    Spread 1.56TB of weights across 8 GPUs, and that is about 195GB per card. This means an 8-GPU-per-node high-memory accelerator is the price of entry, not an optimization. More importantly, watch the headroom:

    • The weights already consume most of a single node’s HBM, leaving very little room for the KV cache and activations;
    • As a result, practical usable concurrency is usually far lower than intuition would suggest for a dense model of the same parameter count;
    • The main way to raise concurrency is not adding cards, but KV quantization, prefix reuse, expert parallelism, and CPU/host-memory offloading.

    3.3 The KV part: write down your assumptions before calculating

    To demonstrate the order of magnitude with illustrative assumptions (the following is an example calculation; the parameters are arbitrarily set and do not represent any specific model configuration): assume 93 layers, 8 KV heads, head_dim 128, and KV stored in FP16. Then the per-token KV overhead is roughly:

    1. per_token = 2 * 93 * 8 * 128 * 2 # ≈ 372 KB / token
    2. for ctx, conc in [(8_000, 32), (128_000, 4), (1_000_000, 1)]:
    3. print(ctx, conc, per_token * ctx * conc / 1024**3, "GiB")

    The conclusion is intuitive: a normal business workload with short context (8K context, 32-way concurrency) and “a single million-token request” are two completely different resource profiles. The latter, even at concurrency 1, can consume hundreds of GB—which is exactly why sparse attention exists. If you quantize the KV cache to 4-bit, this part can shrink by about 75%, which is usually a better deal than continuing to compress the weights.

    3.4 Two hidden costs beyond the weight part

    • MoE communication overhead: when experts are spread across multiple GPUs, all-to-all communication grows with the degree of expert parallelism, and inter-card interconnect bandwidth (rather than single-card compute) often becomes the throughput bottleneck;
    • The cost of CPU/host-memory offloading: placing cold experts in host memory or local NVMe can break through the single-node HBM ceiling, but it introduces a PCIe bandwidth bottleneck that shows up as first-token latency jitter, and it needs to be paired with warm-up and affinity scheduling.

    4. MXFP4 weights + MXFP8 activations: the tradeoff of quantization-aware training

    4.1 Why only one precision version is released

    The official release offers only one version—MXFP4 weights with MXFP8 activations—and no BF16 or FP8. This is usually the inevitable result of quantization-aware training (QAT): the low-bit numerical range was folded into the optimization objective during training, so the model’s robustness to this format is already “baked into the weights.” Dequantizing back to BF16 not only doubles the footprint but may not recover higher precision, because the training process never targeted BF16.

    Three implications for users:

    1. You cannot trade precision for quality: the path of switching precision formats to improve results is essentially closed off;
    2. Fine-tuning must choose the right route: stacking low-rank adaptation on top of low-bit weights is a common approach, but dequantization and repacking consume extra VRAM and host memory;
    3. Hardware compatibility must be verified in advance: not all accelerators natively support these micro-scaled block formats; without support, software emulation is required and throughput may drop substantially.

    4.2 Where low-bit tends to lose points

    Low-bit is less friendly to “long-tail reasoning tasks.” It shows up as: very little difference on routine Q&A and code completion, but a noticeable drop on multi-step scientific reasoning and problems requiring long chains of computation. This is two sides of the same phenomenon seen in the structural differences in the vendor’s self-reported data, discussed in the next section.

    5. How to read the vendor’s self-reported data: from “leading across the board” to “structural differentiation”

    The comparison data in the public model card (all vendor-reported) shows a clear split:

    Benchmark This model Closed-source flagship A Closed-source flagship B
    Agentic coding (terminal tasks) 65.7% 66.4% 58.2%
    Long-horizon software engineering tasks 51.7% 46.3% —
    Scientific agentic tasks 38.1% 63.3% —
    Hard knowledge reasoning (Diamond subset) 39.4% 60.9% —

    The correct conclusion: it is competitive on executable, verifiable agentic tasks, and shows a clear gap on knowledge- and reasoning-intensive tasks—not “across-the-board superiority.” Three reading principles:

    • Look at the evaluation harness, not just the score: agentic benchmarks are highly sensitive to prompts, tool definitions, retry strategies, and sandbox environments; the same model can differ by double digits across harnesses;
    • Look at sampling and judging methods: pass@k versus single-attempt pass, whether tool calls are allowed, and whether a repair opportunity is given all change the ranking;
    • Re-test with your own evaluation set: cover at least task success rate, token cost per successful task, and tail latency (P95/P99), and fix the version and random seed so you can run regression tests.

    6. License terms: four checkpoints for the additional conditions on open weights

    The essence of “two conditions” is open weights plus additional usage constraints. Verify each item before deploying:

    1. Commercial scope: whether commercial use is allowed, and whether a user-scale threshold triggers additional obligations;
    2. Redistribution and derivative models: whether distributing the model or fine-tuning it requires attribution, renaming, or source labeling;
    3. Output usage restrictions: whether there are explicit clauses prohibiting the use of generated content to train competing models;
    4. Compliance and export: content-safety obligations, cross-border data transfer, and model-provenance compliance review.

    It is a good idea to formalize the above into an internal “model admission checklist,” archived alongside the license, the model hash, and the weight source, to avoid disputes during later audits.

    7. An executable deployment checklist

    Hardware and topology

    • Start with 8 high-memory accelerators per node; for multi-node deployment, prioritize inter-card and inter-node interconnect bandwidth and avoid low-speed networks;
    • Reserve host memory as an offload pool for experts and the KV cache, with a recommended capacity of no less than 1/4 of the weight footprint.

    Parallelism strategy

    • Tune the combination of tensor parallelism and expert parallelism, choosing the configuration by measured throughput rather than theoretical FLOPs;
    • Watch the all-to-all communication share; when expert parallelism is set too high, its gains get eaten by communication.

    Serving optimizations

    • Continuous batching plus paged KV management to raise concurrency;
    • Prefix caching to reuse system prompts and long documents, which pays off significantly in agentic scenarios;
    • KV quantization, speculative decoding, and length-based request routing (short requests go to a smaller model or the cache).

    Weight preparation

    • Multi-shard releases need index-file and shard-integrity verification, with support for resumable downloads;
    • Pull weights via object storage to avoid tying up business bandwidth for long periods.

    Observability

    • Monitor GPU utilization, VRAM usage, KV cache hit rate, first-token latency, and OOM events;
    • In public cloud environments such as Baidu AI Cloud, container platforms typically host the distributed inference, object storage holds the weights, and monitoring, alerting, and logging services track capacity trends—so scaling decisions are built on metrics rather than experience.

    8. Run locally or not: a decision table

    Scenario Recommendation Reason
    Data cannot leave your domain, and tasks are mainly code/tool calls Build your own 8-GPU cluster Capability on this dimension is close to the closed-source flagship
    Knowledge-intensive or long-chain scientific reasoning is needed Be cautious about self-building; prefer multi-model routing Self-tests show a large gap on this dimension
    Million-token ultra-long context is needed Self-build, but stress-test the KV cache specifically The KV cache is the first bottleneck
    Highly variable peaks, and 8 GPUs cannot be held long-term Hosted service plus elastic scheduling Idle cost is high and hard to amortize locally
    The team lacks distributed-inference operations capability Start hosted, then evaluate self-building Expert parallelism and communication tuning have a high barrier to entry

    One-line summary: The value of these weights is bringing agentic capability close to the closed-source flagship to a position where you can self-host it. But the 1.56TB footprint, the 8-GPU floor, the single low-bit precision, and the additional license terms together form four real barriers. Before deploying, calculate the VRAM budget first, then verify the capability structure with your own evaluation set, and finally go through the license checklist—once these three steps are done, you will know whether it is an asset or a burden.