Trillion-Parameter MoE Open-Weight Deployment: Calculating VRAM, Throughput, and License Feasibility

    When a trillion-parameter MoE model releases its weights, the first questions are always about hardware and rights: will it fit on 8×141GB GPUs, are 16 cards really necessary, and how do you verify commercial licensing? Using a public model as a worked example—2.8T total parameters, 104B activated per token, 896 experts, 93 layers, and a 1M-token context—this article walks through the five dimensions that determine whether such a model can move into production: weights, VRAM, throughput, licensing, and evaluation. You will get concrete formulas for weight volume and KV cache, a memory budget that explains the 8-card shortfall and the 16-card floor, a bandwidth-bound throughput model, a practical approach to checking license terms, and a deployment self-check list. Note that the weight calculations here use MXFP4 as the theoretical baseline; actual downloaded packages are often larger, so always size deployment against the real weight files.

    When a trillion-parameter MoE model releases open weights, readers ask the same three questions: Will it fit on 8×141GB GPUs? Are 16 cards necessary? How do you verify commercial use rights? This article uses a public model as a worked example—2.8T total parameters, 104B activated per token, 896 experts, 93 layers, and a 1M-token context—to show how to judge production readiness across five dimensions: weights, VRAM, throughput, licensing, and evaluation.

    A note on the weight figures before we start: the calculations below use MXFP4 as the theoretical baseline. Here, 2.8T parameters are estimated at 4.25 bits per parameter, giving roughly 1.49TB (about 1388GiB). The actual download package may be larger because of higher-precision tensors such as embedding and normalization layers, so always size deployment against the real weight file volume.

    1. Weights and Precision: Start With Size and Precision Ceiling

    The core idea of an MoE model is to raise capacity through a large pool of expert parameters, while each token activates only a small subset of experts. In the example model, each token is routed to 16 experts, plus 2 shared experts that are always active. That gives roughly 104B activated parameters, an activation rate of 104B ÷ 2.8T ≈ 3.7%.

    In practice, this means more than 96% of parameters take no part in a single forward pass, which is friendly to compute. However, the inference engine cannot predict which experts the next token will hit, so it usually keeps all expert weights resident in VRAM or immediately readable. VRAM and communication pressure are therefore significantly higher.

    For MXFP4 weights, each element uses 4 bits, and every 32 elements share one 8-bit scaling factor. The effective cost per parameter is 4 + 8/32 = 4.25 bits, or about 0.531 bytes. This gives a weight volume of roughly:

    2.8 × 10¹² × 4.25 ÷ 8 ≈ 1.49 × 10¹² bytes ≈ 1.49TB

    For comparison:

    • Stored as FP8: about 2.8TB
    • Stored as BF16: about 5.6TB

    Low-precision weights make downloading practical, but they also bring limitations:

    • Without an original-precision version, it is hard to assess quantization loss.
    • Older hardware may lack native MX kernel support and must run after dequantization, which raises VRAM usage.
    • Fine-tuning relies more heavily on dequantization or low-rank adaptation (LoRA) paths.

    If the official example code is tied to a specific runtime or parallelism scheme, third-party inference engines need to re-validate expert routing, quantization kernels, tensor parallelism, and KV management.

    2. The VRAM Ledger: Why 8 Cards Fall Short and 16 Is the Floor

    A VRAM budget cannot look at weights alone. It should cover at least:

    • Weights
    • KV cache
    • Activation and communication buffers
    • Framework overhead

    KV cache grows linearly with context length and is the main variable for long-context serving. The estimation formula is:

    KV cache = 2 × num_layers × num_kv_heads × head_dim × sequence_length × bytes_per_element

    Using 93 layers, 8 KV heads, and a head dimension of 128, a single request’s KV cache is roughly:

    Context Length FP16 FP8
    10K tokens ~3.5GiB ~1.8GiB
    100K tokens ~35.4GiB ~17.7GiB
    1M tokens ~354.8GiB ~177.4GiB

    Even with the KV cache quantized to FP8, a single request that fills a 1M-token context exceeds a single 141GB card. It must be split across multiple cards via tensor parallelism.

    Can 8×141GB hold it?

    • Total memory: 8 × 141 = 1128GiB
    • Against ~1388GiB of weights (the 1.49TB figure), the shortfall is about 260GiB, roughly 19% of the weight footprint.
    • If the actual download package is larger, the gap only grows.

    So 8×141GB cannot hold the full weights on standard datacenter accelerators, unless a single card exceeds 200GB, or the framework can offload some experts to host memory.

    What 16 and 32 cards give you

    • 16×141GB: total 2256GiB. After reserving 10% for framework overhead, about 2030GiB remains. Fitting 1388GiB of weights leaves roughly 642GiB to cover some KV cache, activations, and communication buffers—suitable for a single request or limited concurrency.
    • 32×141GB: total 4512GiB. After the reserve, about 4061GiB remains, leaving roughly 2673GiB—better suited to multiple requests, long context, and higher concurrency.

    The takeaway: with 141GB single cards, 16 cards is a realistic lower bound. Sixteen 141GB cards usually will not fit in one ordinary server, so you often need two nodes connected by a high-speed interconnect. If you insist on single-machine deployment, choose accelerators with 192GB or more per card. If you later switch to FP8 weights (~2.8TB), 16 cards will usually not be enough, and you should recompute the budget around precision, dequantization, and offloading strategy.

    3. Decode Throughput: The Bottleneck Is Memory Bandwidth, Not Paper Compute

    MoE saves some compute during prefill, but in the decode phase, generating each token typically reads the full set of activated weights once. For the example model, ~104B activated parameters at MXFP4 (about 0.531 bytes including scaling factors) means roughly 55GB of weights read per token.

    The compute is about 2 × 104B = 208GFLOP, giving an arithmetic intensity of roughly 208GFLOP ÷ 55GB ≈ 4 FLOP/Byte—far below the GPU compute-to-bandwidth ratio. This is a classic bandwidth-bound workload.

    The decode throughput ceiling can be written as:

    throughput ≈ N × B × η ÷ 55 (in tokens/s)

    where N is the number of deployed cards, B is single-card memory bandwidth in GB/s, η is effective bandwidth utilization, and 55GB is the activated weight volume read per token.

    For a 16-card deployment:

    • Single-card bandwidth 3.35TB/s: ~490 tokens/s at 50% efficiency, ~780 tokens/s at 80% efficiency
    • Single-card bandwidth 7.68TB/s: ~1120 tokens/s at 50% efficiency, ~1790 tokens/s at 80% efficiency

    Plugging 8 cards into the formula gives a theoretical value of roughly half the 16-card figure, but 8 cards usually cannot meet the VRAM budget, so it is not suitable for production capacity planning.

    Real throughput must also be multiplied by the expert-parallel load-balance rate and reduced by All-to-All communication overhead. Cross-node MoE communication costs are especially noticeable; cross-machine efficiency is often only 60–70% of what you get within a single machine.

    Optimization directions include:

    • Keep expert parallelism within the node first; use hierarchical All-to-All across nodes.
    • Apply KV quantization, paged caching, and prefix-cache reuse.
    • Use sparse attention and chunked prefill for long context.
    • Separate Prefill and Decode deployment so compute-intensive and bandwidth-intensive workloads can scale independently.

    Multimodal input also produces large numbers of image tokens, which require extra budget for encoder memory, tokens per image, and long-context KV cost.

    4. Licensing: Open Weights Are Not the Same as Open Source

    Open-source licenses typically grant freedom to use, research, modify, and redistribute. Open weights only mean the weights can be downloaded and inspected; this does not automatically mean commercial use, modification, or redistribution is allowed.

    Custom licenses commonly include terms such as:

    • Thresholds for ultra-large-scale commercial use
    • Attribution obligations
    • Naming and license inheritance rules for derivative models
    • Prohibited uses such as military or large-scale surveillance
    • Distillation restrictions
    • Liability disclaimers

    To decide whether you can use a model, rely on the license text itself, not on marketing language. Before going live, you can run a “term diff”: normalize volatile fields such as version numbers and product names across the old and new license, then compare. If only names changed and the substance is unchanged, you can reuse existing compliance conclusions. If there are substantive changes—a tightened commercial threshold, a new distillation restriction, a new geographic restriction—bring in legal review.

    5. Evaluation and a Pre-Deployment Checklist

    Vendor-reported benchmarks are only useful for initial screening and cannot replace business-level validation. When reading these tables, ask four questions:

    • Is the result reproducible?
    • Is there possible contamination from training data?
    • Does a one- or two-point difference exceed the noise level?
    • Are dimensions like long context, multi-turn tool calls, code executability, and domain reasoning missing?

    A more reliable approach is to build your own mini evaluation set:

    • Code fixes: measure pass@1
    • Multi-step tool calls: track task completion rate and average number of steps
    • Long-document QA: measure accuracy
    • Domain reasoning: use manual or automated evaluation

    Before deployment, run through this checklist:

    • Does the license allow commercial use, redistribution, and distillation?
    • Is the weight volume a theoretical value or the actual file value?
    • Does the VRAM budget cover weights, KV cache, activation/communication buffers, and framework overhead?
    • Do the card count and interconnect support intra-node expert parallelism?
    • Is storage provisioned for multiple replicas, hot standby, and loading buffers?
    • Has decode throughput been recalculated using single-card bandwidth and cross-machine efficiency?
    • Does long context have KV quantization and a concurrency limit?
    • Is an original-precision weight available, or an acceptable fine-tuning path?
    • Can you reproduce key metrics with your own evaluation set?
    • After launch, do you monitor load time, decode throughput, expert load, and VRAM utilization?

    Overall, the value of trillion-parameter MoE lies in pushing the capability frontier higher, but whether it lands in production depends on the engineering ledger. Calculate VRAM and throughput with the formulas first, confirm the compliance boundary with the license terms, then validate results with your own evaluation set—only then can “open weights” become genuinely usable production capability.