Local Deployment of Trillion-Parameter MoE Models: VRAM Requirements and an 8-GPU Setup Explained

    Deploying a trillion-parameter mixture-of-experts (MoE) model locally is far more complex than scaling hardware linearly with parameter count. This article breaks down the key numbers: the difference between total and activated parameters, how a 2.8T-parameter model with 104B active parameters per token translates into compute and memory requirements, what 1.56TB of weights means across 8 GPUs, and which parallelism strategies—tensor parallelism, expert parallelism, pipeline parallelism, and offloading—make local deployment feasible. It also covers practical engineering points for serving, including KV cache estimation and batch scheduling.

    1. Why Very Large Models Need a Separate Deployment Assessment

    Once parameter counts reach the trillion scale, deployment difficulty no longer grows linearly with the number of parameters. Weight size determines the VRAM floor, activated parameters determine the compute cost of each inference pass, inter-expert communication determines multi-GPU scaling efficiency, and concurrent request volume determines the trade-off between throughput and latency. Judging hardware requirements by parameter count alone easily leads to over- or under-estimation.

    According to the source material, this model is a natively multimodal agent model, further trained from its predecessor. It has 2.8T total parameters, activates 104B parameters per token, and uses a mixture-of-experts (MoE) architecture. The rest of this article works through the key calculations involved in deployment based on these publicly disclosed figures.

    2. Total Parameters vs. Activated Parameters: What Each Number Determines

    Total parameters determine how much storage and VRAM the weights require. Whether or not a given expert is used during inference, its parameters must reside somewhere—GPU memory, CPU memory, or disk. Total parameters are therefore a hard constraint on deployment resources.

    Activated parameters determine how much computation each token actually involves. The activation ratio of this model is:

    • Activated / total = 104B / 2.8T ≈ 3.7%

    This means the forward-pass compute per token is far lower than a dense model of the same total size. For rough estimation, a common approximation is about 2 FLOPs per parameter, giving roughly 2 × 104 × 10⁹ = 2.08 × 10¹¹ FLOPs per token—about 208 GFLOPs. This figure excludes attention overhead, communication, and memory traffic, so it serves only as an order-of-magnitude reference.

    Note that “activated parameters” does not mean “only activated experts run per layer.” Attention layers, embeddings, and shared experts typically execute for every token, so the 104B includes all components that always participate in computation.

    3. The MoE Architecture: 896 Experts, 16 Routed per Token, 2 Shared Experts

    In a mixture-of-experts architecture, the feed-forward network is split into multiple expert sub-networks, and a router selects a subset of experts for each token. Per the source material, this model has 896 experts, selects 16 routed experts per token, and additionally has 2 shared experts.

    The routing process can be illustrated with pseudocode:

    1. def route(hidden, router_weight, top_k=16):
    2. # hidden: hidden state of a single token at one layer
    3. logits = matmul(hidden, router_weight) # shape [896]
    4. probs = softmax(logits)
    5. topk_idx = argsort_desc(probs)[:top_k] # top 16 experts by score
    6. weights = normalize(probs[topk_idx]) # normalized gating weights
    7. return topk_idx, weights
    8. def moe_forward(hidden, router_weight, routed_experts, shared_experts):
    9. topk_idx, weights = route(hidden, router_weight)
    10. out = sum(shared(hidden) for shared in shared_experts) # shared experts always run
    11. for i, w in zip(topk_idx, weights):
    12. out += w * routed_experts[i](hidden) # only selected experts run
    13. return out

    This code illustrates the logic only and is not an actual implementation. It highlights several engineering implications.

    First, the share of routed experts is small. Per layer, 16 out of 896—about 1.8%—of routed experts participate in computation, which is the source of the compute savings from sparsity.

    Second, shared experts handle general-purpose features and process every token, so their quality directly affects all inputs.

    Third, routing carries a load-imbalance risk. If certain experts are consistently selected, the GPUs hosting them become bottlenecks. Training typically adds load-balancing constraints; at inference time, you need to monitor the actual invocation frequency of each expert.

    Fourth, when experts are distributed across multiple GPUs, each layer must send tokens to the devices hosting the selected experts and collect the results. This all-to-all communication is sensitive to interconnect bandwidth and latency.

    4. The VRAM Math Behind 1.56TB

    Using decimal units, the weight sizes of 2.8T parameters at different precisions are:

    1. total_params = 2.8e12
    2. for name, bits in [("BF16", 16), ("FP8/INT8", 8), ("INT4", 4)]:
    3. size_tb = total_params * bits / 8 / 1e12
    4. print(f"{name}: {size_tb:.2f} TB")
    5. # BF16: 5.60 TB
    6. # FP8/INT8: 2.80 TB
    7. # INT4: 1.40 TB

    The source material states the weight size as 1.56TB. Converting, the average bits per parameter are approximately 1.56 × 10¹² × 8 / 2.8 × 10¹² ≈ 4.46. This falls between 4-bit and 8-bit, indicating the weights are most likely low-bit quantized, with metadata such as scaling factors included in the quantization format, and some layers possibly kept at higher precision. The exact distribution should be confirmed against the released file manifest—do not infer it from the total size alone.

    Now consider the 8-GPU claim. Splitting 1.56TB evenly across 8 GPUs, each GPU must hold about 1.56 × 10¹² / 8 ≈ 1.95 × 10¹¹ bytes, or roughly 195GB. This does not yet account for:

    • KV cache: grows linearly with context length and concurrency, and can be very large for long-context workloads.
    • Activations and temporary buffers: memory needed for intermediate inference results.
    • Framework and communication buffers: varies significantly across implementations.

    If each GPU has 80GB of VRAM, 8 GPUs provide 640GB in total—far from enough to hold the weights. Therefore, “running locally on 8 GPUs” is only feasible under one of the following conditions: each GPU has roughly 200GB or more of VRAM; more GPUs are used; or weights and caches are offloaded to CPU memory or local disk, which significantly increases latency. Before planning hardware, confirm the VRAM capacity of the GPU model, verify the actual weight file sizes, and reserve space for the KV cache.

    5. Deployment Paths and Parallelism Strategies

    Very large models typically require a combination of parallelism methods. The table below compares common strategies:

    Strategy What it splits Main benefit Main cost
    Tensor parallelism (TP) Matrices within a single layer Reduces per-GPU weight footprint Frequent intra-layer communication; requires high-speed interconnect
    Expert parallelism (EP) MoE experts Distributes experts across devices, reducing per-GPU pressure All-to-all communication; hotspots under load imbalance
    Pipeline parallelism (PP) Layers into stages Scales to more nodes Pipeline bubbles add latency
    Data parallelism (DP) Full model replicas Improves throughput No VRAM savings; weights fully duplicated
    Offloading Weights or caches to CPU memory or disk Reduces GPU count requirements Latency and bandwidth constrained; high variance

    For a single 8-GPU node, the common approach is to combine tensor parallelism with expert parallelism within the node, assuming sufficiently high interconnect bandwidth. When scaling across nodes, the communication overhead of pipeline and expert parallelism rises quickly, and throughput targets must be re-evaluated.

    6. Engineering Points for Inference Serving

    6.1 KV Cache Estimation

    The KV cache size can be approximated with the following formula:

    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    2. # The factor of 2 accounts for Key and Value
    3. return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem

    The number of layers, KV heads, and head dimension must be read from the model configuration file—do not estimate them from experience. When long context and high concurrency occur together, the KV cache often exhausts VRAM before the weights do. It is recommended to run a separate estimate for your target concurrency and maximum context length.

    6.2 Batching and Scheduling

    Continuous batching allows requests of different lengths to be merged dynamically, reducing GPU idle time. For MoE models, larger batches mean each expert may receive more tokens, improving compute utilization—but routing load also becomes more concentrated. During load testing, monitor both throughput and the load distribution across experts.

    6.3 Quantization and Quality