Local Deployment of Trillion-Parameter MoE LLMs: A Complete Breakdown

    Open weights make large models accessible, but "accessible" is not the same as "runnable." Once parameter counts reach the trillion (T) scale, weight files are measured in terabytes, and local deployment turns from a simple load operation into a systems-engineering effort centered on memory capacity, parallelism, and communication. This article uses a typical large sparse mixture-of-experts (MoE) model as its sample: roughly 2.8T total parameters, only 104B activated per token, 896 routed experts plus 2 shared experts, weights around 1.56TB in size, and inference that in practice needs 8 accelerators working together. Each of those numbers carries a technical constraint worth unpacking.

    From “downloadable” to “actually runnable”: an underrated gap

    Open weights have dramatically improved how accessible models are, but they are not the same as out-of-the-box usability. When parameter counts reach the trillion (T) scale, weight files are measured in terabytes, and local deployment stops being a simple load operation and becomes a systems-engineering effort organized around memory, parallelism, and communication.

    This article uses a typical large sparse mixture-of-experts (MoE) model as its sample:

    • roughly 2.8T total parameters;
    • only 104B parameters activated per token;
    • 896 routed experts plus 2 shared experts;
    • weights around 1.56TB on disk;
    • inference that in practice requires 8 accelerators working in concert.

    We’ll work through these numbers layer by layer to expose the constraints behind them.

    1. Total parameters vs. active parameters: two different budgets

    The first thing to understand about this class of models is the distinction between total parameters and active parameters.

    The core idea behind MoE (Mixture of Experts) is to split one very large feed-forward network into many parallel “expert” subnetworks, then let a router (or gating network) dynamically pick a small subset of experts for each token. For our sample model:

    • Total parameters: 2.8T, the sum of every weight in the model;
    • Active parameters: 104B, the portion that actually participates in the forward pass for a single token;
    • Structure: 896 routed experts, with 16 selected per token, plus 2 shared experts that always participate.

    This sparse activation leads to two key conclusions — and they point at completely different resource dimensions:

    1. Compute (FLOPs) is determined by active parameters. Each token only traverses a path through 104B parameters, so the compute cost of inference is far lower than that of a dense model with the same total parameter count. This is the elegant part of MoE: trading larger capacity for compute roughly comparable to a much smaller dense model.
    2. Memory footprint is determined by total parameters. Whether or not a given expert is selected by the current token, its weights must stay resident in GPU memory or system RAM, or the router cannot hit it. 2.8T of weights will not somehow fit into 104B worth of memory just because only 104B are activated.

    Many deployment failures trace back to confusing these two budgets: compute is sufficient, memory is nowhere close.

    2. The weight budget: where 1.56TB comes from

    Weight size isn’t a mystery. It’s determined by parameter count × bytes per parameter, where the byte count depends on quantization precision:

    Precision format Bytes per parameter Theoretical size of 2.8T parameters
    FP32 4 B ≈ 11.2 TB
    FP16 / BF16 2 B ≈ 5.6 TB
    INT8 1 B ≈ 2.8 TB
    INT4 / FP4 0.5 B ≈ 1.4 TB

    The sample model’s weights come in at around 1.56TB, well below FP16’s 5.6TB, which tells us the released weights use low-bit quantization (around 4-bit) while keeping a subset of tensors at higher precision — typically things like word embeddings, LayerNorm, shared experts, or the routing layers. That overhead is exactly what pushes 1.4TB up to roughly 1.56TB.

    The engineering lesson here is important: the storage precision of your weights is the first gate on whether the model fits in the memory you have. INT4 quantization cuts size to a quarter of FP16 and lowers the floor for what trillion-parameter models require; but quantization is not free — very low bit widths can cost accuracy, so quality needs to be validated on evaluation sets.

    3. Why “local” usually means 8 GPUs

    Even at 1.56TB, this is far beyond the memory capacity of any single accelerator. Deploying a model like this is fundamentally about satisfying one inequality:

    1. Aggregate available memory ≥ weights + KV cache + activations/intermediate tensors + framework and communication buffers

    Even if a single card offered hundreds of gigabytes of memory, 8 cards aggregated would only just cover the weights themselves. Add runtime overhead, and the KV cache — which grows linearly with context length and concurrency — easily becomes the last straw for available memory. The “8 cards” figure isn’t arbitrary: it’s the ceiling of (weight footprint + runtime memory) ÷ usable memory per card, then adjusted for sharding efficiency.

    Spreading the model across multiple cards requires combining several parallelism strategies:

    • Tensor parallelism (TP): shards individual layer matrices across dimensions on multiple cards, requiring frequent All-Reduce between cards — so it depends heavily on inter-card interconnect bandwidth, such as a high-speed fabric.
    • Expert parallelism (EP): distributes the 896 experts across different cards, a natural fit for MoE structure, but introduces All-to-All communication.
    • Pipeline parallelism (PP): splits by layer, reducing per-card memory pressure, at the cost of pipeline bubbles.

    Very large-scale MoE deployments typically mix all three. The finer the sharding, the lower the pressure per card — but the higher the communication overhead and implementation complexity.

    4. Hidden costs in MoE inference: routing and full residency

    Beyond memory, MoE inference carries two costs that are easy to overlook:

    1. Routing and load imbalance. The router assigns experts for every token. If certain “hot” experts get picked by a large share of tokens, you get localized overload while experts on other cards sit idle, dragging down compute utilization. In practice this is mitigated with mechanisms such as auxiliary load-balancing loss and capacity factor.
    2. All-to-All communication. With expert parallelism enabled, tokens must be sent to the card hosting their target expert, computed, and the results gathered back. The volume of this exchange varies with expert distribution and batch composition, and it’s often the bottleneck in distributed inference. High-speed interconnect and sensible overlap of communication with computation are the keys to extracting throughput.

    Put another way: MoE’s sparseness saves compute — not memory, and not communication. All three have to be considered together when evaluating a deployment plan.

    5. Deployment decision checklist: precision, parallelism, hardware

    For real-world rollouts, here’s a practical order of decisions:

    1. Decide precision first. Compare FP16 against INT4/INT8 on your evaluation sets, then choose the lowest precision that fits the hardware.
    2. Then do the memory math. Estimate aggregate memory using the formula above, leaving headroom for the KV cache and concurrency (typically an extra 20–30% buffer).
    3. Then pick a parallelism strategy. If inter-card interconnect is strong, favor tensor parallelism; introduce expert parallelism when expert count is high; layer on pipeline parallelism only when memory is genuinely tight.
    4. Finally, load-test and tune. Track time-to-first-token (TTFT) and throughput, and watch for load imbalance and communication bottlenecks.

    If you’re running on cloud resources, general-purpose cloud capabilities can take a lot of operational weight off your shoulders: a container platform to orchestrate multi-card inference services with elastic scaling, object or file storage to hold TB-scale weights, monitoring and logging services to track memory usage, latency, and throughput, and load balancing to distribute requests across replicas. Platforms such as Baidu AI Cloud typically offer these general capability categories, which makes it easier to fit large-model inference into a standardized deployment and observability setup.

    6. Illustrative deployment configuration (pseudocode, not a specific product)

    The configuration below shows how the different dimensions fit together. Values need to be adjusted to the actual hardware:

    1. model:
    2. architecture: "MoE"
    3. total_params: "2.8T"
    4. active_params: "104B"
    5. num_routed_experts: 896
    6. experts_per_token: 16
    7. num_shared_experts: 2
    8. weight_precision: "INT4" # the key factor in weight file size
    9. weight_file_size: "~1.56TB"
    10. runtime:
    11. parallelism:
    12. tensor_parallel_size: 8 # intra-layer sharding
    13. expert_parallel_size: 8 # expert distribution
    14. pipeline_parallel_size: 1
    15. kv_cache:
    16. dtype: "fp16" # share grows with context length
    17. max_context_length: 32768
    18. max_concurrency: 8
    19. buffer:
    20. reserved_ratio: 0.25 # reserved memory to avoid OOM

    Conclusion

    Local deployment of trillion-parameter MoE models has never been hard because of downloading. The difficulty lies in finding a balance point across four dimensions: total parameters, memory, compute, and communication. Keep one line in mind — total parameters set memory, active parameters set compute, precision sets file size, parallelism sets communication — and you can quickly estimate the hardware requirements for any large open-weight model, choose an appropriate quantization and sharding scheme, and use standardized cloud deployment and observability capabilities to turn “it won’t run” into “it runs reliably and economically.”