A Practical Guide to Trillion-Parameter MoE LLM Inference Deployment
This article provides a practical guide to deploying trillion-parameter Mixture-of-Experts (MoE) large language models for inference. It explains the engineering trade-offs behind common deployment choices, including GPU memory budgeting, expert parallelism, data parallelism, FP8 KV cache, paged attention block size, runtime requirements, model-specific parsers, single-node deployment, Prefill/Decode disaggregation, API calls, and production checklists. The goal is to help developers understand why key parameters are configured in a certain way and how to adapt the approach to other models.
A newly released deployment guide for a next-generation MoE large language model breaks down how to run a trillion-parameter model on limited hardware into a set of directly executable instructions. The most valuable part is not the parameter list itself, but the engineering trade-offs behind it:
- Why GPU memory budget is decisive;
- Why expert parallelism and data parallelism need to be combined;
- Why the KV cache is reduced to FP8 and why the paged attention block size is fixed;
- Why Prefill and Decode are split into two services and forwarded by an independent routing layer.
Using this guide as a starting point and combining it with common engineering experience for LLM inference services, this article systematically outlines the full path from feasibility assessment to single-node deployment, then to disaggregated deployment and API calls. It helps developers understand why each critical parameter is set in a particular way, so they can make independent judgments and migrate the approach to other models.
1. Why trillion-parameter MoE deployment differs from dense models
The Mixture-of-Experts (MoE) architecture uses a gating network to select a small number of experts for each token, so the total parameter count and the number of parameters activated during each inference step can differ significantly. The new guide provides two model sizes: a flagship model with around 1.6 trillion total parameters and a lightweight model with 284 billion total parameters.
- Memory is determined by total parameters, while compute is determined by activated parameters. Even if only a small portion of experts is activated per step, all expert weights must reside in GPU memory or be fetched quickly enough.
- Expert parallelism requires global communication. Once experts are split across GPUs, each token must first be routed to the GPU holding the corresponding expert and then gathered back after computation. One forward pass often requires two All-to-All communication rounds.
- Long chain-of-thought reasoning causes the KV cache to grow rapidly. Reasoning models may generate long intermediate thoughts, and long contexts make the cache grow linearly with sequence length.
This means the inference bottleneck shifts from raw compute to GPU memory capacity plus inter-GPU bandwidth. Memory determines whether the model can run and how much concurrency it can support. Bandwidth determines how fast it runs. The first step is therefore to calculate the memory budget.
2. Three thresholds before deployment: calculate first, then deploy
2.1 Threshold 1: Is there enough GPU memory?
GPU memory usage can be broken down into four parts:
GPU memory requirement ≈ model weights + KV cache + runtime overhead + activations and temporary buffers
- Weights: Weight bytes ≈ total parameters × bytes per parameter (2 bytes for BF16, 1 byte for FP8).
- KV cache: Per-token cache bytes ≈ 2 (K and V) × number of layers × number of KV heads × head dimension × bytes per element. When the architecture uses low-rank compression, the actual layout should be used for estimation instead of the standard multi-head attention layout.
- Runtime overhead: GPU memory fragmentation, temporary buffers, and memory used by compute graph capture such as CUDA Graph.
- Activations and communication buffers: Buffers introduced by batch size and expert-parallel All-to-All communication.
Using FP8 weights for a rough estimate:
- The flagship model’s weights are about 1.6 TB. An 8-GPU node with 80 GB per GPU has only 640 GB total memory, which is not enough to hold the weights themselves. Multi-node deployment is therefore required.
- The lightweight model’s weights are about 280 GB. This type of node becomes very tight once runtime overhead is included and usually requires 16 GPUs. An 8-GPU node with 141 GB per GPU, providing about 1128 GB, is a safer choice.
- Therefore, the guide defines the simplest deployment of the lightweight model as a single node with four high-end accelerator cards that support native FP8 computation. With more than 1 TB of total memory across four cards, both weights and cache can be accommodated comfortably.
Converting this into concurrency. When the model has 128 routed experts, activates 6 experts per token, and uses an expert intermediate dimension of 2048, each token activates about 1.5 million parameters, and the compute per step is about 3 GFLOPs. From this, we can infer that during Decode, generating each token requires moving about 300 KB of expert weights from GPU memory.
This allows us to estimate the theoretical single-GPU output speed limit under different inter-GPU interconnect types:
| Inter-GPU interconnect type (indicative bandwidth) | Single-GPU theoretical upper bound | Two-GPU tensor parallelism theoretical upper bound |
|---|---|---|
| High-bandwidth interconnect, about 900 GB/s | About 3000 tokens/s | About 6000 tokens/s |
| PCIe, about 64 GB/s | About 210 tokens/s | About 430 tokens/s |
It should be emphasized that this is a theoretical upper bound considering only weight reading. Actual performance is also affected by gating overhead, communication overhead, and GPU utilization. However, the order-of-magnitude difference still makes the point clear: hardware without high-bandwidth inter-GPU interconnect makes it difficult to provide smooth streaming output for a trillion-parameter MoE model. Whether tensor parallelism should be used should be determined by comparing actual measurements against the model’s native parallelism strategy.
2.2 Threshold 2: Runtime and deployment form
Deploying a trillion-parameter MoE model usually requires a relatively new GPU runtime and inference engine images compiled specifically for that hardware generation and that compute precision. The hardware generation, runtime, image, and model weights must correspond strictly. Building from source may take several days and can encounter operator compatibility issues.
For deployment form, the simplest option given by the guide is a containerized single-node service: one command can pull the image, load the model, start the service, and expose an HTTP port compatible with a general Chat Completions-style API. When exposing the service, authentication should be enabled by default and the listening address should be restricted.
2.3 Threshold 3: Model-specific components
A new model usually requires new supporting components. This is one of the most easily overlooked deployment thresholds:
- Dedicated tokenizer: Used to correctly handle the vocabulary, chat template, and special tokens;
- Tool call parser: Used to parse tool calls from model output;
- Chain-of-thought parser: Used to separate reasoning content from the final response;
- Automatic tool choice switch: Automatically changes the output mode when tool candidates are present.
These components are tightly coupled with the model’s chat template, special-token conventions, and output format. Parsers from older models are often incompatible. If they are reused, three types of failures may occur:
- The reasoning path fails, and the model does not output reasoning;
- Tool calls cannot be parsed into structured arguments;
- Multi-turn conversation history formatting becomes incorrect.
3. Single-node deployment: key parameters explained
First, look at the simplest startup parameters for the lightweight model on four GPUs (illustrative):
# Illustrative: exact flag names depend on the inference engine version documentation<inference-engine-image> <model-identifier> \--kv-cache-dtype fp8 \--block-size 256 \--enable-expert-parallel \--data-parallel-size 4 \--tokenizer-mode <model-specific-tokenizer-mode> \--tool-call-parser <model-specific-tool-parser> \--reasoning-parser <model-specific-reasoning-parser> \--enable-auto-tool-choice \--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
| Parameter category | Function | Tuning considerations |
|---|---|---|
| Parallelism configuration | --enable-expert-parallel + --data-parallel-size 4, with one data-parallel replica per GPU |
Expert parallelism splits expert weights across four GPUs; dense parts such as attention can use tensor parallelism across four GPUs |
| KV cache precision | --kv-cache-dtype fp8 |
Halves memory usage compared with BF16, allowing longer contexts and higher concurrency |
| Paging granularity | --block-size 256 |
Block indexing is more compact, but must match quantization granularity and the attention kernel implementation |
| Compilation optimization | Custom operators for all operators + full-graph and piecewise graph capture | First ensure that model operators can be compiled correctly, then try different graph capture strategies |
The flagship model has the same form, except that --data-parallel-size is changed to 8, running on an 8-GPU high-end accelerator node.
The following sections explain each point in detail.
3.1 How expert parallelism and data parallelism work together
The non-expert parts of an MoE model, such as attention and embedding layers, are computed densely over all tokens, while the expert parts are activated sparsely according to routing results. These two types of computation are suitable for different parallelism strategies.
- Expert parallelism distributes experts across GPUs, reducing per-GPU weight pressure at the cost of introducing All-to-All communication.
- Data parallelism allows each replica to hold complete non-expert parameters and handle different requests, making horizontal scaling easier.
- When the two are combined, attention inside a replica can optionally use tensor parallelism to spread activation computation across multiple GPUs.
How can you tell whether the parallelism configuration is reasonable? With a fixed number of GPUs, two indicators can be observed:
- Per-GPU weight usage: If expert parallelism is not effective, every GPU will hold all experts and GPU memory will be exhausted quickly.
- GPU utilization during Decode: If it is clearly low, All-to-All communication is usually the bottleneck. In this case, check whether high-bandwidth inter-GPU interconnect is enabled and whether topology affinity is correct.
3.2 Why the KV cache uses FP8 directly
Changing cache precision from BF16 to FP8 directly reduces GPU memory usage by half. With the same amount of GPU memory, this can support longer contexts, larger batch sizes, or both.
However, the impact of lower precision on model output cannot be generalized. The same deployment recipe was evaluated separately for the lightweight and flagship models, and the two model sizes did not behave identically across inference effort levels. This means precision configuration must be validated with the specific model and business evaluation sets. It is not safe to simply assume that a larger model can tolerate lower precision better.
3.3 Block size: key to block-based addressing
PagedAttention-style paged attention mechanisms draw on the idea of operating-system virtual memory:
- Split the KV cache of each sequence into fixed-size blocks;
- Use a block table to map logical positions to physical blocks.
Larger blocks make the addressing data structures more compact, and K/V values within the same block are contiguous in memory, which helps vectorized reads.
However, for models that use low-rank compressed cache structures, the number of blocks directly determines the size of the addressing structure. Larger blocks mean fewer blocks and narrower address widths, reducing two types of overhead:
- Memory access: reading the addressing structure itself;
- Computation: bit operations during decoding.
This is why the guide treats block size as a hard requirement. It is not a performance parameter that can be tuned arbitrarily. It is a contract bound to:
- The model cache layout;
- Quantization granularity;
- The attention kernel implementation.
When these three do not match, the result can range from runtime errors to severe precision anomalies.
4. Prefill/Decode separation: run two types of workloads separately
The lightweight model also has a configuration that is more suitable for strict latency requirements: on a high-memory single node, four GPUs run Prefill and four GPUs run Decode, KV cache is transferred over an RDMA high-speed network, and an independent routing layer handles request scheduling. This configuration is also supported on systems with eight GPUs.
4.1 Why separate Prefill and Decode
LLM inference has two stages:
- Prefill: Processes the entire input at once, is compute-intensive, and determines time to first token (TTFT);
- Decode: Generates tokens one by one, is memory-access-intensive, and determines time per output token (TPOT).
When the two are colocated, long-input Prefill can repeatedly interrupt Decode and increase the tail of output latency, such as P99.
Prefill/Decode disaggregation separates the two types of workloads onto different instances. Each can choose its own parallelism strategy and batching strategy, making performance characteristics more controllable and less interfering with each other.
The cost is KV cache transfer. After Prefill completes, the cache must be sent to the Decode instance, which is an additional overhead:
- The transfer volume is approximately equal to “input tokens × per-token cache bytes”;
- If only intra-node interconnect is used, intra-node separation is the cost-effective option;
- Cross-node separation generally requires a combination of intra-node separation, high-bandwidth inter-node networking, and topology-aware scheduling.
4.2 Illustrative deployment and routing
# Illustrative: parameter names depend on the routing tool version documentation<routing-tool> \--policy round_robin \--prefill http://localhost:8000 \--decode http://localhost:8001 \--host 127.0.0.1 \--port 30000 \--intra-node-data-parallel-size 4 \--kv-connector <RDMA-transfer-backend>
There are three key routing-layer configurations:
- Scheduling policy:
round_robinis the simplest to implement and is sufficient for initial link validation; - Disaggregated endpoints: Explicitly declare the addresses of the Prefill and Decode services;
- KV transfer backend: Choose an RDMA transfer backend and declare the intra-node parallelism size so that the scheduler can be topology-aware.
Callers only access port 30000 on the routing layer and are unaware of the separation details.
This approach has noticeably higher deployment and operational complexity:
- Independent RDMA device access and permissions are required;
- Metrics, logs, and scaling policies must be separated for the two instance types;
- End-to-end latency must be attributed by stage so that bottlenecks can be located in Prefill, transfer, or Decode.
For selection, a three-step approach is recommended:
- First use a single-machine multi-GPU configuration to validate functionality and accuracy baselines;
- Then enable PD separation and observe the composition of end-to-end latency;
- Only when Prefill interference with Decode is confirmed to be the main bottleneck should the additional operational cost of disaggregated deployment be accepted.
5. Inference effort levels and API calls
The Flash series provides three inference effort levels: Non-think, Think High, and Think Max. These correspond to “no reasoning output,” “output reasoning of a certain length,” and “output full reasoning,” respectively. For callers, this means that the same API and the same model can be used to dynamically adjust the “reasoning budget” according to task difficulty.
5.1 Illustrative request
import jsonimport urllib.requestpayload = {"model": "<model-identifier>","messages": [{"role": "user", "content": "What is 17*19? Return only the final integer."}],"chat_template_kwargs": {"thinking": True,"reasoning_effort": "max",},}req = urllib.request.Request("http://localhost:8000/v1/chat/completions",data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json"},)with urllib.request.urlopen(req, timeout=600) as resp:result = json.load(resp)print(result["choices"][0]["message"])
This code relies only on the Python standard library. Most mainstream inference engines provide an HTTP service compatible with a general Chat Completions-style API. Other HTTP clients or native SDKs can also be used.
5.2 What to pay attention to when calling the API
- Context window: The Think Max level requires
--max-model-lento be at least 393216. Higher output effort and longer reasoning require a larger reserved window and a correspondingly larger KV cache budget. In addition, this value must be no smaller than the sum of input and maximum output; otherwise, requests will be rejected. - Long-lived connections: Deep reasoning may last for tens of seconds or longer. Client timeouts should not use settings intended for ordinary requests. When integrating with a gateway, idle timeout settings should also be increased accordingly.
- Separated output fields: After chain-of-thought parsing is enabled, reasoning content and the final response belong to different fields. In streaming scenarios, they should be handled separately: first stream the reasoning process, then show the final answer. Otherwise, both the user experience and output assembly can suffer.
- Task routing: Assign inference effort according to task difficulty. Simple question answering can use a lower level, while complex reasoning should use a higher level, balancing latency, throughput, and cost.
6. From validation to scale: deployment checklist
Getting a single node working is only the starting point. To stably serve business workloads, the following areas must also be covered.
Pre-launch validation:
- Functionality: Conversation, streaming output, tool calls, chain-of-thought separation, and multi-turn history.
- Accuracy: Use business-specific evaluation sets to test each inference effort level separately. When the KV cache uses FP8, separately regression-test long-context tasks.
- Performance: Run stepped concurrency tests with different input/output length combinations. Record TTFT and TPOT separately, as well as KV cache usage, preemption counts, and queuing time.
- Stability: Run long-duration continuous load tests and observe whether GPU memory slowly increases and whether very long requests trigger preemption or rejection.
Runtime observability:
- Metrics: Request volume, error rate, P50/P95/P99 latency, token throughput, KV cache usage, and queue length;
- Logs: Record inference effort level, input and output token counts, latency, and tool call structures output by parsers;
- Alerts: Cover high KV cache watermark, latency threshold breaches, and sudden increases in error rate.
Scaling deployment:
- Model weights are centrally managed in object storage and pulled to compute nodes on demand;
- Inference services run on a container platform and scale by replica count rather than by single-GPU count;
- Log services are used for request auditing and problem tracing;
- Monitoring and alerting are used to uphold the SLA.
7. Common pitfalls checklist
- Hardware mismatch: Forcing a trillion-parameter MoE configuration onto hardware that lacks native FP8 compute or high-bandwidth interconnect.
- Parser mixing: Loading a new model with parsers from an older model, causing reasoning or tool calls to fail.
- Changing block size as an “optimization”: Modifying block size arbitrarily for “tuning” and breaking the contract with quantization granularity and the attention kernel.
- Inference effort and context window mismatch: Using the Think Max level without increasing
--max-model-len, causing requests to be rejected or reasoning to be truncated. - Using default timeout settings: Gateways or clients disconnect before reasoning completes.
- Copying precision conclusions from another model: Applying evaluation results from one model size to another model size.
- Introducing PD separation too early: Accepting the operational cost of disaggregated deployment without a clear latency bottleneck.
Conclusion
The deployment difficulty of trillion-parameter MoE models can be summarized in one sentence: make trade-offs under the constraints of GPU memory, bandwidth, and precision. The memory budget determines the deployment form, inter-GPU bandwidth determines the parallelism strategy, the precision of the KV cache and index cache determines context capacity, PD separation determines the latency profile, and inference effort levels determine the cost per request.
Once the relationships among these constraints are understood, it becomes easier to understand each parameter in this type of deployment guide and to transfer the same approach to building inference services for other large models.
