Deploying a 2.8T MoE Model on Two Chinese NPU Nodes: A Practical Guide
This hands-on guide walks through deploying a 2.8T-parameter multimodal MoE model (about 104B activated per inference) with a 1M-token context window across two Chinese NPU servers, using TP16 + PP2 for a world size of 32. It explains the memory math behind the two-node requirement, then covers onboarding and connectivity checks, parameter network setup, and a "download once, distribute directly" approach to the 1.47 TB W4A8 weights with measured timings. It also covers version pinning for images and weight revisions, load testing and tuning advice (chunked prefill, prefix cache), a troubleshooting checklist, and five best practices for large-scale MoE deployment.
1. Why One Node Isn’t Enough
This deployment targets a native multimodal MoE model with 2.8T total parameters, only about 104B activated per inference, and a 1M-token context window. At first glance, 104B activated parameters doesn’t sound extreme. However, MoE sparsity only applies to computation. Every expert parameter must stay resident in memory, and inference reads only the experts selected by the router. When sizing resources, total parameter count is therefore a more meaningful figure than activated parameter count.
Memory capacity can be estimated directly:
- Weights: With W4A8 quantization (4-bit weights, 8-bit activations), the weights take up about 1.47 TB (about 1.33 TiB) across 346 weight shards.
- Per-die capacity: Each Chinese NPU card consists of 2 compute dies, each with 64 GiB of memory. One server has 16 dies and 1024 GiB (1 TiB) in total, and two servers have 32 dies and 2048 GiB.
- Conclusion: The weights alone exceed a single node’s capacity, so two nodes are required. After subtracting the weights, the 32 dies leave about 680 GiB for KV cache, activations, communication buffers, and the vision encoder. Real 4-bit weights also contain a few higher-precision parameters, so actual usage may be slightly higher than the theoretical figure.
These numbers also define the model’s capability boundary. Whether the 1M-token context can actually run depends on how quickly the KV cache grows with sequence length and on concurrency levels. Keep the memory budget in mind throughout planning, deployment, and load testing.
2. Preparation: Nodes, Network, and Weights
2.1 Node Onboarding: One Tunnel for Control and Data
Heterogeneous compute platforms typically use a “control plane + node agent” architecture. Both components ship as a single static binary, so deployment just means running the install command generated by the console on each target machine.
The agent then initiates a connection to the control plane. Control commands, resource monitoring, log uploads, and inference traffic all share this single tunnel. Nodes need no inbound ports and no static public IP, and machines behind NAT can still be managed. NPU and GPU nodes onboard in exactly the same way. Once the command finishes, the node comes online with its accelerators, network cards, and local storage.
In this deployment, the platform identified 16 910C compute dies on each server. As soon as a node is online, verify the following:
- The card count and die count are complete.
- Each die has 64 GiB of memory.
- The internet-facing network cards are correctly identified.
(On Baidu AI Cloud: Compared with a self-built cluster, using heterogeneous compute on public cloud platforms such as Baidu AI Cloud usually requires extra attention to whether security groups or network access control policies allow the tunnel. Confirm outbound connectivity before onboarding.)
2.2 Networking: Putting All 32 Dies in One Parameter Network
Distributed inference relies on two types of network:
- Parameter network: Carries inter-card and inter-node collective communication and is the most sensitive to bandwidth and latency.
- Storage/management network: Used for pulling models, collecting logs, and sending control signals.
Once the nodes are online, create a parameter network under network management in the platform. Add all the HCCS ports used for high-speed inter-card links from both servers to this network. The platform then runs connectivity checks (node probing, port connectivity, and latency measurement) and schedules nodes only within this network during deployment.
Keep three things in mind when planning the parameter network:
- The internet-facing ports on both machines must belong to the same parameter network. If either one is missing, distributed initialization will fail.
- Cross-node communication should use RDMA over a lossless network (for example, with PFC/ECN configured). Falling back to TCP significantly increases collective communication time.
- Host-level settings such as hugepages, network card multi-queue, and NUMA affinity should be configured consistently. Otherwise, performance differences between the two nodes can slow down the entire system.
2.3 Weights: 1.47 TB, Downloaded Only Once
Store the weights in shared object storage, then add the corresponding storage configuration in model management in the platform. The W4A8 weights used here come from a community quantization release. Before going live, confirm three things:
- The source is trustworthy and checksums can be verified.
- The quantization configuration is provided together with the weights.
- Auxiliary files such as the tokenizer and image-processing configurations are complete, because multimodal capabilities depend on them.
For distribution, the first node downloads directly from object storage. The second node then uses the “Copy from…” feature to transfer the weights directly from the first node. The measured results were as follows:
| Distribution path | Measured speed | Estimated time for 1.47 TB |
|---|---|---|
| Node 1: public network download | 50–57 MB/s | About 7.2–8.1 hours |
| Node 2: direct transfer between nodes | About 490 MB/s | About 50 minutes |
| Total | — | About 8–9 hours |
| Both nodes downloading over the public network (comparison) | 50–57 MB/s | About 14.4–16.3 hours |
Direct transfer is nearly 10 times faster than public network download and cuts 6–7 hours off the total. It doesn’t depend on the model repository’s server-side bandwidth; speed depends only on the link between the two nodes. In large deployments with more nodes, this “download once, distribute to many” approach delivers even greater benefits.
Once the transfer is complete on both sides, the platform verifies every file. Given 346 shards and more than a quadrillion bytes of data, it’s best to handle verification as follows:
- Hash verification is single-threaded by default, and verifying 1.47 TB can take tens of minutes. Run it in parallel with subsequent environment checks.
- If you only care about integrity and need it faster, use a parallel hashing tool.
- Pay particular attention to the large compute-intensive shards (attention and MoE experts).
3. Deployment Configuration: TP16 + PP2, world size 32
The test environment was configured as follows:
| Item | Configuration |
|---|---|
| Nodes | 2 910C servers, each with 16 dies (64 GiB per die), for 32 dies total |
| Interconnect | Inter-node compute communication over HCCS |
| Parallelism | Tensor parallelism (TP) of 16 within each node, pipeline parallelism (PP) of 2 across nodes, world size 32 |
| Model | W4A8 quantized weights, about 1.47 TB, 346 shards |
| Runtime | The official inference container image released with the model |
| Platform | Heterogeneous compute platform; load testing is initiated from the management node and reaches the instance through the platform gateway |
The parallelism strategy follows one principle: place the most communication-intensive dimension on the highest-bandwidth link.
- TP: When matrices within the same layer are split, every layer needs AllReduce/AllGather operations. These are frequent and cannot overlap with computation. TP groups are therefore confined within a single node, using the high-speed inter-die interconnect.
- PP: Only activations are passed at pipeline stage boundaries, so the communication volume is relatively small. However, pipeline stages have bubbles, and inter-stage transfer latency adds directly to these bubbles. The cross-node link is therefore better suited to handling it than the high-bandwidth links within a node.
Pin the image by digest and specify a concrete weight revision for the model (bc632341). This prevents the two nodes from loading different versions if a tag or branch is later overwritten. In distributed environments, such “silent version swaps” are extremely hard to troubleshoot.
Illustrative orchestration configuration (the fields below are example values for explanation only; use the actual interface of your platform):
# Instance orchestration configuration (illustrative)name: moe-multimodal-2nodenodes: 2 # 16 dies per node, 32 in totalimage: <inference-image>@sha256:<digest> # Pin the image version with a digestmodel:source: <model identifier in object storage>revision: bc632341 # Pin the weight versionquantization: w4a8network: <parameter-network-name> # HCCS ports of both nodes must be in the same parameter networkparallel:tensor-parallel-size: 16 # TP within a nodepipeline-parallel-size: 2 # PP across nodes# world size = 16 × 2 = 32limits:max-model-len: 262144 # Adjust to the KV cache budget; approaching 1M requires incremental validation
4. Load Testing and Tuning: Understand Every Latency Metric
Once the instance is ready, you can start load testing. In this test, requests were sent from the management node and forwarded through the platform gateway to the instance, so end-to-end latency naturally includes gateway overhead. When analyzing results, use server-side metrics (such as queueing time and time to first token measured by the engine) together with gateway access logs. This separates time spent outside the gateway from time spent inside the engine.
Design test scenarios across three dimensions:
- Plain text chat: Cover short, medium, and long inputs, and focus on how TTFT grows with input length.
- Multimodal requests: Use mixed text and image input. Image resolution directly determines the number of visual tokens, and therefore the prefill compute required.
- Long context: Gradually approach 1M tokens to verify the KV cache capacity limit and whether the service exits due to out-of-memory (OOM) errors.
Key metrics to watch:
- TTFT (time to first token): Mainly affected by the prefill phase, and rises noticeably in multimodal and long-context scenarios.
- TPOT/ITL (per-token generation latency): During decode, each step activates only about 104B parameters, so compute is light. Latency is mainly determined by weight-read bandwidth.
- Throughput (output tokens/s) and concurrency: Whether MoE expert load is balanced shows up directly as latency jitter between requests.
- Error rate and timeout rate: Timeout thresholds on the gateway and the engine should be coordinated so that long prefill requests aren’t cut off prematurely.
For optimization, try the following first:
- Enable chunked prefill: This slices long prompts and schedules them together with decode requests, reducing how much long inputs block other requests.
- Enable prefix cache: Repeated prefixes, such as system prompts and multi-turn conversation history, can reuse cached KV, reducing redundant computation.
- Set max-model-len and maximum concurrency based on the KV cache budget: Start with conservative values rather than defaulting to 1M and stress testing from there.
5. Troubleshooting Checklist
| Symptom | Possible cause | What to check |
|---|---|---|
| Node doesn’t come online or frequently disconnects | Agent not running; tunnel blocked by a security policy | Check the agent process and outbound connectivity, and review agent logs |
| Card or die count not fully recognized | Driver or firmware issue; devices not enumerated properly | Check device status and memory readings; if needed, fix the issue and re-enroll the node |
| Parameter network connectivity check fails | Ports not added, IP conflicts, RDMA not ready | Check port and routing configuration, and confirm the lossless network settings are consistent |
| Instance stuck in distributed initialization | Inconsistent weight version or image digest across nodes, world size mismatch | Compare the revision and digest on both nodes, and confirm the product of the parallelism degrees equals the world size |
| OOM when loading weights | Insufficient memory budget; TP group unexpectedly spanning nodes or sharding unevenly | Check that TP stays within a node, and consider adding PP stages (as long as the layer count matches the number of stages) |
| Weight file verification fails | Interrupted transfer or corrupted files on the storage side | Redistribute and compare checksums file by file, and confirm all 346 shards are present |
| Excessive TTFT for long-context requests | Heavy prefill compute, long queueing times | Enable chunked prefill and set a reasonable max-num-seqs |
| Noticeable latency jitter | Unbalanced expert routing, PP bubbles | Observe utilization on each die and adjust batching based on engine metrics |
6. Best Practices
Deploying a massive MoE model across two nodes comes down to five lessons:
- Do the memory math first: Size resources by total parameters rather than activated parameters, and leave plenty of headroom for KV cache and the vision encoder.
- Get the network ready first: Have the parameter network cover all participating nodes, keep TP within nodes, and put PP across nodes.
- Download the weights only once: One download plus direct node-to-node transfer is nearly 10 times faster than downloading over the public network on each machine, and each file should be verified afterward.
- Pin every version: Keep the image digest and model revision consistent across nodes to eliminate silent version swaps.
- Test by dimension: Evaluate text, multimodal, and long-context scenarios separately, and analyze end-to-end metrics together with engine metrics.
The same approach applies to other massive MoE models and other domestic accelerator platforms. The key is to quantify resource boundaries first, then design the network and parallelism topology, and finally verify the results with reproducible versions and multi-dimensional load testing.
