Deploying a Trillion-Parameter MoE Model on Dual-Node Domestic AI Accelerator Cards
This article details a practical approach to deploying a trillion-parameter Mixture of Experts (MoE) large multimodal model on a dual-node cluster of domestic AI accelerator cards. It covers the complete deployment lifecycle, including node access, parameter network planning, cross-node weight distribution, and parallelism configuration. With a global parallelism degree of 32 (16-way tensor parallelism and 2-stage pipeline parallelism), the system successfully passed stress tests covering standard chat, tool calling, and long-context agent scenarios with 836 requests, validating its functional correctness and operational stability for large-scale model inference.
1. Background and Objectives
As trillion-parameter MoE multimodal models enter practical applications, the memory and interconnect capabilities of a single node are often insufficient to host a complete instance. Cross-node deployment has become a necessary choice, but it introduces a series of engineering challenges, such as parameter network planning, weight distribution, parallel splitting, and long-context stability. Based on a dual-node cluster of domestic AI accelerator cards, this article provides a reusable deployment and stress testing practice: each node contains 16 compute dies with 64 GiB per die, totaling 32 dies across two nodes. The configuration uses 16-way tensor parallelism per node and 2-stage pipeline parallelism across nodes, resulting in a global parallelism degree of 32.
The goal of this practice is not to rank the performance of a single product, but to verify whether an ultra-large-scale MoE model can achieve a complete closed-loop workflow on a general inference platform with domestic accelerator cards. This includes node access, weight preparation, instance launch, conversation, inference, tool calling, and long-context agent scenarios. Ultimately, all 836 requests across three stress test groups succeeded, demonstrating that the solution is reproducible in terms of functional correctness and operational stability.
2. Overall Architecture and Deployment Path
The overall architecture is divided into the control plane, data plane, and inference plane. The control plane is responsible for node registration, network orchestration, template distribution, logging, and monitoring. The data plane includes the high-speed interconnect between node accelerators and the weight distribution channel. The inference plane loads quantized weights via the inference runtime to provide conversation and tool calling APIs externally. The node agent uses an active outbound connection to the server, allowing control, monitoring, logging, and inference traffic to reuse the same tunnel, thereby reducing cross-network access complexity.
The deployment path can be abstracted into five steps:
- Node Access: Generate an installation command in the console, execute it on the target machines, and the platform automatically identifies the accelerator dies, network cards, and storage information.
- Parameter Network: Add the high-speed interconnects of the two nodes’ accelerators to the same parameter network, and perform an inter-card connectivity check before deployment.
- Weight Preparation: Download the quantized weights from the model repository to the first node, then copy them directly to the second node via inter-node transfer.
- Template Deployment: Select the inference template adapted to the accelerator card, set the tensor parallelism, pipeline parallelism, and global parallelism degree, and launch the instance.
- Verification and Stress Testing: Send requests from the server, route them through the platform gateway to the instance, covering short sequences, long contexts, and agent tool calling chains.
3. Node Access and Parameter Network Planning
The key to node access is enabling the platform to accurately recognize the hardware topology. After the two nodes went online, the platform automatically identified 16 compute dies per node. For multi-node inference, the parameter network must be distinguished from the business network: the business network carries management, logging, and external requests, while the parameter network carries the high-frequency communication in tensor parallelism and pipeline parallelism. After integrating the high-speed interconnects of both nodes into the same parameter network, the platform performs an inter-card connectivity check and automatically selects nodes based on the parameter network during deployment, avoiding communication failures caused by incorrect network selection.
In practice, it is recommended to front-load the following checks:
- Confirm that the die count per node, single-die memory, and topology recognition results are consistent.
- Confirm that all nodes within the parameter network are mutually reachable and the communication link status is stable.
- Confirm node time synchronization, driver versions, and runtime versions are consistent.
- Confirm that the storage path capacity is sufficient to hold approximately 1.5 TB of weights and temporary files.
- Confirm the connectivity of the service port from the platform gateway to the instance.
4. Weight Acquisition and Cross-Node Distribution
This deployment uses approximately 1.47 TB of W4A8 quantized weights, totaling 346 weight shards. The weight preparation adopts a strategy of “one public network download, direct inter-node transfer”: the complete weights are first downloaded from the model repository on the first node. The second node does not repeat the public network download; instead, it receives the weights via the platform’s “copy to” capability directly from the first node. In testing, the public network download speed was about 50–57 MB/s, while the inter-node direct transfer reached about 490 MB/s, improving efficiency nearly tenfold. Upon completion on both sides, a file-by-file checksum was performed to ensure shard integrity and consistency.
The value of this strategy lies not only in saving time but also in reducing the impact of public network fluctuations on deployment success rates. For ultra-large weights, it is recommended to:
- Reserve sufficient disk space before downloading, and avoid mixing it with logging or cache directories.
- Perform a file-by-file checksum after shard transfer; do not only verify directory size.
- Record the weight version and quantization format to ensure both nodes load the identical weights.
- Conduct bandwidth and stability pre-checks on the cross-node direct transfer channel.
5. Parallelism Strategy and Configuration
At this model scale, 16-way tensor parallelism within a single node fully utilizes the high-bandwidth intra-node interconnect, while 2-stage pipeline parallelism across nodes splits the model layers across the two machines. The global parallelism degree is 32. An example configuration is as follows:
model:path: /models/moe-w4a8quantization: w4a8parallel:tensor_parallel: 16pipeline_parallel: 2world_size: 32network:parameter_network: accel-high-speedruntime:backend: vendor-adapted
During configuration, three points require attention: First, the tensor parallelism degree should match the intra-node die count and interconnect topology. Second, the number of pipeline parallelism stages should cover the cross-node splitting to prevent single-node out-of-memory (OOM) errors. Third, the global parallelism degree must equal the tensor parallelism degree multiplied by the pipeline parallelism degree. After deployment, verify whether each rank initializes normally, whether the weights are fully loaded, and whether the service can respond to external requests.
6. Stress Test Design and Results
Stress tests were initiated from the server, routed through the platform gateway to the inference instance, covering three scenarios: standard requests with random input length of 2048 and output of 512; an agent scenario with approximately 80K context; and an agent long-context scenario with approximately 200K context. All 836 requests across the three groups succeeded. The conversation, inference process, and tool calling all functioned normally.
It is important to emphasize that this set of data primarily validates functional correctness and operational stability, and does not equate to a throughput or latency benchmark. For production selection, further testing should include concurrency gradients, P99 latency, peak memory usage, communication ratio, and long-term stability tests. However, the success of all 836 requests demonstrates that the dual-node 32-die instance, under W4A8 weights with 16-way tensor parallelism and 2-stage pipeline parallelism, can successfully cover standard conversation and long-context agent chains.
7. Operations and Troubleshooting Checklist
Common issues can be troubleshooted by layer: at the node layer, check die recognition and driver status; at the network layer, check parameter network connectivity and bandwidth; at the weight layer, check shard checksums and loading logs; at the parallelism layer, check rank initialization and communication timeouts; at the service layer, check gateway forwarding and request timeouts. It is recommended to retain deployment templates, weight versions, image digests, and stress test request distributions to facilitate issue reproduction.
8. Conclusion
The key to deploying a trillion-parameter MoE model on dual-node domestic AI accelerator cards is not merely stacking single-point parameters, but achieving systematic collaboration across nodes, networks, weights, parallelism, and service links. Through templated deployment, parameter network pre-checks, cross-node direct weight transfer, and layered stress testing, real-machine verification of a 32-die instance can be completed in a relatively short time. This practice provides a reusable engineering path for deploying similar ultra-large-scale models on domestic accelerator clusters.
