Deploying a Trillion-Parameter MoE Model on Two Multi-Die Accelerator Nodes: A Hands-On Practice

    This article describes a validated deployment of a trillion-parameter MoE model across two nodes with 32 accelerator dies in total. It covers TP16×PP2 parallelism, W4A8 quantized weights, and the network, weight distribution, and deployment steps required to run the model. It also reports functional checks and three stress tests totaling 836 requests, all of which succeeded, and lists practical guidelines for capacity planning, reproducibility, and observability.

    1. Background and Goals

    As trillion-parameter MoE models move into production validation, deployment has shifted from a single question, whether the model runs on one card, to a harder one: can it run reliably, reproducibly, and observably across multiple nodes and dies while handling long contexts and tool calls? This article draws on a real validation of a domestic multi-die accelerator setup on two nodes. It distills the deployment method into reusable steps. The setup uses two nodes with 16 dies each, for 32 dies in total. The deployment applies TP16×PP2 parallelism and runs one inference instance with a world size of 32. The validation covered chat, reasoning output, tool calling, and three stress tests. The article does not tie the approach to a specific hardware model. It focuses on the engineering path and the key decisions.

    The target is a natively multimodal, ultra-large MoE model. It has a total parameter count in the trillions, roughly tens of billions of active parameters, and a context window of one million tokens. Deploying such a model at full precision places heavy demands on memory and bandwidth. In practice, teams often use W4A8 quantization, with 4-bit weights and 8-bit activations. This reduces weight memory while keeping inference accuracy. The quantized weights in this validation were about 1.5 TB in decimal units. The weights are split into many shards, so the deployment needs the pooled memory and parallel capacity of multiple nodes and dies.

    The validation aimed for more than a successful startup. It covered four production concerns:

    1. Whether the model starts correctly on 32 dies across two nodes with TP16×PP2.
    2. Whether chat, reasoning output, and tool calling work correctly.
    3. Whether three stress tests remain stable: short-to-medium sequences, agent-style 80K-token contexts, and 200K-token contexts.
    4. Whether the operational pipeline, including weight distribution, network selection, and monitoring and logging, can be reused.

    All 836 requests across the three stress tests succeeded. This gives a baseline for future capacity planning and rollout.

    2. Overall Architecture

    The architecture has four layers: the control plane, the parameter network, the inference data plane, and artifact distribution. The control plane handles node enrollment, template delivery, deployment orchestration, and collection of logs and metrics. The parameter network carries collective communication between dies across nodes. The inference data plane handles request entry, gateway routing, and instance responses. Artifact distribution fetches and verifies model weights and the prebuilt inference image.

    For general enterprise deployments, Baidu AI Cloud’s container platform, object storage, log service, and monitoring and alerting can support this workflow. Specific implementations should follow the interfaces, permissions, and network model of the platform in use.

    This article focuses on platform-independent engineering practices. The node agent connects outward to the control plane. Control, monitoring, logging, and inference traffic can then share one tunnel, which reduces internal network exposure. Nodes with domestic accelerator cards and nodes with other general-purpose accelerators follow the same enrollment process. After the installation command runs, the node reports its accelerator, network interface, and storage information.

    3. Resource and Parallelism Planning

    The two nodes provide 32 dies with 64 GiB each, for about 2 TiB of total memory. The W4A8 weights are about 1.5 TB in decimal units, or about 1.3 TiB, which takes roughly two-thirds of total memory. The remaining memory must cover the KV cache, activations, communication buffers, and runtime overhead.

    With TP16×PP2, each pipeline stage spans 16 dies, and the weights are split across pipeline stages. This lowers the weight load on each stage. However, pipeline parallelism across nodes adds transfer delays between stages and pipeline bubbles, so the team needs stress testing to verify performance.

    TP16 matches the topology of 16 dies per node, which supports high-bandwidth interconnect within a node. PP2 splits the model across the two nodes, so both nodes serve one instance. A world size of 32 means the parallel groups, communication domains, and failure domains must all be fixed in the deployment template. In practice, teams should record the topology, quantization method, context window, image digest, and parameter network in the template. This prevents configuration drift from manual changes. An example configuration follows:

    1. deployment:
    2. model: <target trillion-parameter MoE model>
    3. quantization: w4a8
    4. context_window: 1M
    5. parallelism:
    6. tp: 16
    7. pp: 2
    8. world_size: 32
    9. network: param-net
    10. image: <prebuilt inference image>
    11. health_check:
    12. startup_timeout: <set based on measured startup time>
    13. request_probe: true

    4. Node Enrollment and Network Setup

    Nodes join the cluster by running an installation command generated in the console on each node. The platform detects the 16 accelerator dies on each node and collects information about network interfaces, storage, and topology. Both nodes must join the same parameter network. The platform then checks connectivity between cards, and during deployment it selects nodes based on the parameter network.

    The key point is to separate the business network from the parameter network. The business network carries requests and operational traffic. The parameter network carries collective communication across nodes. If the two share a tunnel, evaluate bandwidth, latency, and isolation requirements. Otherwise, control traffic may interfere with inference traffic during long-context stress tests.

    5. Weight Distribution and Storage

    Distributing 1.5 TB of weights strongly affects delivery time. In practice, the team first downloads the weights from the artifact repository to the first node. Then the first node transfers them directly to the second node, which avoids a second public internet download. In measurements, the public download ran at about 50–57 MB/s, and the node-to-node transfer reached about 490 MB/s. At 1.5 TB, the public download takes several hours, while the node-to-node transfer takes about 50 minutes. After distribution, verify the integrity of every shard, lock the version, and use a fixed image digest so repeated deployments produce the same result.

    6. Deployment and Validation

    To deploy, select the prebuilt template, pin the image digest, set TP16×PP2 and the parameter network, and start the instance across the two nodes. The 32-die instance then starts.

    Functional validation covered chat, reasoning output, and tool calling. The stress tests covered three workloads: random short-to-medium sequences (2048 input tokens, 512 output tokens), agent-style 80K-token long contexts, and 200K-token long contexts. The tests sent 836 requests in total. Validation tracked request success rate, timeouts, out-of-memory errors, communication errors, queue backlog, and log anomalies. All 836 requests succeeded, for a 100% success rate. Long-context workloads need close attention to KV cache capacity, page fragmentation, and concurrency limits, so that a single request does not occupy too much memory.

    7. Best Practices

    1. Plan capacity with headroom. Reserve at least 30% of memory beyond the weights for the KV cache and communication buffers.
    2. Distribute weights through direct node-to-node transfer, and verify every shard.
    3. Orchestrate the parameter network separately, and run a connectivity check before deployment.
    4. Fix the image digest, parallelism parameters, and quantization settings in the template.
    5. Test short, medium, and long sequences along with tool-calling paths. Build dashboards for success rate, latency, memory, communication, and queue depth.
    6. Before launching a quantized model, run accuracy regression tests and check operator coverage to confirm numerical stability.

    With these methods, a two-node, multi-die deployment of a trillion-parameter MoE model can move from a one-time validation to a reproducible, maintainable production practice.