Optimizing LLM Inference Throughput on an 8-Card Domestic AI Accelerator
This article walks through inference throughput tuning for a Mixture-of-Experts large language model on an 8-card domestic AI accelerator. It compares a TP8/DP1 baseline with a TP2/DP4 configuration that adds expert parallelism, sequence parallelism communication optimization, graph capture, and operator compilation. It also explains why results must be compared under the same SLO, which metrics to record, and how to reproduce the tuning process step by step with a rollback plan. The reported gain of up to 53.5% applies only to the specific model, hardware, and SLO in this case.
1. Problem Background: Deploying on Domestic Compute Involves More Than Choosing a Model
Deploying a large model in an IT application innovation (Xinchuang) environment starts with choosing the model, but that is only the first step. Deployment efficiency and final performance depend more on whether the inference engine matches the container image, how many accelerator cards the model needs, how parallelism is configured, and whether communication and compilation optimizations fit the actual workload. On an 8-card domestic accelerator, these issues are more concentrated. Per-card memory, inter-chip interconnect bandwidth, operator support, graph capture capability, and scheduling overhead all affect throughput and latency.
Therefore, the most useful output of practice is not a single runnable command but a deployment plan that can be compared, reused, and verified. For a given model and hardware, record the inference engine, image, resource requirements, parallelism parameters, and optimization switches. For tuned plans, keep the baseline version, the optimized version, and the performance report together. Teams can then start from an existing configuration and validate and adjust it against their own hardware and workloads, rather than starting trial and error from scratch.
This article examines one case: inference on a Mixture-of-Experts (MoE) large model running on an 8-card domestic accelerator. The baseline uses TP8 and DP1. The optimized configuration uses TP2 and DP4, combined with expert parallelism, sequence parallelism communication optimization, computation graph capture, and operator compilation optimization. Under the same SLO constraints, the optimized configuration achieved up to a 53.5% increase in total throughput over the baseline. This figure comes from a test with a specific model, specific hardware, and a specific SLO. It should not be extrapolated as a general gain for all workloads. It does show that systematic tuning can substantially unlock hardware potential on domestic compute.
2. Metrics: Why the “Same SLO” Condition Matters
When discussing inference throughput, peak tokens/s alone is not enough. Production environments usually define explicit service level objectives (SLOs), such as P99 first-token latency, P99 per-token latency, request success rate, and timeout rate. If throughput gains come from relaxing latency constraints, the performance numbers become misleading. Therefore, every comparison in this article is made under the same SLO. A throughput gain only matters to the business when latency, error rate, and other constraints still hold.
When evaluating, record at least the following metrics:
- Total throughput: the total input and output tokens, or the total number of requests, successfully processed per unit of time;
- First-token latency: reflects queuing, prefill, and scheduling efficiency;
- Per-token latency: reflects stability during the decoding stage;
- P99 and P95 percentiles: prevent averages from hiding tail latency;
- Memory usage and OOM rate: show whether a configuration can run sustainably;
- Communication wait ratio: shows whether the parallelism strategy is limited by inter-chip bandwidth;
- Compilation and graph capture gains: separate the contributions of runtime optimization and scheduling optimization.
The “up to +53.5%” figure in this case means the largest total throughput gain of the optimized configuration over the baseline, measured across the load points that met the SLO. In practice, plot a throughput-latency curve instead of comparing single points.
3. Core Differences Between the Baseline and Optimized Configurations
The baseline uses TP8 and DP1. This means 8-way tensor parallelism (TP) with a single data-parallel (DP) replica. For large model inference, TP8 splits per-card weights and compute across all 8 cards, but tensor parallelism also adds substantial cross-card communication. When a model contains many expert layers, simply increasing TP can add overhead from activation splitting, communication synchronization, and scheduling.
The optimized configuration switches to TP2 and DP4 and enables expert parallelism. The basic idea is to organize the 8 cards into 4 data-parallel replicas, with each replica running TP2 internally. In the MoE layers, expert parallelism places different experts on different cards. This keeps each replica able to serve requests while increasing overall concurrency, and it reduces communication pressure within any single TP group. Expert parallelism requires attention to routing balance. If requests concentrate on a few experts, hot cards can form. In deployment, monitor expert load distribution, routing fluctuation, and cross-card communication volume.
The following pseudo-configuration shows the tuning differences. It is illustrative only and does not represent actual interfaces:
# Illustrative pseudo-configuration: fields are only meant to show tuning dimensionsbaseline:tensor_parallel: 8data_parallel: 1expert_parallel: falsesequence_parallel: falsegraph_capture: falseoperator_compile: falseoptimized:tensor_parallel: 2data_parallel: 4expert_parallel: truesequence_parallel: truegraph_capture: trueoperator_compile: true
The configuration shows that the optimization is a combined adjustment of the parallel topology, communication paths, and runtime compilation. Any single change may help, or it may do nothing if it does not match the workload. All changes must therefore be validated together.
4. Key Optimization 1: Redesigning the Parallelism Strategy
A parallelism strategy balances memory, compute, and communication. TP8/DP1 splits the model finely and keeps per-replica memory pressure low, but its tensor parallel group is large, so intra-layer communication happens more often. TP2/DP4 creates multiple data-parallel replicas, which raises overall concurrency. Each TP group has only 2 cards, so intra-layer communication stays within a smaller range. For models with a clear expert structure, expert parallelism further spreads expert computation and lowers the load on each card.
TP2/DP4 does not always perform better. If a single layer has very large parameters, TP2 may leave per-card weights too large to fit. If batch sizes are small, the DP4 replicas may be underused. If expert routing is uneven, expert parallelism creates hot spots. Therefore
