Eight-Card Ascend Inference: Improving Throughput Under the Same SLO
Throughput gains on multi-card inference only count when service objectives stay fixed. This article explains how to define SLO before measuring throughput, build a comparable baseline, choose tensor, data, pipeline, or combined parallelism for an eight-card Ascend deployment, verify communication and compilation changes, tune scheduling without missing latency targets, and record a minimal reproducible experiment—without treating a reported 53.5% gain as an expected result for your own workload.
1. Define the SLO before talking about throughput
A throughput number is meaningful only under a clear service objective. Online inference usually has to meet first-token latency, per-token generation speed, and request success rate at the same time. If those constraints are not written down first, a team can treat “more requests per second” as success without checking whether tail requests have already timed out.
Before testing, write down three things: the request types and the distribution of input and output lengths; the latency metrics and how they are calculated (for example, time to first token, TTFT, and time per output token, TPOT, at P50, P90, and P99); and the unit of throughput (output tokens per second, or completed requests per second). Once all three are explicit, the same data set can be reviewed by different people.
2. How to build a baseline, and how to read 53.5%
A baseline is the result measured under the same conditions before optimization, not a vague default configuration. It should freeze the model version, card count, input and output characteristics, request arrival pattern, software versions, and SLO pass criteria. Otherwise, a throughput difference may come from a change in test conditions rather than from the optimization itself.
The source material reports that one eight-card Ascend deployment raised total throughput by as much as 53.5% versus its baseline under the same SLO. That material does not disclose the baseline configuration, model size, input and output lengths, concurrency, measurement tools, or number of repetitions, so the figure is only a directional reference. It shows that multi-card configuration changes can leave room to optimize, but it should not be taken as the expected gain for your own service.
When you build your own baseline, the test record should include at least:
- Hardware: device count, model, number of nodes, and how cards are connected.
- Software: versions of CANN, drivers and firmware, the inference engine, and its dependent libraries.
- Model and load: model name and weight precision, distributions of input and output length, and request arrival pattern.
- Service constraints: SLO definition, statistics window, and error criteria.
- Results: throughput, latency at each percentile, device utilization, and test duration.
3. Choosing a parallel strategy for eight cards
More cards do not mean that a higher degree of parallelism is always better. The strategy determines memory use, how compute is spread, and how often devices communicate. Start from model size and remaining memory to set the feasible range, then use controlled comparisons to pick the best combination inside that range.
| Strategy | When it fits | Main cost | What to check | How to roll back |
|---|---|---|---|---|
| Tensor parallelism (TP) | Weights do not fit on one card, or a single layer is compute-heavy | Frequent intra-layer communication; sensitive to inter-card bandwidth | Communication share, compute utilization, TPOT P99 | Lower the TP degree and return to the last stable configuration |
| Data parallelism (DP) | One copy of the weights fits, and request volume is high | Each replica holds a full copy of the weights in memory | Throughput gain as replica count rises, and remaining memory | Reduce the number of replicas |
| Pipeline parallelism (PP) | The model has many layers that can be split by layer | Pipeline bubbles increase, and time to first token may grow | Bubble ratio, TTFT P99, queue length | Reduce the number of pipeline stages |
| Combined strategy (for example, TP × DP) | Large model and relatively high concurrency | Large configuration space and high tuning cost | Throughput and SLO pass rate of each combination under the same load | Fix one dimension and adjust the others one at a time |
When selecting, first get feasible configurations running on a typical load, then check them with tail requests and high concurrency. Looking only at the average load can hide timeouts that appear for some combinations at peak.
4. How to verify communication and compilation changes
After compute is split across cards, data synchronization and result aggregation can sit on the critical path. On Ascend, collective communication is typically handled by HCCL, while operators and lower-level capabilities are provided by CANN. For exact interfaces and default behavior, follow the official documentation for the version you use.
Communication issues can be checked from the following points:
- Communication primitives: confirm the collective type used at each step (such as AllReduce or AllGather) and how often it is called.
- Overlap of communication and compute: use profiling timelines to see whether communication runs in parallel with compute or waits in series.
- Topology: confirm that tensor-parallel groups stay on high-bandwidth links and avoid unnecessary cross-node communication.
- Load balance: compare compute time across cards. If one card waits for a long time, the work is split unevenly.
Compilation and graph-mode changes are verified the same way. Turn only one option on or off at a time—for example, graph mode, operator fusion, or a class of kernels—and record steady-state throughput and TTFT before and after. Compile time is a cold-start cost and should be counted separately from steady-state throughput; otherwise it can wrongly inflate or cancel the gain. Also record whether the compilation cache hit, because cache state affects the first test run.
5. Tuning the scheduler while holding the SLO
Batching can raise device utilization, but forming a batch adds request wait time. Raising concurrency may increase work done per unit time, and it may also lengthen the queue and push up tail latency. Every scheduler parameter should therefore be judged under the SLO constraints.
Treat a change as effective only when all of the following hold:
- Throughput improves and stays stable inside the statistics window, rather than being a one-off spike.
- The SLO pass rate does not fall, and TTFT and TPOT P99 remain inside the target range.
- Error rate and timeout rate do not rise.
- Queue length stays within a controllable range and does not keep growing.
Allow a warm-up period before the test and discard warm-up data. Record both warm-up duration and test duration, and make sure the test covers several full cycles of traffic variation. Sample device state with tools such as npu-smi, and at least record NPU utilization, memory use, and power so issues can be investigated later.
6. A minimal reproducible experiment template
A team can manage each run with one table. Fields should include at least: experiment ID, what changed, configuration values before and after, load identifier, repetition count, throughput, latency at each percentile, error rate, device utilization, conclusion, and rollback point.
Run experiments in this order:
- Fix the load and the SLO, measure the baseline, and save the full configuration.
- Change only one factor at a time, such as the parallel strategy, a communication setting, or one compilation option.
- Repeat the test under the same conditions, record multiple rounds, and compare how spread out the results are.
- If the result is stable and meets the SLO, make it the new baseline. If it gets worse, roll back immediately to the previous baseline and record why.
- After single-factor checks are done, combine several changes and run the full test again.
The benefit is that each performance change maps to one explicit change. If performance regresses after deployment, the team can find the cause quickly.
7. Where this article applies
This article is about throughput optimization for multi-card large-model inference. It does not cover training, and it does not promise a fixed improvement ratio. The methods apply when configurations are compared under one shared SLO. Actual gains have to be reproduced on the target model, real traffic, and the local environment.
Before promoting any optimization to production, confirm that tail latency, remaining capacity, and abnormal cases (such as a single-card failure or a request burst) meet requirements, and keep a baseline you can roll back to.
