Optimizing Large Language Model Inference on Eight Domestic AI Accelerators
Adding more accelerators does not guarantee linear throughput gains. This article examines an eight-card inference benchmark and presents a reusable optimization methodology covering SLO definition, baseline design, parallelism, communication, compilation, scheduling, and regression monitoring. Under the same SLO requirements, coordinated system-level optimization increased throughput by up to 53.5% in a specific large language model workload, illustrating why inference performance must be optimized end to end rather than at the individual-operator level.
In production large language model inference, increasing the number of accelerators does not necessarily produce a proportional increase in throughput. As request-length distributions, concurrency patterns, and tail-latency requirements change, bottlenecks may emerge in parallel partitioning, collective communication, operator execution, batch scheduling, or cache management.
A public comparison involving eight domestic AI accelerators showed that, under the same SLO requirements, adjusting the parallelism strategy and enabling communication and compilation optimizations increased total throughput for a large language model inference workload by up to 53.5%.
The value of this result is not the specific percentage itself. Rather, it demonstrates that inference optimization should move beyond isolated operator acceleration toward coordinated, end-to-end system optimization.
This article uses that case as a starting point to summarize a reusable methodology. It focuses on defining SLOs, designing baseline and optimization comparisons, locating bottlenecks by layer, and avoiding the mistake of treating scenario-specific gains as universal performance guarantees.
1. Define the SLO First: Why Throughput Requires Constraints
Inference throughput is commonly measured in requests per second or generated tokens per second. However, throughput without latency constraints has limited engineering value. A more meaningful definition is the maximum stable throughput that the system can sustain while meeting its SLO.
Common SLO metrics include:
- TTFT (time to first token): Determines the perceived responsiveness of a service.
- TPOT (time per output token): Affects the experience of streaming generation.
- P99 tail latency: Measures stability for long-running or unfavorable requests.
- Error and timeout rates: Ensure that optimization does not come at the expense of reliability.
The baseline and optimized configurations must therefore be compared using the same hardware, model version, input and output length distributions, concurrency model, and SLO thresholds. Otherwise, an apparent increase in peak throughput may simply result from relaxed tail-latency requirements or a changed request distribution.
In practice, teams should record:
- Normalized throughput
- P99 TTFT
- P99 TPOT
- Memory usage
- Accelerator utilization
- Communication wait time as a percentage of execution time
Measurements should be collected over multiple load-test rounds in a stable state, with the median used as the primary result. This helps reduce interference from cold starts and cache warm-up.
2. Abstracting the Case Study: The Baseline Must Be Explainable
In the eight-card case, the baseline can be understood as a general-purpose deployment configuration. Its parallelism strategy was not carefully matched to the model structure or accelerator topology. Communication and compilation optimizations used conventional settings, while service-level batching targeted general throughput.
The optimized configuration focused on three areas:
- Reorganizing the parallelism strategy;
- Optimizing communication and compilation;
- Introducing SLO-aware scheduling at the service layer.
A controlled comparison should include at least the following steps:
- Fix the model and precision strategy, then record the maximum stable throughput under the target SLO.
- Collect phase-level latency data, including request queuing, preprocessing, first-token computation, token-by-token generation, and output post-processing.
- Classify the bottleneck as compute-bound, memory-bound, communication-bound, or scheduling-bound.
- Change only one category of variables at a time and measure the marginal impact on throughput and tail latency.
- Run A/B regression tests on the combined optimizations to verify that the gains are reproducible and do not worsen tail behavior.
An illustrative normalized result can be expressed as follows: baseline throughput is 100, while optimized throughput reaches 153.5, representing a maximum improvement of 53.5%. At the same time, P99 TTFT and P99 TPOT remain within their SLO thresholds.
This number comes from a specific model, hardware configuration, and request distribution. It should therefore be treated as evidence that the methodology can be effective, not as a performance guarantee for all workloads.
3. Key Optimization Paths: Four Layers Must Work Together
3.1 Parallelism: Match Partitioning to Model Structure and Topology
Common parallelism dimensions for large model inference include tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism.
At an eight-card scale, excessive partitioning can increase the number of collective communication operations, while insufficient partitioning may place excessive memory or compute pressure on individual cards. During optimization, key dimensions should first be checked for divisibility, including the number of attention heads, hidden dimensions, and the number of experts. The size of each communication domain should then be selected according to the interconnect bandwidth, keeping high-frequency communication within domains that offer higher bandwidth whenever possible.
For dense models, the combination of tensor parallelism and data parallelism can be evaluated first. For sparse expert models, expert parallelism introduces additional concerns, including load balancing and routing overhead.
A larger parallelism degree is not automatically better. The goal is to balance computation, memory, and communication.
3.2 Communication and Compilation: Reduce Waiting and Increase Execution Density
The main goals of communication optimization are to reduce synchronization frequency, overlap communication with computation, reuse communication buffers, and make collective communication topology-aware.
At the service layer, merging requests within a batch can reduce the frequency of small messages. At the execution layer, graph fusion, operator fusion, static shapes, and memory-pool reuse can reduce launch and allocation overhead.
Compilation optimization should focus on whether graph-level fusion actually reduces memory traffic, whether operator selection matches the current precision and shapes, and whether host-side scheduling has become a bottleneck.
Teams should avoid assuming that more fusion is always better. Incompatible fusion may introduce additional copies or trigger fallbacks, increasing tail latency instead. After optimization, engineers should verify whether communication wait time, operator fallback frequency, peak memory usage, and accelerator utilization improve together.
3.3 Service Scheduling: Use SLO-Aware Batching to Improve Stable Throughput
Service-level optimization often determines whether low-level gains are realized in production.
Continuous batching can improve accelerator utilization, but the batch size must remain within SLO limits. Length bucketing can reduce head-of-line blocking caused by mixing short and long requests. Paged KV-cache management can reduce memory fragmentation and support more flexible concurrency. Request prioritization and rate limiting can help protect tail latency.
A practical approach is SLO-aware scheduling:
- When P99 TTFT approaches its threshold, reduce the rate at which new requests enter batches or trigger preemption.
- When TPOT remains stable and memory capacity is available, gradually increase the batch token limit.
Scheduling policies must be evaluated together with load testing. Otherwise, offline metrics may look favorable while online tail latency deteriorates.
3.4 Monitoring and Regression: Turn a One-Time Optimization into a Continuous Capability
After deployment, teams should establish continuous regression testing. Throughput, P99 latency, error rate, and resource utilization should be monitored by request length, concurrency level, and model version.
New versions should be released through gradual rollout with a fast rollback path. The baseline should also be rerun periodically to detect performance degradation caused by dependency upgrades or traffic changes.
Only a closed loop of baseline, experiment, gradual rollout, and regression testing can make gains such as the reported 53.5% improvement consistently reproducible.
4. Illustrative Pseudo-Configuration: Expressing Intent Without Binding to a Specific Interface
# Illustrative pseudo-configuration; not tied to any specific product interfacedeployment:accelerator_count: 8model_precision: target_precisionparallel:tensor: auto # Select based on divisibility of heads and hidden dimensionsdata: autopipeline: 1communication:topology_aware: trueoverlap_with_compute: truebuffer_reuse: truecompile:graph_fusion: truestatic_shape: truememory_pool: trueservice:continuous_batching: truelength_bucketing: truekv_cache: pagedschedule_policy: slo_awareslo:ttft_p99_ms: <target_value>tpot_p99_ms: <target_value>max_error_rate: <target_value>
This snippet only expresses the optimization dimensions: parallelism that respects divisibility, topology-aware communication, communication-computation overlap, graph fusion, static shapes, continuous batching, length bucketing, paged caching, and SLO-aware scheduling.
In a real deployment, configuration fields should be based on the capabilities and documentation of the target platform. The parameter names and values above should not be copied directly.
5. Implementation Guidance and Boundaries
First, define the SLO before tuning throughput. Throughput gains without tail-latency constraints can mislead capacity planning.
Second, build a baseline before discussing optimization. The baseline should be explainable and reproducible, with the hardware, model, precision, request distribution, and scheduling parameters recorded.
Third, run small experiments and validate combinations. Parallelism, communication, compilation, and service scheduling are coupled. A gain from one isolated change may be canceled after the changes are combined.
Fourth, monitor precision and output consistency. Cache, fusion, and scheduling optimizations must not alter output semantics.
Fifth, treat the 53.5% improvement as case-specific data rather than a general promise. Model architecture, sequence-length distribution, interconnect bandwidth, and SLO thresholds can all change the available optimization headroom.
Overall, large language model inference optimization on eight domestic AI accelerators is a systems engineering task. SLOs provide the constraints, baseline comparisons provide the methodology, parallelism, communication, compilation, and service scheduling provide the main levers, and monitoring with regression testing provides long-term assurance.
Only by turning a one-time case study into a reproducible engineering process can teams achieve a stable balance between output per unit of compute and user experience.
