Optimizing Large Model Inference on Eight Domestic AI Accelerators: A Practical Guide to SLO-Aware Throughput

    This guide explains how to optimize large language model inference on a single server with eight domestic AI accelerators. It focuses on SLO-aware throughput rather than raw token throughput, covering baseline methodology, parallelism, communication and compilation, continuous batching, chunked prefill, prefix caching, quantization, memory management, reproducible deployment, regression testing, and production monitoring.

    1. Why SLO-Aware Throughput Matters More Than Raw Throughput

    Training and inference have different optimization goals. Training aims to converge as quickly as possible, while inference aims to serve as many requests as possible without violating service-quality requirements.

    An inference request generally has two phases:

    • Prefill: The model processes the entire input sequence at once. This phase is compute-intensive and largely determines time to first token (TTFT).
    • Decode: The model generates tokens autoregressively, repeatedly reading model weights and the KV cache. This phase is memory-bandwidth-intensive and largely determines the time per output token (TPOT), also called inter-token latency (ITL).

    A common approach to maximizing raw throughput is to increase concurrency and batch size. The number of generated tokens per second may rise, but TTFT and TPOT usually deteriorate at the same time. Beyond a certain point, total throughput barely increases while many requests exceed their timeout. The resulting computation provides little practical value.

    A more useful objective is:

    Under clearly defined SLO constraints, maximize the total throughput of requests that meet those constraints.

    • SLO: A service-level objective, such as P99 TTFT ≤ 2 seconds and P99 output-token interval ≤ 50 milliseconds.
    • Total throughput: The total number of input and output tokens generated per unit of time by requests that satisfy all SLO requirements.
    1. SLO throughput (tokens/s)
    2. = Total output tokens from requests meeting all SLOs / Measurement duration
    3. Example:
    4. Over a 300-second window, 1,200 requests all meet the SLOs.
    5. Their total output is 360,000 tokens.
    6. SLO throughput = 360,000 / 300 = 1,200 tokens/s
    7. Comparison:
    8. After increasing concurrency, 400 of 1,500 requests time out.
    9. The requests that meet the SLOs produce 380,000 output tokens.
    10. SLO throughput = 380,000 / 300 ≈ 1,267 tokens/s
    11. The increase is limited, while raw throughput, which counts all requests,
    12. may appear significantly higher and overstate the actual benefit.

    In large model inference, “throughput” commonly has at least three definitions:

    1. The number of generated tokens from all requests, regardless of latency;
    2. The number of generated tokens from requests that meet the SLO;
    3. A stricter metric that counts only requests whose end-to-end latency, including queueing time, falls within the service-level agreement.

    These definitions can produce completely different conclusions. To make configurations comparable, practical SLOs commonly include:

    • Time to first token;
    • The interval between consecutive output tokens;
    • End-to-end latency, usually constrained by P95 or P99 targets.

    The goodput ratio, or the proportion of requests that meet the SLO, is also useful. It helps distinguish insufficient throughput from poor service quality.

    2. Scenario and Baseline

    This guide uses the following scenario:

    • Hardware: One server with eight domestic AI accelerators;
    • Model: An open-source, medium-sized dense large language model;
    • Serving interface: An inference engine compatible with the OpenAI-style API;
    • Workload: Online chat with approximately 1K input tokens and 256 output tokens.

    Strictly speaking, changing the accelerator model, device count, model size, or SLO threshold can change the optimal configuration. The numbers below are examples intended to illustrate the methodology. Actual results must be measured on the target system.

    2.1 Establish a Trustworthy Baseline

    The baseline is the default configuration before dedicated tuning. Record at least four categories of information:

    • Software versions: Inference engine, communication libraries, compiler toolchain, drivers, firmware, and image tags;
    • Hardware and topology: Accelerator model, memory per device, interconnect type and bandwidth, and NUMA placement;
    • Deployment parameters: Parallelism, batch limits, memory utilization, and maximum sequence length;
    • Acceptance criteria: SLO thresholds, concurrency levels used for testing, and input/output length distributions.

    The value of this information is reproducibility. The same configuration should produce similar conclusions when run on another machine of the same type or by another engineer.

    2.2 Use Consistent Load-Testing Rules

    Control Recommended practice Purpose
    Request generation Fix the random seed and sample from a realistic length distribution Prevent different experiments from receiving different inputs
    Warm-up Discard data from the first N seconds Exclude weight loading and operator compilation from steady-state results
    Load levels Gradually increase concurrency from 1 Identify the performance knee instead of measuring only one point
    Repetition Run each configuration at least three times and use the median Reduce the impact of machine-level variance
    Comparison method Change only one variable at a time Make each conclusion attributable

    Treat each experiment as a performance curve:

    • The x-axis is concurrency or request rate (QPS);
    • The y-axis is SLO throughput;
    • Goodput ratio and average queueing time can be added as supporting curves.

    The curve usually has three stages:

    1. Growth stage: Increasing concurrency steadily increases throughput;
    2. Saturation stage: Throughput approaches a platform limit while latency starts to rise;
    3. Decline stage: The goodput ratio drops quickly and SLO throughput decreases.

    The objective is to raise the entire curve and move the knee to the right. The knee can also shift with traffic fluctuations during the day, so continuous measurement is preferable to a one-time benchmark.

    3. Common Bottlenecks in Eight-Accelerator Deployments

    Before changing parameters, identify where the system is constrained. Common bottlenecks on domestic AI accelerators generally fall into four categories.

    3.1 Inefficient Parallelism

    • Tensor parallelism (TP) is too large: Every layer requires AllReduce synchronization. Although each device performs less computation, inter-device communication can become the new bottleneck.
    • The eight devices are not fully utilized: Batch scheduling efficiency decreases and available memory is wasted.
    • Cross-node parallelism is used unnecessarily: Inter-node bandwidth is usually much lower than intra-node bandwidth, so a poor partitioning strategy adds latency.

    3.2 Communication and Compilation Paths Are Not Hardware-Aware

    • The communication library may not use topology-aware scheduling or overlap communication with computation, exposing synchronization time directly on the critical path.
    • The inference engine may not use the graph-optimization capabilities of the accelerator backend. Operators are dispatched individually, and fusion and memory-access optimizations are lost.
    • The model weight format may not match the hardware-friendly layout, causing additional memory movement.

    3.3 Coarse Batch Scheduling and Memory Management

    • Static batching: The batch waits for the longest request to finish. Short requests are forced to run alongside longer ones, wasting accelerator time.
    • Long prefill operations: A long prompt can temporarily consume most compute resources and block decode requests, causing spikes in output-token latency.
    • Contiguous KV-cache reservation: Reserving memory for the maximum sequence length can cause severe fragmentation. Actual concurrency may be far below the theoretical memory limit.

    3.4 Weak Coordination with Upstream and Downstream Systems

    • A client timeout shorter than the server-side SLO causes requests to be canceled early, wasting computation.
    • The gateway may not provide request merging or priority scheduling.
    • Multiple business workloads may share one service and one SLO, allowing long requests to degrade short-request performance.

    Use the following troubleshooting order:

    Order Metric to inspect Typical conclusion Possible action
    1 Decode-stage device utilization and memory bandwidth Low utilization suggests scheduling or memory constraints; near-saturation indicates the hardware is close to its limit See Sections 4.1 and 4.2
    2 Prefill/decode time ratio and communication time ratio Prefill is blocking decode, or AllReduce consumes too much time See Sections 4.1 and 4.3
    3 Tokens per batch and request queueing time Batches are poorly formed or requests experience long-tail queueing See Section 4.2
    4 KV-cache memory share and fragmentation Available concurrency is limited by memory See Section 4.3
    5 End-to-end P99 versus client timeout Timeouts occur on the client side See Section 5

    4. Four Core Optimization Strategies

    4.1 Adjust the Parallelism Strategy

    Parallelism determines how the model is distributed across devices. Common dimensions include:

    • TP (tensor parallelism): Splits individual operators, such as attention heads or feed-forward weight matrices, across devices. Each layer requires collective communication.
    • PP (pipeline parallelism): Splits the model by layers and transfers activations between devices. Pipeline bubbles may reduce efficiency.
    • DP (data parallelism): Deploys multiple complete model replicas and distributes requests among them, with little inter-device communication.
    • EP (expert parallelism): Used for mixture-of-experts (MoE) models by placing different experts on different devices.

    The key principle is:

    Keep communication-heavy parallel dimensions within the highest-bandwidth interconnect domain, and use communication-sparse dimensions for horizontal scaling.

    For a single server with eight accelerators, a practical priority is:

    1. Preferred:
    2. Use all eight devices for one instance (TP=8), then scale with replicas (DP).
    3. Alternative:
    4. TP=4 + DP=2, or TP=4 + PP=2, when the model is smaller
    5. and communication overhead is a major concern.
    6. Use with caution:
    7. Cross-node PP or TP, because inter-node bandwidth is usually the main constraint.

    The following commands are illustrative. Parameter names vary by inference engine.

    1. # Option A: One TP=8 instance, prioritizing single-request latency
    2. python -m inference_server \
    3. --model /models/your-llm \
    4. --tensor-parallel-size 8 \
    5. --max-num-seqs 64 \
    6. --gpu-memory-utilization 0.90
    7. # Option B: TP=4 with two data-parallel replicas, prioritizing total throughput
    8. python -m inference_server \
    9. --model /models/your-llm \
    10. --tensor-parallel-size 4 \
    11. --data-parallel-size 2 \
    12. --max-num-seqs 48 \
    13. --gpu-memory-utilization 0.90

    The per-token compute cost of TP=4 + DP=2 is usually higher than that of TP=8. Each replica has only four devices for the model weights, so each device may need to read more data. However, the configuration avoids synchronization across all eight devices. On accelerators limited by interconnect bandwidth, this trade-off can substantially improve throughput, provided that single-request latency remains within the SLO.

    Several system-level details are easy to overlook but can significantly affect end-to-end latency:

    • Thread affinity: Bind the inference-engine process to CPU cores on the same NUMA node as the accelerators.
    • Communication-thread isolation: Prevent communication threads from competing with compute threads.
    • NUMA affinity: In multi-replica deployments, assign each replica to devices under the same NUMA node whenever possible.

    4.2 Optimize Communication and Compilation

    Once the parallelism strategy is selected, focus on the execution cost of each computation and synchronization step.

    Communication optimization

    • Topology-aware scheduling: Let the communication library recognize the hierarchy of accelerator interconnects, prioritizing collective communication within high-speed domains.
    • Hierarchical AllReduce: Reduce within a node first, then perform cross-node reduction to reduce inter-node traffic.
    • Computation and communication overlap: Split synchronization into smaller buckets and communicate the previous bucket asynchronously while computing the current one.
    • Collective algorithm selection: Use low-latency algorithms for small messages and high-bandwidth algorithms for large transfers.

    Compilation and operator optimization

    • Graph compilation and operator fusion: Fuse adjacent operations such as RMSNorm, RoPE, attention scaling, and quantization/dequantization to reduce intermediate tensors and kernel launches.
    • Memory access and layout optimization: Keep weights in a hardware-friendly layout to avoid repeated format conversion during forward passes.
    • Compilation caching: Persist the graph-compilation cache after the first startup so later launches can load it directly and reduce cold-start time.
    • Vendor acceleration libraries: Prefer the accelerator vendor’s operator and communication libraries when they are optimized for the target hardware.

    Precision formats

    When the SLO remains satisfied, 8-bit activation formats such as W8A8 can be more efficient than pure FP16 or BF16. Weight traffic is reduced, which directly lowers decode-stage memory pressure. However, any quality loss must be verified on a business evaluation set rather than judged only by perplexity.

    Output sampling is another frequently overlooked cost. Performing temperature sampling, top-p/top-k filtering, and stop-word detection token by token on the Python side can create significant CPU overhead, especially at high concurrency. Moving these operations into the inference engine and processing them in batches can provide additional throughput with little functional change.

    4.3 Improve Batch Scheduling

    Batch scheduling determines what each accelerator is doing at every moment and is often one of the fastest ways to obtain measurable gains.

    Continuous batching

    With continuous batching, requests can enter and leave a batch at the token level. When one request finishes, its slot is immediately assigned to a waiting request instead of remaining idle until the entire batch completes.

    For a chat workload with approximately 1K input tokens and 256 output tokens, continuous batching can often provide a substantial throughput improvement.

    Chunked prefill

    Split a long prompt into smaller prefill chunks and schedule them together with decode requests:

    • Decode output intervals become smoother, improving P99 latency;
    • Compute resources are used more evenly.

    Prefix caching

    Reuse previously computed KV-cache states for shared prefixes, such as system prompts and few-shot examples. For workloads with a fixed persona and multi-turn conversations, prefix caching can significantly reduce TTFT.

    Output limits and admission control

    • Set a reasonable maximum generation length to prevent individual long requests from occupying slots indefinitely.
    • When queue length or estimated waiting time exceeds a threshold, reject or defer requests quickly instead of allowing all requests to time out together.

    Protecting the SLO means that allowing some requests to succeed quickly is often better than making every request slow.

    4.4 Balance Quantization and Memory

    The primary value of quantization—such as INT8/INT4 weights or INT8 KV cache—is using freed memory to increase concurrency:

    • Lower KV-cache usage allows more concurrent requests within the same memory budget.
    • Fewer weight bytes need to be read during decoding, reducing per-token latency.

    Quantization is not free:

    • Low-bit quantization can reduce output quality and must be evaluated against a baseline;
    • Dequantization operators may offset part of the gain, depending on hardware support;
    • KV-cache quantization can have a greater impact on long-context workloads and should be validated separately.

    A practical sequence is to first optimize scheduling and parallelism without quantization, then introduce quantization as a second optimization stage. Record both quality and throughput at every step.

    5. Production Engineering for Reproducible Optimization

    One-time tuning is relatively easy. Maintaining the improvement over time is more difficult. The following engineering practices help preserve performance gains.

    5.1 Make Deployment Reproducible

    Turn versions and runtime settings into an executable deployment artifact. Pin the inference engine, container image, driver, and startup parameters so that the baseline can be reproduced in any target environment.

    1. # Example deployment manifest; not tied to a specific engine
    2. service:
    3. model: /models/your-llm
    4. engine_image: inference-engine:<pinned-tag> # Pin the version; do not use latest
    5. runtime_image: accelerator-runtime:<pinned-tag>
    6. accelerators: 8
    7. parallelism:
    8. tensor_parallel: 8
    9. scheduler:
    10. max_num_seqs: 64
    11. enable_chunked_prefill: true
    12. enable_prefix_caching: true
    13. memory:
    14. gpu_memory_utilization: 0.90
    15. slo:
    16. ttft_p99_ms: 2000
    17. tpot_p99_ms: 50

    5.2 Use Configuration Templates and Baseline Comparisons

    Organize templates by model, hardware, and SLO tier. Each template should include both baseline and optimized configurations, along with measured results. New workloads can then start from an existing template instead of repeating the entire tuning process.

    5.3 Add Load Tests to CI

    Make SLO testing a release gate:

    • Run an automated benchmark after upgrading the inference engine or driver;
    • Block the release if SLO throughput decreases beyond a defined threshold, such as 5%.

    This helps detect otherwise subtle performance regressions caused by version upgrades.

    5.4 Monitor Production and Plan Capacity

    • Use SLO throughput rather than accelerator count as the scaling signal. Trigger scaling when goodput falls or queueing time rises.
    • At minimum, monitor TTFT and TPOT percentiles, goodput ratio, tokens per batch, memory utilization, and communication time.
    • Use the measured throughput-versus-concurrency curve as an input to capacity planning.

    5.5 Deploy in the Cloud

    If operating a cluster in-house is not desirable, the inference service can be containerized and deployed on a cloud platform. For example, Baidu AI Cloud provides container-oriented products that can host this type of GPU or NPU inference workload, together with logging, monitoring, and alerting services for observing service quality. Model weights can be stored in object storage to support rapid startup of multiple replicas.

    The actual availability and capabilities of cloud products depend on the target region and service configuration. Consult the provider’s official documentation before deployment.

    6. Common Mistakes and How to Avoid Them

    Mistake Result Prevention
    Comparing only raw throughput and ignoring goodput Benefits are overstated and production timeouts increase Use SLO throughput as the primary metric
    Using different request distributions in different tests Results cannot be reproduced Fix the random seed and length distribution
    Measuring before warm-up completes Initial compilation time is included and performance is underestimated Discard warm-up data
    Increasing TP without considering topology Communication overhead consumes the compute gains Select parallelism based on interconnect bandwidth
    Skipping quality regression after quantization Output quality declines and users report issues Compare every version on a fixed evaluation set
    Ignoring client timeouts The server may meet its SLO while the client has already canceled the request Align end-to-end timeouts and retry policies

    7. Summary and Action Checklist

    For a medium-sized open-source large language model deployed on eight domestic AI accelerators in one server, increasing SLO throughput by more than 50% over a baseline is achievable in some environments. The key is to optimize in the right order:

    1. Establish a baseline with consistent and reproducible measurements;
    2. Define the SLO with explicit percentile thresholds and measurement rules;
    3. Tune parallelism so communication remains within high-bandwidth domains;
    4. Optimize communication and compilation to reduce exposed synchronization and operator overhead;
    5. Improve scheduling with continuous batching, chunked prefill, and prefix caching;
    6. Introduce quantization last to trade memory capacity for concurrency while protecting model quality;
    7. Operationalize the process with templates, CI regression tests, monitoring, and alerting.

    A minimal action list is:

    • Pin inference-engine, driver, and image versions in a deployment manifest;
    • Use a consistent load-testing script to generate a concurrency-versus-SLO-throughput curve;
    • Confirm that the parallelism strategy matches the accelerator interconnect topology and NUMA layout;
    • Enable continuous batching and chunked prefill, then compare P99 latency;
    • Enable KV-cache reuse for shared prefixes;
    • Run quality regression tests before introducing INT8 weight quantization;
    • Integrate load testing into CI and define a throughput-regression threshold.

    Optimization has no permanent endpoint. Hardware drivers, inference engines, and model architectures continue to evolve, so yesterday’s best configuration may become today’s baseline. The sustainable advantage comes from turning measurement methods, controlled experiments, and regression testing into a repeatable engineering process.