8-GPU Domestic AI Accelerator Inference Optimization: Throughput Practices Under SLO Constraints

    This article presents a complete engineering approach to inference optimization on 8-GPU domestic AI accelerators under SLO constraints. It covers how to define throughput and SLO metrics consistently, the four typical challenges of 8-GPU deployment (tensor, data, and expert parallelism, plus long-Prefill blocking), a TP8 baseline configuration, and a five-part optimization path spanning parallelism restructuring, communication, graph execution and compilation, scheduling, and KV Cache/memory management. It also details load-testing and canary-release procedures with an executable checklist, and stresses that throughput gains are only meaningful when attributed with a fixed measurement methodology. Readers will gain a transferable, reproducible framework for moving inference from "runnable" to "operationally stable," without hardware- or model-specific claims.

    From “Can It Run” to “Can It Run Stably and at High Throughput”

    Once large-model inference moves into production, the engineering goal is no longer to load the model successfully, but to serve higher load under stable, controllable, and predictable service quality. This article focuses on 8-GPU inference deployment and tuning on domestic accelerators, walking through the complete chain of SLO-driven throughput optimization: parallelism strategies, communication, compilation, scheduling, memory management, and the evaluation, load-testing, and rollback mechanisms. The discussion is not tied to a specific piece of hardware or a specific model; the emphasis is on a transferable, reproducible engineering methodology.

    On an 8-GPU domestic AI accelerator, getting the model to run completes only 20% of the work. The remaining 80% is: the communication overhead introduced by weight sharding, interference between Prefill and Decode workloads, load imbalance caused by MoE expert routing, and the sensitivity of the vendor runtime to graph execution and operator compilation.

    1. Define Your Metrics First: How to Count Throughput and Judge SLO

    Without a unified definition, optimization easily turns into “local metrics improve while overall experience degrades.” It is best to fix the following table before you start:

    Item Recommended Definition
    Total throughput (Token/s) Total number of output tokens successfully returned within the measurement window ÷ window duration; the window is recommended to be 60s at steady state, discarding the warm-up phase (the first 30s or first N requests)
    Request throughput (RPS) Number of requests that complete successfully within the window ÷ window duration
    Failed request handling Failed requests are not counted in throughput; separately record the failure rate. If the failure rate > 0.5%, the current round of data is considered invalid and should be re-run
    TTFT / TPOT Measured per-request with server-side instrumentation; do not average across requests; report the median and P95/P99
    P95/P99 calculation Use a fixed sampling window (e.g., 60s or ~100k token samples) and apply nearest-rank or a consistent interpolation method; use a single method throughout. When the sample size is too small (e.g., < 1000 requests), percentiles fluctuate widely and should not drive decisions
    Input/output length distribution Fix and explicitly record mean/p50/p95; throughput measured under different length distributions cannot be directly compared
    Memory and stability Record peak memory, steady-state usage, and the number of allocation failures caused by fragmentation

    Key constraint: Every throughput number in this article, and any you produce, is comparable only under “the same input length distribution, the same output length distribution, the same concurrency model, the same SLO thresholds, and the same measurement method.” Relaxing TTFT or allowing P99 to rise will almost certainly increase throughput, but that kind of increase does not count as an optimization gain.

    2. Four Typical Challenges of 8-GPU Deployment

    2.1 Tensor Parallelism: The Larger the Communication Domain, the More Easily Compute Waits

    Tensor parallelism (TP) shards weight matrices by row or column across GPUs, reducing per-GPU memory pressure, but it triggers collective communications such as AllReduce and AllGather at every layer or key stage. The larger the TP group, the more GPUs must participate in synchronization, and the more easily per-GPU compute is stretched by communication waits, making it highly sensitive to interconnect bandwidth and topology.

    2.2 Data Parallelism: With More Replicas, Load Balancing Becomes the Bottleneck

    Data parallelism (DP) improves concurrency with multiple replicas, but uneven request distribution overloads some replicas while leaving others idle; long requests block short ones; and oversized batch windows raise tail latency.

    2.3 Expert Parallelism: MoE Load Imbalance Is an Invisible Ceiling

    Expert parallelism (EP) places different experts on different devices, introducing routing synchronization and expert load imbalance. When the GPU holding a hotspot expert is saturated for long periods while others idle, overall throughput cannot scale linearly with the number of GPUs.

    2.4 How to Prevent Long Prefills from Blocking Short Requests

    This is the most common experience problem in online serving. Four categories of techniques are available: chunked Prefill (split a long prompt into chunks so short requests can interleave), continuous batching (new requests join a batch midway without waiting for the whole batch to finish), request queuing with priorities (put long requests in a separate queue to avoid head-of-line blocking), and prefix caching (reuse computed results when system prompts are shared).

    2.5 How to Choose Among TP / DP / EP: Benefits, Risks, and Applicable Scenarios

    Strategy Main Benefit Main Risk Applicable Scenario
    TP (Tensor Parallelism) Reduces per-GPU memory pressure; fits larger models Large communication domain, topology-sensitive; throughput ceiling constrained by synchronization Weights don’t fit on a single GPU, or still don’t fit after grouping multiple replicas
    DP (Data Parallelism) Improves concurrency and overall throughput ceiling Uneven replica scheduling; per-replica memory and KV Cache must be duplicated Throughput-oriented; a single replica can hold the model
    EP (Expert Parallelism) Distributes expert compute pressure; improves resource utilization for sparse models Expert load imbalance; routing synchronization overhead MoE models where the number of experts is divisible by the parallelism degree
    SP (Sequence Parallelism) Reduces activation memory pressure for long sequences Adds sharding and merging communication Long context, high proportion of long prompts

    3. Baseline: A TP8 Single-Replica Minimal Viable Configuration

    Establish a reference baseline before discussing optimization.

    1. parallel:
    2. tensor_parallel_size: 8
    3. data_parallel_size: 1
    4. expert_parallel: false
    5. sequence_parallel: false
    6. scheduling:
    7. enable_chunked_prefill: false
    8. enable_prefix_cache: false
    9. graph:
    10. enable_graph_mode: false
    11. enable_kernel_compile: false

    Its advantages are clear logic and the ability to fit a relatively large model in a single replica. Its limitations are equally clear: the communication domain covers all 8 GPUs; with only one replica, the concurrency ceiling is limited; without continuous batching, chunked Prefill, or prefix caching enabled, request processing is inefficient; with graph execution and kernel compilation disabled, hardware performance may not be fully unlocked; and under high concurrency, head-of-line blocking degrades P95/P99.

    It is recommended to fill in the following table during the baseline phase and copy it for every subsequent optimization:

    Record Item Baseline After Optimization
    Hardware and interconnect topology (model/count/links)
    Model, precision, engine and version
    Parallelism strategy (TP/DP/EP/SP)
    Input/output length distribution (mean/p50/p95)
    Concurrency model and load-test duration
    Total throughput (Token/s, RPS)
    TTFT P95/P99, TPOT P95/P99
    Failure rate, peak memory

    4. The Optimization Path: Five Categories of Techniques

    4.1 Parallelism Strategy Restructuring: TP8 → TP2 + DP4 + EP

    1. parallel:
    2. tensor_parallel_size: 2
    3. data_parallel_size: 4
    4. expert_parallel: true
    5. sequence_parallel: true

    The meaning is: the model is sharded across a 2-GPU tensor-parallel group, forming 4 data-parallel replicas; MoE expert layers enable expert parallelism; and sequence parallelism is enabled for long-sequence scenarios. The gains come from four places: the communication domain shrinks from 8 GPUs to 2, significantly reducing per-layer synchronization waits; the number of replicas rises from 1 to 4, raising the overall throughput ceiling; expert compute pressure is distributed; and smaller parallelism domains combine more easily with topology optimization and compute-communication overlap.

    The prerequisite must be verified: whether single-GPU memory can hold the weights and KV Cache under a TP2 grouping. If they don’t fit, do not force it; fall back to TP4/TP8, or combine weight quantization and KV Cache compression.

    4.2 Communication Optimization

    Reduce unnecessary cross-GPU synchronization (shrink the TP domain); overlap computation and communication (let data preparation, transfer, and subsequent computation run as parallel as possible to reduce idling); be topology-aware (route high-frequency communication over high-bandwidth links, avoiding cross-NUMA and cross-PCIe-switch bottlenecks); and use sequence parallelism (shard the sequence dimension to relieve long-context activation pressure).

    Parameter (semantics) Effect Impact of Increasing Risk
    tensor_parallel_size Tensor parallelism group size Lower per-GPU memory pressure Larger communication domain; throughput may drop
    data_parallel_size Number of replicas Higher concurrency ceiling Higher total memory demand; possible load imbalance
    expert_parallel Expert parallelism switch/degree Distributes expert pressure Number of experts must be divisible, otherwise imbalanced
    sequence_parallel Sequence sharding Lower long-sequence activation memory Extra sharding/merging communication

    4.3 Graph Execution and Compilation Optimization

    Domestic accelerators are usually sensitive to graph execution and kernel compilation. Operator-by-operator eager execution generates substantial scheduling overhead; enabling graph mode, operator fusion, and auto-tuning generally reduces idling. Points of focus: static graph / computation graph mode reduces per-operator scheduling overhead; operator fusion reduces intermediate tensor reads and writes; automatic kernel compilation selects better implementations based on hardware and input shapes; and dynamic shape compatibility avoids frequent fallback and recompilation when shapes change.

    4.4 Scheduling: Continuous Batching, Chunked Prefill, Prefix Caching

    Parameter (semantics) Effect Impact of Increasing Risk
    max_num_seqs (max concurrent sequences) Upper bound on requests per batch Throughput rises Higher memory usage; worse P99 latency
    max_num_batched_tokens (batch token budget) Upper bound on tokens per batch Throughput rises Worse head-of-line blocking; higher TTFT tail latency
    chunked_prefill (chunk size) Granularity of long-prompt chunking Smaller chunks make it easier for short requests to interleave Too-small chunks increase scheduling and kernel-launch overhead
    prefix_cache Shared prefix reuse Saves repeated computation when prefixes are identical Extra memory usage; wasted when the hit rate is low

    Batch parameters are not “the bigger the better”: too large raises throughput but also raises tail latency, while too small wastes compute. You must use load testing to find the inflection point, rather than fixing values once based on experience.

    4.5 KV Cache and Memory Management

    Set the maximum sequence length reasonably to avoid reserving resources for extreme long requests; adopt paged KV Cache to reduce fragmentation; apply KV Cache quantization or compression where precision is acceptable; and set eviction and reclamation policies for timed-out and low-priority requests. Memory determines the sustainable concurrency scale, so it is simultaneously a stability issue and a throughput issue.

    5. How to Read the Optimization Gains: Conclusions Are Conditional, Numbers Must Be Reproducible

    Optimization gains usually come from coordination across multiple stages, not from a single switch: shrinking the communication domain reduces synchronization waits; multiple replicas raise the concurrency ceiling; expert parallelism distributes load; graph execution and compilation improve operator efficiency; and scheduling and caching improve batching efficiency and tail latency.

    It should be emphasized that specific throughput-improvement percentages cannot be extrapolated across environments. This article gives no universal numbers, because they depend heavily on the following variables: hardware model and interconnect topology, model structure and number of experts, inference engine and version, concurrency model, input/output length distribution, SLO thresholds, and the measurement window and whether failed requests are excluded. The correct approach is to attribute item by item in your own environment according to the table below:

    Technique Throughput Change When Enabled Alone P99 Change When Enabled Alone Still Effective When Stacked Notes/Repro Command
    TP8→TP2+DP4
    Enable EP
    Graph execution + kernel compile
    Continuous batching + chunked Prefill
    Prefix cache

    Numbers become an asset only when each item comes with a definition and a reproduction method; otherwise they are merely unverifiable claims.

    6. How to Load Test and Canary Release: An Executable Checklist

    The load-testing tool must match the inference engine. Common choices include the engine’s built-in serving benchmark scripts, genai-perf-style load-testing tools, or a self-built script (which needs to support TTFT/TPOT instrumentation).

    1. Function and correctness: run at low concurrency (1→8) and verify output correctness, precision alignment with the baseline, and absence of abnormal logs.
    2. Concurrency ramp-up: increase concurrency step by step through 1, 2, 4, 8, 16, 32…, holding each level for 3–5 minutes of steady state before sampling, to find the SLO inflection point.
    3. Fix the length distribution: use the same set of input/output length distributions at each level, record mean/p50/p95, and avoid distribution drift contaminating the conclusions.
    4. SLO judgment rule: specify in advance, for example, TTFT P99 ≤ X ms, TPOT P95 ≤ Y ms, failure rate ≤ 0.5%; if any one is not met, that level fails.
    5. Sampling requirements: base P95/P99 on a fixed window and a fixed percentile algorithm, with a sufficient sample size per level (recommended ≥ 1000 requests).
    6. Monitoring items: peak and steady-state memory, accelerator utilization, communication wait time, request queue time, failure rate, OOM and recomputation counts, and memory-allocation-failure counts.
    7. Long-run test: run continuous load testing for ≥ 2–4 hours, observing memory-fragmentation growth, cache-hit decay, and failure-rate drift.
    8. Rollback thresholds (example; set per business): P99 TPOT exceeds 1.2× baseline for 5 minutes; failure rate > 0.5%; any single OOM occurs; TTFT P99 exceeds SLO for 3 minutes. If any condition is hit, immediately switch back to the baseline configuration.
    9. Canary path: 1% → 5% → 20% → 50%, observing each stage for no less than 30 minutes, and keeping a baseline instance online on standby.

    7. An Illustrative Optimized Configuration (Read the Prerequisites First)

    The usage prerequisites must be explicit: the following YAML is a semantic illustration; field names and value ranges depend on the specific inference engine and its version (field naming, defaults, and supported capabilities differ substantially across engines/vendor runtimes). Before deploying, you must map it to the official parameter documentation for the version you use. If you cannot confirm a field’s semantics, first run the baseline with the engine defaults, then enable features one by one.

    Applicable boundaries for 8 GPUs + TP2 + DP4 + EP: single-GPU memory can hold the weights and KV Cache after 2-GPU sharding; the number of experts is divisible by the EP parallelism degree and expert distribution is relatively balanced; the inter-GPU interconnect topology supports high-speed communication across 2 GPUs (gains drop noticeably when it is cross-NUMA or PCIe-only); the model is indeed an MoE structure (enabling EP on a dense model is meaningless); and sequence length and concurrency fit within the memory budget.

    1. model:
    2. dtype: bfloat16
    3. quantization: weight_only
    4. max_model_len: 8192
    5. parallel:
    6. tensor_parallel_size: 2
    7. data_parallel_size: 4
    8. expert_parallel: true
    9. sequence_parallel: true
    10. scheduling:
    11. enable_continuous_batching: true
    12. enable_chunked_prefill: true
    13. enable_prefix_cache: true
    14. max_num_seqs: 256
    15. max_num_batched_tokens: 16384
    16. runtime:
    17. enable_graph_mode: true
    18. enable_kernel_compile: true
    19. compile_mode: auto
    20. dynamic_shape_fallback: true
    21. memory:
    22. kv_cache_block_size: 128
    23. gpu_memory_utilization: 0.90

    Core idea: a small TP domain lowers communication pressure, DP raises concurrency, EP/SP adapt to the model structure, graph execution and compilation improve operator efficiency, and scheduling and caching improve batching effectiveness. Note that when gpu_memory_utilization is set too high (e.g., above 0.90), activation peaks or cache growth can easily trigger OOM, so it should be lowered based on actual load-test results.

    8. Common Problems and Risks

    Risk warning: modifying the parallelism strategy, batch parameters, compilation options, and memory parameters can all directly change memory usage and scheduling behavior, leading to OOM, latency degradation, throughput drops, or even service unavailability. Any change should first be validated for functionality and memory at low concurrency, then go through load testing and canary release, and always retain a one-click path to roll back to the baseline. It is recommended to change only one category of parameters at a time; otherwise gains cannot be attributed and failures cannot be located.

    What if throughput improves but tail latency worsens? It is usually because batching is too large or scheduling too aggressive: lower the maximum batched token count, reduce the maximum number of concurrent requests, enable finer-grained chunked Prefill, set priorities or separate queues for long requests, and re-check whether the SLO is too strict.

    How to troubleshoot insufficient memory? Check in order: whether the sequence-length setting is too large, whether KV Cache usage is too high, whether prefix caching brings extra overhead, whether weight quantization can reduce weight usage, and whether you can lower the maximum concurrency or the number of cache blocks.

    Does compilation optimization always yield gains? Not necessarily. It is more clearly beneficial for static workloads, stable input shapes, and long-running stability; when input shapes change drastically and dynamic-compatibility mechanisms are insufficient, frequent fallback or excessive compilation overhead may occur.

    Is a more complex parallelism strategy always better? No. The more complex the strategy, the harder it is to tune and the more easily it is affected by topology and load imbalance. In engineering, prefer solutions that are explainable, rollback-capable, and monitorable.

    Common mistakes checklist: comparing throughput without fixing the length distribution; reporting only averages and not P95/P99; counting failed requests in throughput; changing multiple parameters at once; comparing TP scales on machines with non-equivalent topologies; and going to full production without a long-run test.

    9. Summary and a One-Page Launch Checklist

    Optimizing inference on 8-GPU domestic AI accelerators is not about stacking parameters, but about building a closed loop around the SLO: the parallelism strategy determines how resources are utilized, communication optimization determines multi-GPU coordination efficiency, compilation and graph execution determine operator-level efficiency, scheduling and caching determine request-level service capability, and evaluation and canary release determine whether the solution lands stably.

    Launch checklist:

    • Baseline: hardware/topology, model and precision, engine version, parallelism strategy, length distribution, throughput and P95/P99, and failure rate recorded
    • Configuration: field names checked against the engine version in use; the applicability boundaries of the parallelism strategy (memory, number of experts, topology) confirmed
    • Load testing: low-concurrency functional verification, concurrency ramp-up, SLO judgment, P95/P99 sampling method, and a long-run of ≥ 2h
    • Canary: 1%→5%→20%→50%, each stage observed for ≥ 30min, with the baseline instance online
    • Monitoring: alerts connected for peak memory, utilization, communication wait, queue time, failure rate, and OOM counts
    • Rollback: trigger thresholds defined, the rollback procedure rehearsed once, and the ability to switch back to baseline within minutes

    Only by managing the model, hardware, engine, parallelism strategy, parameter configuration, and performance data in a unified way can you move from “runnable” to “operationally stable.”