8-GPU LLM Inference Throughput Optimization: SLO-Oriented Tuning Practices and Lessons Learned
This article explains how to raise throughput on an 8-GPU large language model (LLM) inference service while staying within service-level objectives (SLOs). It covers why raw throughput alone is misleading, the bottlenecks that commonly appear in baseline deployments (parallelism, communication, compilation, batching, and scheduling), a reference optimization configuration framework, and systematic load-testing and validation methods. By coordinating parallelism strategy, communication overlap, compiler and operator optimization, dynamic batching, and scheduling with overload protection, teams can improve effective throughput rather than surface-level QPS. A baseline-versus-optimized comparison shows up to 53.5% higher total throughput under equivalent SLOs, though the result depends heavily on model structure, hardware interconnect, and request distribution.
1. Why an 8-GPU Inference Service Needs SLO-Oriented Throughput Optimization
In LLM inference, each request demands substantial compute. As model size grows, context length increases, and concurrent requests pile up, the focus of an inference system shifts from “can it run on a single card?” to “can a multi-GPU cluster serve traffic stably?” For an 8-GPU inference service, the core goal is usually not raw, maximum utilization, but maximizing effective throughput per unit time while meeting SLOs.
An SLO can typically be broken down into several metrics:
- First-token latency: the time from sending a request to receiving the first generated token.
- Inter-token latency: the interval between adjacent output tokens during generation.
- End-to-end latency: total time from when a request enters the service to when the full response is returned.
- P99/P95 tail latency: the latency ceiling that the vast majority of requests can still tolerate under high concurrency.
- Request success rate: the proportion of requests that still return successfully under timeouts, throttling, or overload.
If you chase throughput alone and ignore tail latency, the system can inflate the batch size and raise concurrency to boost apparent throughput, but some requests will end up queuing too long and eventually violate the SLO. So 8-GPU optimization must revolve around one core question:
Under the same hardware, the same model size, and the same SLO constraints, how can the system serve more effective requests?
In one baseline-versus-optimized comparison, total throughput rose by up to 53.5% under equivalent SLOs. It is worth noting that results like this depend on model structure, hardware interconnect, request distribution, context length, batching strategy, and tuning depth, and cannot be simply extrapolated to every scenario. But they point to an important direction: inference performance gains usually come not from a single optimization point, but from the combined effect of parallelism strategy, communication, compilation, scheduling, and service-layer parameters.
2. Common Bottlenecks in Baseline Configurations
Many 8-GPU inference services start from a baseline configuration that is “good enough to run and stable.” Such configurations tend to be conservative, simple, and easy to maintain, but under high-concurrency production load they often expose the following bottlenecks.
2.1 Parallelism Strategy Mismatched with the Workload
Eight GPUs do not automatically mean that 8-way tensor parallelism is the optimal choice. Different model structures, request-length distributions, and batching strategies demand different parallelization schemes.
Common mistakes include:
- The model is well suited to a higher degree of data parallelism, but excessive tensor parallelism is forced onto it;
- Communication overhead is significant, yet communication is not sufficiently overlapped with computation;
- Certain layers lose local compute density after being split;
- High-concurrency short requests and low-concurrency long requests share one parallelism strategy, leading to uneven resource utilization.
Once the parallelism strategy no longer matches the workload, you get the classic situation of “plenty of cards, but throughput won’t scale.”
2.2 Communication Becomes a Hidden Bottleneck
In multi-GPU inference, communication overhead is frequently underestimated. It becomes more pronounced in cases such as:
- High tensor-parallel degree, with frequent synchronization between layers or operators;
- Intermediate model results that must be exchanged frequently across cards;
- Large variation in request length, which throws off the ratio between communication volume and computation;
- Interconnect topology that is not fully exploited, creating cross-node or cross-link bottlenecks;
- Communication not overlapped with computation, leaving compute units waiting on data synchronization.
In an 8-GPU environment, if communication is not well optimized, overall throughput can be capped by the communication path even when per-card compute is ample.
2.3 Insufficient Compilation and Operator Optimization
An inference engine may run the model correctly, but without thorough graph-level and operator-level optimization, performance can stay at a “usable but not optimal” state. Common issues include:
- Insufficient operator fusion, which increases memory accesses;
- Unfriendly memory access patterns that leave bandwidth underutilized;
- Overly strong static-shape assumptions, triggering frequent recompilation under dynamic requests;
- Certain key operators that never reach the high-performance path;
- Poor cache, memory-pool, and tensor-reuse strategies.
These issues may be invisible in single-card tests, but under multi-GPU high concurrency they are amplified into an overall throughput drop.
2.4 Batching and Scheduling Too Conservative
A key lever for raising inference throughput is increasing the effective batch size within the limits the SLO allows. If the scheduling strategy is too conservative, several problems appear:
- The batching window is too small to fully use the available compute;
- Request queue times grow too long, pushing up tail latency;
- When long and short requests are mixed, short requests get blocked behind long ones;
- Priority policies are unclear, so critical business requests go unprotected;
- There is no degradation, throttling, or request-reordering mechanism under overload.
Therefore, optimizing an 8-GPU inference service is not just about tuning model parameters; it also requires building finer-grained control at the service-scheduling layer.
3. Optimization Approach: From Baseline to SLO-Oriented High Efficiency
The optimization goal for an 8-GPU inference service can be summed up in four ideas: sensible parallelism, efficient communication, full compute utilization, and controllable scheduling. Around these four directions, a fairly complete optimization path can be formed.
3.1 Establish a Quantifiable Baseline First
Before any optimization begins, you should establish a stable baseline. A baseline is not just running inference once; it is a complete test plan covering typical load, peak load, and abnormal load.
The baseline test should include at least these metrics:
- Total throughput: requests or generated tokens successfully processed per unit time;
- Effective throughput: the number of requests or tokens that meet the SLO;
- P50/P95/P99 latency;
- First-token latency and inter-token latency;
- Accelerator utilization and memory (VRAM or host) occupancy;
- Request failure rate, timeout rate, and throttling rate;
- Throughput curves across different concurrency levels.
A reusable way to express this is:
effective throughput = number of SLO-compliant successful requests / test durationoverall throughput = total successful requests / test duration
If you only watch overall throughput, you will easily miss tail-latency degradation. It is therefore recommended to record both overall throughput and SLO-compliant throughput at the same time.
3.2 Parallelism Strategy: Don’t Blindly Stack Parallelism
In an 8-GPU environment, the parallelism strategy usually involves choosing among these options:
- Tensor parallelism: splits the computation of a single layer across cards to reduce per-card compute pressure;
- Pipeline parallelism: splits model layers into different stages to reduce per-card memory pressure;
- Data parallelism: runs multiple replicas to carry requests and raise overall service capacity;
- Expert parallelism or hybrid parallelism: suited to specific model structures, reducing communication/computation imbalance in expert-routing or sparse-computation scenarios.
When tuning, don’t look only at theoretical compute; focus on three questions:
- Does the split reduce critical-path latency?
- Can the communication it introduces be overlapped with computation?
- Does the strategy fit the current request-length distribution?
For example, if requests are generally short, excessive tensor parallelism may add communication overhead and actually reduce effective throughput; if requests are generally long, per-card memory and compute pressure are high, and moderate splitting may pay off more.
3.3 Communication Optimization: Keep Data Movement From Blocking Compute
The goal of communication optimization is not simply to reduce communication, but to hide it behind computation as much as possible. Common strategies include:
- Choosing a more sensible split based on the hardware interconnect topology;
- Increasing the degree of communication/computation overlap to cut synchronization waits;
- Merging small message transfers to lower communication launch overhead;
- Optimizing the transfer path for intermediate results to avoid unnecessary cross-card copies;
- Caching or reusing high-frequency communication patterns;
- Reducing communication idling caused by imbalance at the request-scheduling layer.
In multi-GPU inference, communication problems often do not surface as errors. Instead, they show up as throughput that won’t scale, large latency jitter, and noticeable utilization swings. When optimizing, combine timeline analysis to watch how the share of communication, computation, memory copies, and scheduling waits changes.
3.4 Compilation and Operator Optimization: Turn a Generic Execution Path into an Efficient One
The core of compilation optimization is to make the inference execution path as close to the hardware characteristics as possible. Work at several levels:
- Graph-level optimization: fold constants, eliminate redundant computation, and streamline control flow;
- Operator fusion: reduce intermediate tensor writes and memory-access overhead;
- Memory reuse: lower the frequency of temporary buffer allocation;
- Shape-strategy optimization: pre-optimize common context lengths to reduce jitter from dynamic switching;
- Precision/performance balance: choose a suitable precision path within what the business can tolerate;
- Warmup mechanism: complete compilation of key paths and cache initialization before real traffic arrives.
Note that compilation optimization should not be judged only by single-request speed; it must also be evaluated for stability under high concurrency. If some optimization works only at low concurrency but causes memory jitter or scheduling delays at large batch sizes, it is not suitable to ship directly.
3.5 Batching and Continuous Batching: Raise Effective Utilization Within the SLO
Batching is a key lever for inference throughput. Static batching often struggles to adapt to dynamic requests, whereas continuous batching can dynamically assemble batches over a request’s lifetime, making the system more flexible.
A good batching strategy usually needs to balance the following:
- Maximum batch size;
- Maximum number of batched tokens;
- Request wait window;
- Mixing strategy for long and short requests;
- Insertion mechanism for high-priority requests;
- Rejection or degradation strategy under overload.
If the batching window is too small, the system cannot fully use the compute; if it is too large, tail latency rises noticeably. Batching parameters should therefore not be set in isolation, but tuned together with the SLO target, concurrency levels, and request-length distribution.
3.6 Scheduling and Stability: SLO Optimization Must Include Overload Protection
A production-grade inference service cannot perform well only under normal load. When traffic spikes, requests run unusually long, or concurrency climbs too high, the system needs overload protection. Common mechanisms include:
- Request throttling: cap total concurrency at the entry layer;
- Priority queues: protect critical business requests;
- Request degradation: in extreme cases, shorten the generation length or drop non-core capabilities;
- Fast failure: avoid requests sitting in a queue for a long time;
- Replica scaling: adjust service capacity as load changes;
- Health checks and circuit breaking: prevent faulty instances from continuing to receive traffic.
SLO-oriented optimization is not about squeezing the system to its limit; it is about raising efficiency within an acceptable boundary of service quality.
4. A Reference Configuration Framework for 8-GPU Inference
The following is an illustrative configuration framework showing the dimensions to consider when optimizing an 8-GPU inference service. The actual values must be determined through load testing based on the model structure, hardware interconnect, request distribution, and business SLOs.
inference_service:parallel_policy:tensor_parallel_size: 8pipeline_parallel_size: 1data_parallel_size: 1expert_parallel_size: 0batching:max_num_seqs: 256max_num_batched_tokens: 8192enable_dynamic_batching: trueenable_chunked_prefill: truememory:device_memory_utilization: 0.90kv_cache_dtype: autoenable_memory_reuse: truescheduler:priority_policy: slo-awaremax_queue_size: 1024preemption_policy: swap-or-recomputecompile:graph_optimization_level: highoperator_fusion: truewarmup_before_serving: truecommunication:topology_aware: trueoverlap_comm_compute: truereduce_sync_overhead: trueobservability:metrics: throughput, latency_p99, latency_p999, queue_time, token_intervalalerts: slo_breach, overload, high_memory_pressure
The point of this framework is not the specific numbers, but to remind you that tuning an 8-GPU inference service should span multiple layers: parallelism, batching, memory, scheduling, compilation, communication, and observability.
5. Load Testing and Validation: Proving the Optimization Actually Works
After optimizing, you must validate the effect through systematic load testing. Otherwise, it is easy to end up with “test metrics that improved, but instability under real production traffic.”
5.1 Build a Realistic Load Model
It is recommended to simulate at least the following traffic types:
- Low-concurrency long requests: to validate single-request latency and long-context handling;
- High-concurrency short requests: to validate batching throughput and scheduling ability;
- Mixed-length requests: to validate stability when long and short requests are mixed;
- Peak burst traffic: to validate throttling, degradation, and overload protection;
- Sustained pressure traffic: to observe memory leaks, cache jitter, and performance decay over long runs.
5.2 Observe Throughput Curves, Not Single Points
When comparing before and after optimization, don’t just compare throughput at one concurrency level; observe the full curve:
- Throughput changes at concurrency 1, 8, 16, 32, 64, and 128;
- Latency changes across different context lengths;
- SLO compliance rate under different batching parameters;
- Resource utilization under different parallelism strategies;
- Throughput stability after long-running periods.
If a configuration performs better at low concurrency but tail latency degrades at high concurrency, it is not necessarily suitable for production.
5.3 Use SLO-Compliant Throughput as the Core Metric
It is recommended to use “the throughput of requests that meet the SLO” as the final acceptance metric, rather than looking only at raw QPS or token generation speed. For example:
Baseline: overall throughput of 1000 requests/min, of which 800 meet P99 = 2s.Optimized: overall throughput of 1400 requests/min, of which 1200 meet P99 = 2s.Overall throughput improved by 40%, but SLO-compliant throughput improved by50%, showing the optimization is closer to the real production goal.
This way of expressing results reflects the value of the optimization more faithfully.
6. Common Problems and Pitfalls to Avoid
6.1 Watching Only Throughput, Not Tail Latency
Degrading tail latency directly hurts user experience. For online chat, real-time interaction, and code generation, P99 latency matters more than average latency.
6.2 Blindly Increasing the Batch Size
A larger batch does not always mean higher throughput. When the batch is too large, it can bring longer queue times, higher memory pressure, and worse single-request latency.
6.3 Ignoring the Request-Length Distribution
Requests of different lengths have very different compute patterns. Short requests care more about scheduling efficiency, while long requests care more about memory, cache, and compute stability. A single set of parameters rarely fits every scenario.
6.4 Doing Only Offline Inference Tests, No Online Canary Validation
Offline load testing can surface problems, but real traffic is more complex. Before shipping an optimization, do a canary rollout and watch SLOs, error rates, and resource swings under real traffic.
6.5 Lacking Observability
Without metrics you cannot locate bottlenecks. At minimum, track throughput, latency, queue length, cache hit rate, memory usage, communication waits, and error rates.
7. Summary: The Key to 8-GPU Inference Optimization Is System-Wide Coordination
The throughput gains of an 8-GPU inference service are not the result of a single parameter, a single operator, or a single engine setting acting alone; they are the combined outcome of several engineering capabilities working together. A fairly complete optimization path can be summarized as:
- Establish a baseline to clarify the current performance level and bottlenecks;
- Analyze request characteristics to determine the distribution of concurrency, length, and priority;
- Adjust the parallelism strategy so computation and communication match better;
- Optimize the communication path so communication is hidden behind computation as much as possible;
- Strengthen compilation and operator optimization to raise the efficiency of key paths;
- Introduce dynamic batching and priority scheduling to raise effective throughput;
- Improve overload protection and observability to keep SLOs stable;
- Confirm the gains through continuous load testing and canary validation.
From an engineering-practice perspective, the most valuable part of 8-GPU inference optimization is not just making one set of test numbers look better, but building a reusable optimization methodology: first define the SLO, then analyze bottlenecks, and finally raise effective throughput through coordinated parallelism, communication, compilation, batching, and scheduling. This way, even when facing different models, workloads, and compute environments, you can more reliably find a path to performance gains.
