LLM Inference Optimization on a Single 8-Accelerator Node: A Path to Higher Throughput under SLO Constraints
This article walks through optimizing large language model inference on a single node with 8 accelerators to improve throughput under SLO (service-level objective) constraints. It shows how to define effective throughput and establish a baseline, then covers parallelism strategy, communication overlap, graph-mode compilation, KV cache management with chunked prefill, and layered quantization. It also presents a top-down profiling method for locating bottlenecks and a framework for recording each configuration as a reproducible deployment plan, turning performance tuning into a repeatable engineering capability rather than a one-time sprint.
1. Why Can Throughput Vary by 50% with the Same Hardware and Model?
In production compute-cost accounting, the metric that actually matters is never “how fast a card runs.” It is “how many tokens a single node can stably serve under a latency constraint the business can tolerate.” When the accelerator card, the model weights, and the network parameters are all fixed, throughput can still vary by dozens of percentage points. The reason usually isn’t the hardware itself but the quality of the inference stack’s configuration: whether the parallelism scheme matches the inter-card interconnect topology, whether collective communication is effectively overlapped, whether computation follows a compiler-optimized fused path, whether batch scheduling gets blocked by overly long prefills, and whether the KV cache capacity estimate leaves enough headroom for concurrency.
So the right starting point for inference optimization isn’t “tuning parameters” but establishing a baseline: first run a conservative, generic configuration that doesn’t rely on special features, record its throughput under a fixed SLO, then introduce optimization techniques one by one, regressing each with the same measurement script. Optimization without a baseline can’t tell you which variable produced the gain, nor whether a performance change after a version upgrade is a regression or an improvement.
It’s worth noting that any throughput improvement is strongly correlated with model architecture (dense/MoE), weight precision, input/output length distribution, and inter-card interconnect bandwidth. Cross-hardware or cross-model comparisons are meaningless. Every number must be measured on your own target workload.
2. Define the Metrics First: SLO and “Effective Throughput”
Most performance arguments are really arguments about metric definitions. Before experiments begin, lock down these four:
- TTFT (time to first token): reflects prefill latency and directly determines the experience in interactive scenarios;
- TPOT / ITL (inter-token latency): reflects the steady-state decoding speed and determines the smoothness of long-text generation;
- End-to-end latency: TTFT + (output length − 1) × TPOT, used as a backstop constraint for long requests;
- Effective throughput: the number of tokens produced per unit time by requests that meet the SLO, not the total tokens from all requests.
The fourth point is the most easily overlooked. Without SLO filtering, pushing concurrency to the limit always yields a nice peak throughput, but a significant fraction of those requests have already timed out — and such a number is useless for capacity planning.
You must also fix the request distribution: the mean and percentiles of input/output lengths, the arrival process (Poisson or fixed QPS), and whether prefixes are shared. A slight shift in the length distribution completely changes KV cache occupancy and the benefit structure of batching.
# Pseudocode: measuring effective throughput under a fixed SLO and request distributiondef measure_effective_throughput(engine, trace, slo, target_qps):engine.warmup(num_requests=64)engine.reset_metrics()accepted, rejected, produced = 0, 0, 0for req in poisson_arrivals(trace, target_qps): # Inject according to the real arrival processengine.submit(req)while engine.inflight() > 0:engine.step()for r in engine.finished(): # Judge SLO per requestok = (r.ttft_ms <= slo.ttft_ms andr.tpot_ms_p99 <= slo.tpot_ms andr.e2e_ms <= slo.e2e_ms)if ok:accepted += 1produced += r.output_tokenselse:rejected += 1assert engine.kv_cache_usage() < 1.0 # No queue overflow triggeredwall = engine.wall_time_s()return {"tokens_per_s": produced / wall,"slo_pass_rate": accepted / (accepted + rejected),"p99_ttft_ms": engine.p99_ttft(),}
For tuning, use a “gradually increase load to find the knee” approach: start from a small QPS, increase step by step until the SLO pass rate drops below the business threshold (for example, 99%). The tokens_per_s at that knee is the capacity baseline you can commit to externally.
3. Parallelism Strategy: Decide the “Cut” Before Tuning
The parallelism option space on a single 8-card node isn’t large, but the combinations have far-reaching impact. The core constraint is communication frequency: tensor parallelism (TP) performs collective communication on every operator’s forward and backward pass, making it the most communication-intensive; pipeline parallelism (PP) communicates only at stage boundaries but introduces bubbles; expert parallelism (EP) in MoE scenarios dispatches tokens via all-to-all, which is extremely sensitive to bandwidth; data parallelism (DP) is multi-replica with almost no intra-node communication overhead.
The practical decision logic:
- Can a single layer’s weights fit in one card’s memory: if yes, prefer reducing the TP degree, because each step down in TP significantly reduces communication volume; if no, take a higher TP degree as the floor, usually a power of 2 that evenly divides the number of attention heads.
- Do compute or memory hit the bottleneck first: when memory is tight (the KV cache can’t hold enough concurrency), expanding TP shares both weights and KV; when compute is tight, expanding DP replicas is more cost-effective.
- Evaluate EP separately for MoE models: expert parallelism can significantly reduce per-card weight occupancy, but the all-to-all communication volume is strongly correlated with the activated token routing distribution, and load imbalance causes long-tail waiting.
- Use PP cautiously for online inference: bubbles raise TTFT, so it is better suited to offline batch processing and very long contexts.
A common goal is to make the product TP × PP × EP times the DP replica count exactly equal the card count, avoiding cards wasted on a non-divisible combination.
4. Communication Optimization: Hide Waiting Inside Computation
Optimizing communication rarely relies on “changing parameters”; it is more about eliminating three types of waste:
- Topology mismatch: bind high-frequency communication groups to the set of cards with the highest interconnect bandwidth, letting slow cross-node/cross-NUMA links carry low-frequency communication; measure the effective bandwidth between each card pair rather than trusting nominal values.
- Lack of overlap: when computation and collective communication run serially, communication time is fully exposed on the critical path. Enabling compute-communication overlap (splitting communication into chunks and pipelining them with subsequent computation) usually significantly compresses single-step time.
- Redundant copies: unnecessary host-device synchronization, frequent dynamic-shape-triggered recompilation, and tensors whose layout is repeatedly transformed between operators all eat up the gains from “communication optimization.”
The diagnosis method is direct: capture a timeline of single-step latency, then look at the proportion taken by communication kernels and whether they overlap with the computation kernel timeline. If communication accounts for more than ten to twenty percent and is fully serial, this is usually the highest-return stop.
5. Compilation and Operator Optimization: From Eager Mode to Graph Mode
Eager (single-operator) mode is convenient for debugging, but it dispatches a large number of small operators every time, and scheduling overhead becomes significant under high-concurrency decoding. After switching to graph mode, the compiler can perform operator fusion, constant folding, memory reuse, and scheduling reordering. Typical gains come from:
- fusion of the attention path (avoiding repeated read/write of intermediate results to memory);
- merging element-wise operators such as normalization, residual, and activation with their neighbors;
- efficient kernel selection for quantized GEMM;
- a static memory pool that reduces repeated allocation and deallocation.
Two pitfalls deserve attention. First, dynamic shapes cause repeated recompilation — bucket by common lengths and pad within each bucket, converging the number of shapes to single digits. Second, cold-start cost is high — graph compilation and service loading must be counted in the warmup process, and the jitter of the first request should be excluded from SLO statistics during capacity assessment.
# Generic inference service parameter skeleton (keys are generic concepts; replace them with your actual inference engine)model:dtype: bfloat16weight_quant: null # null | int8 | fp8kv_cache_quant: null # null | int8 | fp8parallel:tensor_parallel: 4 # Constrained by inter-card interconnect bandwidthpipeline_parallel: 1 # Use cautiously for online inference; bubbles raise TTFTexpert_parallel: 2 # Only needed for MoE modelsdata_parallel_replicas: 2 # 4 × 1 × 2 = 8 cards, fully utilizedruntime:execution_mode: graph # graph | eager; prefer graphshape_buckets: [1024, 2048, 4096, 8192]max_num_seqs: 128max_num_batched_tokens: 16384enable_chunked_prefill: trueprefill_chunk_size: 2048block_size: 128 # KV cache paging block sizeprefix_cache: trueschedule:preemption_mode: recompute # recompute | swapadmission_timeout_ms: 800
6. KV Cache and Batch Scheduling: The Real Determinant of the Concurrency Ceiling
During decoding, every token generated reads all historical KV, so memory bandwidth is often the bottleneck, and the concurrency ceiling is determined by KV cache capacity. A rough estimate:
KV memory ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_precision × seq_len × concurrency
This formula reveals three optimization paths:
- Compress the per-token KV cost: adopting grouped-query attention (GQA/MLA-type structures) and KV precision quantization (int8/fp8) can multiply the concurrency that fits.
- Paged KV management: allocate memory in fixed-size blocks to reduce fragmentation, bringing memory utilization under mixed long/short requests closer to the theoretical ceiling.
- Prefix reuse: if system prompts, multi-turn conversation history, or shared few-shot templates hit the cache, repeated prefill can be skipped, directly reducing both TTFT and compute consumption. This is especially beneficial for customer-service and assistant workloads with fixed system prompts.
The most effective trick on the batch-scheduling side is chunked prefill: split a long prompt’s prefill into small chunks under a fixed token budget and mix them into batches with decoding requests. It avoids two typical failures — all currently decoding requests stalling when a long prompt arrives (causing TPOT long-tail spikes), and decoding requests being starved. Combined with admission queuing (new requests above a threshold first queue and return queue metrics) and a preemption policy (recompute or swap), it can prioritize protecting the SLO over chasing peak throughput under overload.
# Pseudocode: continuous batching scheduling with a token budgetdef schedule_step(engine, token_budget, slo):batch = []# 1) First guarantee already-decoding requests to avoid ITL long tailfor seq in engine.decoding():batch.append(seq.chunk(n=1))# 2) Use the remaining budget for prefill chunksbudget = token_budget - len(batch)while budget > 0 and engine.waiting():req = engine.pop_waiting()n = min(budget, PREFILL_CHUNK)batch.append(req.chunk(n=n))budget -= nif req.prefill_done():engine.promote_to_decoding(req)# 3) When memory is insufficient, preempt by policy, prioritizing high-priority trafficif engine.kv_cache_needed(batch) > engine.kv_cache_free():engine.preempt(victims=engine.low_priority_decoding())return batch # Hand off to the execution engine for unified forward/backward
7. Quantization and Precision: Trading Controllable Error for Capacity and Bandwidth
Quantization is one of the highest single-point gains available, but the decision must be layered:
- Weight-only quantization W8A8/W4A16: memory and load bandwidth drop and decoding speeds up noticeably, with relatively controllable precision risk;
- Weight + activation quantization: further leverages low-precision compute units but is sensitive to activation outliers, requiring a calibration set and per-layer sensitivity analysis;
- KV cache quantization: orthogonal to weight quantization, with its gain showing up in the number of concurrent requests that fit; it is usually the first thing to try.
For MoE models, additionally note the amplification effect of routing on quantization error: the activation distribution of a small number of experts may deviate from the calibration set, so evaluate quantization error grouped by expert and fall back to high precision for sensitive experts.
Validation can’t rely on overall perplexity alone. At minimum, cover: task-level metric comparison on a real business sample set, consistency checks for long-text generation, and numerical stability at extreme lengths (whether NaN/Inf appears or there is obvious repetitive degradation). If any of these regresses beyond an agreed threshold, fall back to the previous precision tier.
8. Layered Profiling and the Tuning Loop
When throughput falls short, troubleshooting in an “outside-in” order saves a lot of blind trial and error:
- Are you stuck at admission: a high proportion of queue-wait time means concurrency has exceeded the system’s service capability; optimizing kernels is useless here, so adjust the parallelism strategy or reduce per-request resource usage;
- Are you stuck on memory: KV usage persistently topping out means the concurrency ceiling is limited; prioritize KV quantization or raising the TP degree;
- Are you stuck on communication: a high serial proportion of communication in single-step time means you should do topology binding and compute-communication overlap;
- Are you stuck on compute/bandwidth utilization: observe actual MFU and memory bandwidth utilization; if they are low, check whether operators take the fused path and whether shapes recompile frequently;
- Are you stuck on scheduling: observe whether the actual batch size is far below
max_num_seqsand whether prefill blocks decoding; tune the chunk size and the admission threshold.
For parameter search, do coordinate descent first and then a local grid: fix the other parameters, sweep the three items most sensitive to throughput — max_num_batched_tokens, prefill_chunk_size, max_num_seqs — and jointly fine-tune after finding the knee. All experiment records must be bound to the image version, the card model/driver version, and the model weight checksum; otherwise the results aren’t reproducible.
9. Consolidating Optimization into a Reproducible Deployment Plan
If the results of one tuning session stay on someone’s machine, they can’t become an organizational capability. Store each configuration as a structured entry containing at least:
- Applicable model and hardware: model architecture type (dense/MoE), parameter scale, target card model and count;
- Runtime environment: inference engine and image version, accelerator driver and communication library version;
- Resources and topology: TP/PP/EP/DP combination, memory usage estimate, interconnect requirements;
- Complete parameter list: scheduling, quantization, caching, and compilation-related switches;
- Performance data: SLO definition, request distribution, and the effective throughput and percentile latency for both baseline and optimized plans;
- Validation method: measurement scripts, decision thresholds, regression process.
With this structure, teams can do three things: find the right choice by reference when a new workload launches; automatically regress the baseline after a hardware or engine upgrade, catching performance regressions at the first sign; and turn “experience-based tuning” into a reviewable, inheritable engineering asset. The optimization paths discussed here — parallelism re-planning, communication overlap, graph-mode compilation, chunked prefill and KV paging, and layered quantization — are mutually orthogonal; stacking them item by item, a throughput gain of dozens of percentage points is a common result, but each must be individually validated on your own workload so that a regression in one config doesn’t cancel out the overall improvement.
10. Summary
In high-load online inference, effective throughput under SLO constraints is the core metric for gauging the maturity of an inference stack. In terms of optimization order: first define the metrics and baseline, then touch the parallelism strategy and the memory budget, then communication and compilation, and finally scheduling and quantization fine-tuning; regress with a unified script at every step, avoiding the mistake of treating incidental gains as architectural advantages. Only by consolidating these configurations and data into a standardized deployment plan entry does performance optimization turn from a one-time sprint into a sustainable engineering capability.
