LLM Inference Optimization on a Single 8-Accelerator Node: A Path to Higher Throughput under SLO Constraints

    This article walks through optimizing large language model inference on a single node with 8 accelerators to improve throughput under SLO (service-level objective) constraints. It shows how to define effective throughput and establish a baseline, then covers parallelism strategy, communication overlap, graph-mode compilation, KV cache management with chunked prefill, and layered quantization. It also presents a top-down profiling method for locating bottlenecks and a framework for recording each configuration as a reproducible deployment plan, turning performance tuning into a repeatable engineering capability rather than a one-time sprint.

    1. Why Can Throughput Vary by 50% with the Same Hardware and Model?

    In production compute-cost accounting, the metric that actually matters is never “how fast a card runs.” It is “how many tokens a single node can stably serve under a latency constraint the business can tolerate.” When the accelerator card, the model weights, and the network parameters are all fixed, throughput can still vary by dozens of percentage points. The reason usually isn’t the hardware itself but the quality of the inference stack’s configuration: whether the parallelism scheme matches the inter-card interconnect topology, whether collective communication is effectively overlapped, whether computation follows a compiler-optimized fused path, whether batch scheduling gets blocked by overly long prefills, and whether the KV cache capacity estimate leaves enough headroom for concurrency.

    So the right starting point for inference optimization isn’t “tuning parameters” but establishing a baseline: first run a conservative, generic configuration that doesn’t rely on special features, record its throughput under a fixed SLO, then introduce optimization techniques one by one, regressing each with the same measurement script. Optimization without a baseline can’t tell you which variable produced the gain, nor whether a performance change after a version upgrade is a regression or an improvement.

    It’s worth noting that any throughput improvement is strongly correlated with model architecture (dense/MoE), weight precision, input/output length distribution, and inter-card interconnect bandwidth. Cross-hardware or cross-model comparisons are meaningless. Every number must be measured on your own target workload.

    2. Define the Metrics First: SLO and “Effective Throughput”

    Most performance arguments are really arguments about metric definitions. Before experiments begin, lock down these four:

    • TTFT (time to first token): reflects prefill latency and directly determines the experience in interactive scenarios;
    • TPOT / ITL (inter-token latency): reflects the steady-state decoding speed and determines the smoothness of long-text generation;
    • End-to-end latency: TTFT + (output length − 1) × TPOT, used as a backstop constraint for long requests;
    • Effective throughput: the number of tokens produced per unit time by requests that meet the SLO, not the total tokens from all requests.

    The fourth point is the most easily overlooked. Without SLO filtering, pushing concurrency to the limit always yields a nice peak throughput, but a significant fraction of those requests have already timed out — and such a number is useless for capacity planning.

    You must also fix the request distribution: the mean and percentiles of input/output lengths, the arrival process (Poisson or fixed QPS), and whether prefixes are shared. A slight shift in the length distribution completely changes KV cache occupancy and the benefit structure of batching.

    1. # Pseudocode: measuring effective throughput under a fixed SLO and request distribution
    2. def measure_effective_throughput(engine, trace, slo, target_qps):
    3. engine.warmup(num_requests=64)
    4. engine.reset_metrics()
    5. accepted, rejected, produced = 0, 0, 0
    6. for req in poisson_arrivals(trace, target_qps): # Inject according to the real arrival process
    7. engine.submit(req)
    8. while engine.inflight() > 0:
    9. engine.step()
    10. for r in engine.finished(): # Judge SLO per request
    11. ok = (r.ttft_ms <= slo.ttft_ms and
    12. r.tpot_ms_p99 <= slo.tpot_ms and
    13. r.e2e_ms <= slo.e2e_ms)
    14. if ok:
    15. accepted += 1
    16. produced += r.output_tokens
    17. else:
    18. rejected += 1
    19. assert engine.kv_cache_usage() < 1.0 # No queue overflow triggered
    20. wall = engine.wall_time_s()
    21. return {
    22. "tokens_per_s": produced / wall,
    23. "slo_pass_rate": accepted / (accepted + rejected),
    24. "p99_ttft_ms": engine.p99_ttft(),
    25. }

    For tuning, use a “gradually increase load to find the knee” approach: start from a small QPS, increase step by step until the SLO pass rate drops below the business threshold (for example, 99%). The tokens_per_s at that knee is the capacity baseline you can commit to externally.

    3. Parallelism Strategy: Decide the “Cut” Before Tuning

    The parallelism option space on a single 8-card node isn’t large, but the combinations have far-reaching impact. The core constraint is communication frequency: tensor parallelism (TP) performs collective communication on every operator’s forward and backward pass, making it the most communication-intensive; pipeline parallelism (PP) communicates only at stage boundaries but introduces bubbles; expert parallelism (EP) in MoE scenarios dispatches tokens via all-to-all, which is extremely sensitive to bandwidth; data parallelism (DP) is multi-replica with almost no intra-node communication overhead.

    The practical decision logic:

    1. Can a single layer’s weights fit in one card’s memory: if yes, prefer reducing the TP degree, because each step down in TP significantly reduces communication volume; if no, take a higher TP degree as the floor, usually a power of 2 that evenly divides the number of attention heads.
    2. Do compute or memory hit the bottleneck first: when memory is tight (the KV cache can’t hold enough concurrency), expanding TP shares both weights and KV; when compute is tight, expanding DP replicas is more cost-effective.
    3. Evaluate EP separately for MoE models: expert parallelism can significantly reduce per-card weight occupancy, but the all-to-all communication volume is strongly correlated with the activated token routing distribution, and load imbalance causes long-tail waiting.
    4. Use PP cautiously for online inference: bubbles raise TTFT, so it is better suited to offline batch processing and very long contexts.

    A common goal is to make the product TP × PP × EP times the DP replica count exactly equal the card count, avoiding cards wasted on a non-divisible combination.

    4. Communication Optimization: Hide Waiting Inside Computation

    Optimizing communication rarely relies on “changing parameters”; it is more about eliminating three types of waste:

    • Topology mismatch: bind high-frequency communication groups to the set of cards with the highest interconnect bandwidth, letting slow cross-node/cross-NUMA links carry low-frequency communication; measure the effective bandwidth between each card pair rather than trusting nominal values.
    • Lack of overlap: when computation and collective communication run serially, communication time is fully exposed on the critical path. Enabling compute-communication overlap (splitting communication into chunks and pipelining them with subsequent computation) usually significantly compresses single-step time.
    • Redundant copies: unnecessary host-device synchronization, frequent dynamic-shape-triggered recompilation, and tensors whose layout is repeatedly transformed between operators all eat up the gains from “communication optimization.”

    The diagnosis method is direct: capture a timeline of single-step latency, then look at the proportion taken by communication kernels and whether they overlap with the computation kernel timeline. If communication accounts for more than ten to twenty percent and is fully serial, this is usually the highest-return stop.

    5. Compilation and Operator Optimization: From Eager Mode to Graph Mode

    Eager (single-operator) mode is convenient for debugging, but it dispatches a large number of small operators every time, and scheduling overhead becomes significant under high-concurrency decoding. After switching to graph mode, the compiler can perform operator fusion, constant folding, memory reuse, and scheduling reordering. Typical gains come from:

    • fusion of the attention path (avoiding repeated read/write of intermediate results to memory);
    • merging element-wise operators such as normalization, residual, and activation with their neighbors;
    • efficient kernel selection for quantized GEMM;
    • a static memory pool that reduces repeated allocation and deallocation.

    Two pitfalls deserve attention. First, dynamic shapes cause repeated recompilation — bucket by common lengths and pad within each bucket, converging the number of shapes to single digits. Second, cold-start cost is high — graph compilation and service loading must be counted in the warmup process, and the jitter of the first request should be excluded from SLO statistics during capacity assessment.

    1. # Generic inference service parameter skeleton (keys are generic concepts; replace them with your actual inference engine)
    2. model:
    3. dtype: bfloat16
    4. weight_quant: null # null | int8 | fp8
    5. kv_cache_quant: null # null | int8 | fp8
    6. parallel:
    7. tensor_parallel: 4 # Constrained by inter-card interconnect bandwidth
    8. pipeline_parallel: 1 # Use cautiously for online inference; bubbles raise TTFT
    9. expert_parallel: 2 # Only needed for MoE models
    10. data_parallel_replicas: 2 # 4 × 1 × 2 = 8 cards, fully utilized
    11. runtime:
    12. execution_mode: graph # graph | eager; prefer graph
    13. shape_buckets: [1024, 2048, 4096, 8192]
    14. max_num_seqs: 128
    15. max_num_batched_tokens: 16384
    16. enable_chunked_prefill: true
    17. prefill_chunk_size: 2048
    18. block_size: 128 # KV cache paging block size
    19. prefix_cache: true
    20. schedule:
    21. preemption_mode: recompute # recompute | swap
    22. admission_timeout_ms: 800

    6. KV Cache and Batch Scheduling: The Real Determinant of the Concurrency Ceiling

    During decoding, every token generated reads all historical KV, so memory bandwidth is often the bottleneck, and the concurrency ceiling is determined by KV cache capacity. A rough estimate:

    1. KV memory ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_precision × seq_len × concurrency

    This formula reveals three optimization paths:

    1. Compress the per-token KV cost: adopting grouped-query attention (GQA/MLA-type structures) and KV precision quantization (int8/fp8) can multiply the concurrency that fits.
    2. Paged KV management: allocate memory in fixed-size blocks to reduce fragmentation, bringing memory utilization under mixed long/short requests closer to the theoretical ceiling.
    3. Prefix reuse: if system prompts, multi-turn conversation history, or shared few-shot templates hit the cache, repeated prefill can be skipped, directly reducing both TTFT and compute consumption. This is especially beneficial for customer-service and assistant workloads with fixed system prompts.

    The most effective trick on the batch-scheduling side is chunked prefill: split a long prompt’s prefill into small chunks under a fixed token budget and mix them into batches with decoding requests. It avoids two typical failures — all currently decoding requests stalling when a long prompt arrives (causing TPOT long-tail spikes), and decoding requests being starved. Combined with admission queuing (new requests above a threshold first queue and return queue metrics) and a preemption policy (recompute or swap), it can prioritize protecting the SLO over chasing peak throughput under overload.

    1. # Pseudocode: continuous batching scheduling with a token budget
    2. def schedule_step(engine, token_budget, slo):
    3. batch = []
    4. # 1) First guarantee already-decoding requests to avoid ITL long tail
    5. for seq in engine.decoding():
    6. batch.append(seq.chunk(n=1))
    7. # 2) Use the remaining budget for prefill chunks
    8. budget = token_budget - len(batch)
    9. while budget > 0 and engine.waiting():
    10. req = engine.pop_waiting()
    11. n = min(budget, PREFILL_CHUNK)
    12. batch.append(req.chunk(n=n))
    13. budget -= n
    14. if req.prefill_done():
    15. engine.promote_to_decoding(req)
    16. # 3) When memory is insufficient, preempt by policy, prioritizing high-priority traffic
    17. if engine.kv_cache_needed(batch) > engine.kv_cache_free():
    18. engine.preempt(victims=engine.low_priority_decoding())
    19. return batch # Hand off to the execution engine for unified forward/backward

    7. Quantization and Precision: Trading Controllable Error for Capacity and Bandwidth

    Quantization is one of the highest single-point gains available, but the decision must be layered:

    • Weight-only quantization W8A8/W4A16: memory and load bandwidth drop and decoding speeds up noticeably, with relatively controllable precision risk;
    • Weight + activation quantization: further leverages low-precision compute units but is sensitive to activation outliers, requiring a calibration set and per-layer sensitivity analysis;
    • KV cache quantization: orthogonal to weight quantization, with its gain showing up in the number of concurrent requests that fit; it is usually the first thing to try.

    For MoE models, additionally note the amplification effect of routing on quantization error: the activation distribution of a small number of experts may deviate from the calibration set, so evaluate quantization error grouped by expert and fall back to high precision for sensitive experts.

    Validation can’t rely on overall perplexity alone. At minimum, cover: task-level metric comparison on a real business sample set, consistency checks for long-text generation, and numerical stability at extreme lengths (whether NaN/Inf appears or there is obvious repetitive degradation). If any of these regresses beyond an agreed threshold, fall back to the previous precision tier.

    8. Layered Profiling and the Tuning Loop

    When throughput falls short, troubleshooting in an “outside-in” order saves a lot of blind trial and error:

    1. Are you stuck at admission: a high proportion of queue-wait time means concurrency has exceeded the system’s service capability; optimizing kernels is useless here, so adjust the parallelism strategy or reduce per-request resource usage;
    2. Are you stuck on memory: KV usage persistently topping out means the concurrency ceiling is limited; prioritize KV quantization or raising the TP degree;
    3. Are you stuck on communication: a high serial proportion of communication in single-step time means you should do topology binding and compute-communication overlap;
    4. Are you stuck on compute/bandwidth utilization: observe actual MFU and memory bandwidth utilization; if they are low, check whether operators take the fused path and whether shapes recompile frequently;
    5. Are you stuck on scheduling: observe whether the actual batch size is far below max_num_seqs and whether prefill blocks decoding; tune the chunk size and the admission threshold.

    For parameter search, do coordinate descent first and then a local grid: fix the other parameters, sweep the three items most sensitive to throughput — max_num_batched_tokens, prefill_chunk_size, max_num_seqs — and jointly fine-tune after finding the knee. All experiment records must be bound to the image version, the card model/driver version, and the model weight checksum; otherwise the results aren’t reproducible.

    9. Consolidating Optimization into a Reproducible Deployment Plan

    If the results of one tuning session stay on someone’s machine, they can’t become an organizational capability. Store each configuration as a structured entry containing at least:

    • Applicable model and hardware: model architecture type (dense/MoE), parameter scale, target card model and count;
    • Runtime environment: inference engine and image version, accelerator driver and communication library version;
    • Resources and topology: TP/PP/EP/DP combination, memory usage estimate, interconnect requirements;
    • Complete parameter list: scheduling, quantization, caching, and compilation-related switches;
    • Performance data: SLO definition, request distribution, and the effective throughput and percentile latency for both baseline and optimized plans;
    • Validation method: measurement scripts, decision thresholds, regression process.

    With this structure, teams can do three things: find the right choice by reference when a new workload launches; automatically regress the baseline after a hardware or engine upgrade, catching performance regressions at the first sign; and turn “experience-based tuning” into a reviewable, inheritable engineering asset. The optimization paths discussed here — parallelism re-planning, communication overlap, graph-mode compilation, chunked prefill and KV paging, and layered quantization — are mutually orthogonal; stacking them item by item, a throughput gain of dozens of percentage points is a common result, but each must be individually validated on your own workload so that a regression in one config doesn’t cancel out the overall improvement.

    10. Summary

    In high-load online inference, effective throughput under SLO constraints is the core metric for gauging the maturity of an inference stack. In terms of optimization order: first define the metrics and baseline, then touch the parallelism strategy and the memory budget, then communication and compilation, and finally scheduling and quantization fine-tuning; regress with a unified script at every step, avoiding the mistake of treating incidental gains as architectural advantages. Only by consolidating these configurations and data into a standardized deployment plan entry does performance optimization turn from a one-time sprint into a sustainable engineering capability.