LLM Inference Serving: Optimizing Throughput, Latency, Quality, and Cost

    As large language model capabilities grow, engineering teams must move from "it runs" to "it runs reliably, efficiently, and under control." This article lays out a practical engineering approach for serving general-purpose LLM inference. It covers how to turn performance expectations into measurable SLOs, how a request moves through the prefill and decode phases and where each phase bottlenecks, how to size and manage the KV cache (static allocation, paged attention, tiered swapping, prefix caching), how scheduling and continuous batching shape throughput versus tail latency, how quantization, kernel fusion, speculative decoding, and structured output constraints trade accuracy for speed, and how to benchmark, monitor, and alert on a live service. It also addresses long-context challenges and a phased rollout plan with acceptance criteria, rollback conditions, and a go-live checklist.

    Introduction: From “It Runs” to “It Can Be Served”

    As model capabilities keep improving, the real question for engineering teams is not just “what more can the model do,” but can it be run stably, efficiently, and under control?

    Longer context windows, larger parameter counts, more tool calls, and multi-turn conversations all convert capability gains into pressure on GPU memory, bandwidth, scheduling, and operations. Without a serving layer, those capabilities rarely turn into a consistent user experience. The goal of inference serving is to turn “it can infer” into a measurable balance of throughput, latency, quality, and cost, and to give the system observability, rollback, and scale-out capabilities.

    This article focuses on general-purpose LLM inference serving and lays out a practical engineering method: how to define targets, how to analyze the request lifecycle, how to manage the KV cache, how to schedule and batch, how to choose acceleration techniques, how to load test and monitor, and how to roll it out in stages.

    1. Define Goals First: SLOs, Load Testing, and Choosing Optimizations

    The first step in building a serving platform is not picking a framework, but turning “fast,” “stable,” and “cheap” into measurable numbers. Common questions in practice are: How should SLOs be defined? How should load testing be designed? How should quantization and speculative decoding be chosen? How should concurrency be estimated for long contexts?

    LLM inference serving typically has to satisfy four goals at once:

    • Throughput: tokens or requests processed per unit of time; reflects resource utilization.
    • Latency: the speed users perceive, especially time to first token (TTFT) and output rate.
    • Quality: accuracy, stability, and consistency of outputs; determines business results.
    • Cost: compute resources consumed per unit of business volume; determines whether the service can scale.

    These four goals constrain one another, forming the “impossible triangle” of inference serving. For example: a larger batch usually improves throughput but can lengthen queueing delays; more aggressive quantization can reduce memory usage and increase speed but may cost accuracy; longer contexts improve capability but significantly increase KV cache and scheduling pressure. Serving work should therefore start by defining SLOs from the business scenario, and only then select optimization techniques.

    Workload Typical characteristics Core metrics Optimization focus
    Online chat / assistants Short inputs, longer outputs TTFT, output rate Scheduling latency, continuous batching
    Long-document Q&A Long inputs, short outputs Prefill latency, memory usage Context trimming, chunked prefill
    Batch content generation Offline jobs, no real-time requirement Overall throughput, cost per unit Batch size, utilization
    High-concurrency API service High volume, mixed lengths P95/P99 latency, availability Queue management, rate limiting, autoscaling

    For each scenario, define one primary goal and two constraints. For an online support assistant, for instance, make P95 TTFT the primary goal, while constraining output rate to stay above a threshold and error rate to stay below another. Once the metrics are fixed, capacity planning, load testing, and optimization all have a basis for judgment.

    2. Anatomy of an Inference Request: Two Phases, Two Bottlenecks

    Autoregressive LLM generation is usually divided into two phases:

    1. Prefill: all input tokens are processed in parallel, hidden states are computed for every position, and the KV cache is populated. This phase is dominated by matrix-matrix operations, is compute-bound, and directly determines TTFT.
    2. Decode: output tokens are generated one at a time. Each new token requires computing a single new query vector but reading the entire historical KV cache and the model weights. This phase is dominated by matrix-vector operations, is memory-bandwidth-bound, and determines output rate and total per-request time.

    A simplified view of how the two phases differ computationally:

    1. Arithmetic intensity ≈ FLOPs per parameter / bytes read per parameter
    2. Prefill (sequence length S): ≈ 2S (depends on input length)
    3. Decode (batch size B): ≈ B (depends on number of concurrent sequences)

    Note that these arithmetic intensity figures are order-of-magnitude illustrations only. They ignore byte units, activations, temporary buffers, and framework overhead, and should not be used as exact formulas.

    A few engineering rules follow from this:

    • Increasing batch size mainly improves Decode efficiency and yields limited benefit for Prefill, where compute is already the bottleneck.
    • When long-input and long-output requests share the same pool, they tend to block each other, and TTFT can be dragged up by long requests.
    • Input length, output length, and concurrency have to be planned together rather than tuning a single variable.

    3. The KV Cache: The First Constraint in the Long-Context Era

    3.1 Estimating Memory Capacity

    Inference memory consists of three main parts: model weights, the KV cache, and runtime overhead. The KV cache grows linearly with concurrency and sequence length, which makes it the key variable limiting service scale.

    It can be approximated as:

    1. KV bytes ≈ 2 × layers × KV heads × head_dim × sequence length × concurrency × bytes per element

    For 32 layers, 8 KV heads, head dimension 128, stored in FP16:

    1. 2 × 32 × 8 × 128 × 2 ≈ 131072 bytes/token ≈ 0.125 MiB/token

    That means roughly 128 MB of KV cache per 1K tokens per sequence. Estimates at different context lengths:

    Context length (single sequence) KV cache per sequence KV cache for 100 concurrent
    8K ~1 GB ~100 GB
    32K ~4 GB ~400 GB
    128K ~16 GB ~1.6 TB

    One point deserves emphasis: at a 128K context, even 100 concurrent sequences can push the KV cache into the terabyte range. Different layer counts, KV head counts, or precisions will change the numbers, but the order-of-magnitude reasoning stays the same. If the model uses an MoE structure, weights can be distributed across experts, but the KV cache usually still has to cover every decode layer and does not shrink automatically with expert sharding.

    3.2 Three KV Cache Management Strategies

    • Static reservation: pre-allocate contiguous memory for the maximum length. Simple to implement, but wasteful when lengths vary widely, and fragmentation caps concurrency.
    • Paged management (PagedAttention): borrows the operating system paging idea by splitting the KV cache into fixed-size blocks allocated on demand. This substantially reduces internal fragmentation and raises both memory utilization and achievable concurrency.
    • Tiered caching (swap/recompute): when memory runs short, less active cache entries are swapped out to host memory or remote storage, then swapped back or recomputed when needed. It buys concurrency at the cost of transfer or recomputation.

    Prefix caching also matters. Shared prefixes such as system prompts, fixed documents, and multi-turn history can be reused across requests. On a prefix cache hit, the system can skip repeated prefill and lower TTFT.

    3.3 A KV Cache Estimation Script

    For capacity planning, start by estimating KV cache demand across context lengths and concurrency levels:

    1. def kv_cache_gib(num_layers, num_kv_heads, head_dim,
    2. seq_len, concurrency, bytes_per_elem=2):
    3. """Estimate KV cache memory usage in GiB; bytes_per_elem=2 means half precision."""
    4. per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_elem
    5. return per_token * seq_len * concurrency / (1024 ** 3)
    6. # Example: 32 layers, 8 KV heads, head_dim 128, half precision, 100 concurrent
    7. for length in (8192, 32768, 131072):
    8. need = kv_cache_gib(32, 8, 128, length, concurrency=100)
    9. print(f"Context {length // 1024}K: 100 concurrent needs about {need:.1f} GiB of KV cache")

    Comparing this against “total memory − weight footprint − runtime overhead” quickly tells you the concurrency ceiling your hardware can support, instead of discovering the shortfall after launch.

    4. Scheduling and Batching: The Key to Throughput and Tail Latency

    4.1 The Limits of Static Batching

    The simplest batching approach is to “assemble a batch, compute it together, and only start the next batch after all sequences finish.” The problem is that once the short requests in a batch are done, they must wait for the longest sequence, while the GPU does wasted work for those few long sequences. The more dispersed the request length distribution, the greater the waste.

    4.2 Continuous Batching

    Continuous batching refines the scheduling granularity from “the whole request” to “each iteration.” The core idea: as soon as a sequence finishes, its resources are released immediately and a new request is admitted, keeping the GPU working as continuously as possible.

    1. while not all_requests_done:
    2. # 1. Release cache blocks held by finished sequences
    3. batch.remove_finished()
    4. # 2. When under memory pressure, preempt some low-priority sequences
    5. while memory_pressure(batch):
    6. victim = scheduler.select_preemptible(batch)
    7. victim.swap_out_or_recompute()
    8. # 3. Admit new requests from the waiting queue
    9. for req in waiting_queue.pop_by_priority(batch.free_slots()):
    10. batch.add(req)
    11. # 4. Run one forward pass, producing one token for each unfinished sequence
    12. tokens = model.forward_step(batch)
    13. scheduler.commit(batch, tokens)

    4.3 Preemption and Recovery

    When active sequences approach the cache limit, the scheduler can preempt some of them. There are two common recovery strategies:

    • Swap out / swap in: move the KV cache to host memory and bring it back on resume. Suited to sequences with long outputs that are expected to keep running for a while.
    • Drop and recompute: free the KV cache and re-run prefill on resume. Simpler to implement, but incurs duplicate computation.

    This is fundamentally a trade-off between “transfer cost” and “compute cost.” The scheduler should choose dynamically based on sequence length, expected remaining output, memory pressure, and hardware bandwidth.

    4.4 Multiple Queues and Priorities

    A single queue can rarely serve every request type well. A more robust approach is to split queues by business characteristics:

    • Interactive request queue: high priority, short timeout, strict TTFT.
    • Long-task queue: low priority, queuing allowed, optimized for throughput and cost.
    • Batch queue: fully offline, scheduled flexibly during off-peak hours.

    Combined with timeouts, rate limiting, circuit breaking, and input length validation, this prevents a single oversized input or abnormal output from taking down the whole service.

    5. Compute Acceleration: Quantization, Kernel Fusion, and Speculative Decoding

    5.1 Quantization

    Decode is memory-bandwidth-bound, and reading weights usually dominates single-step time. Lowering weight precision reduces both memory footprint and bytes read.

    • Weight quantization: compress weights into lower-precision formats such as INT8 or INT4.
    • Activation and KV cache quantization: further reduce runtime and cache overhead, though outlier effects need attention.
    • Group-wise / channel-wise quantization: determine quantization parameters over smaller groups to reduce the accuracy impact of outliers.

    The gains from quantization must be grounded in evaluation. On a fixed regression set, compare task success rate, output consistency, and human review before and after quantization, rather than looking only at memory savings.

    5.2 Kernel Fusion and Memory-Efficient Attention

    A standard attention implementation explicitly materializes an attention matrix shaped “batch × heads × sequence length × sequence length,” whose memory footprint grows quadratically with context length. Memory-efficient fused attention kernels use tiled computation and online normalization to avoid materializing that large matrix, bringing attention memory overhead closer to linear growth while reducing memory accesses. For long-context scenarios, this kind of optimization is often a prerequisite for running the service stably at all.

    5.3 Speculative Decoding

    The core of speculative decoding is “guess first, verify after”:

    1. A lightweight draft model generates several candidate tokens.
    2. The main model performs one parallel forward pass over the candidates.
    3. An acceptance criterion guarantees the final output distribution matches decoding with the main model alone; rejected positions fall back to resampling.

    Because one main-model forward pass can confirm multiple tokens, output speed increases, and in theory output quality is unchanged. The benefit depends on the draft hit rate: code, structured text, and templated content typically accelerate better.

    5.4 Structured Output Constraints

    When the business requires valid JSON or another fixed format, grammar constraints can be introduced during decoding to mask tokens that violate the syntax. This reduces format errors, downstream parsing failures, and retry requests, improving service stability.

    5.5 Choosing an Acceleration Technique

    Technique Suitable scenario Main benefit Cost and risk How to validate
    Continuous batching High volume with varying lengths Higher GPU utilization and throughput Complex scheduling; long requests may interfere with short ones Compare TTFT P95, throughput, preemption count
    Quantization Tight memory, bandwidth-limited Less memory, higher speed, lower cost Possible accuracy loss Regression set, task success rate, human review
    Speculative decoding Predictable output, latency-sensitive Higher output rate Limited gains when draft hit rate is unstable Compare TPOT, acceptance rate, quality regression
    Prefix caching Repeated system prompts, multi-turn history Lower TTFT Complex cache hit management Hit rate, prefill time, memory usage

    6. Long Context: New Capability, New Challenges

    Extending the context window improves long-document, multi-turn, and complex-instruction handling, but it also amplifies memory, bandwidth, and scheduling pressure in one place:

    • Prefill cost grows with length: hundreds of thousands of input tokens can take seconds, easily blocking subsequent requests.
    • Attention cost remains significant: even with memory-efficient implementations, long-context compute volume is still substantial.
    • KV cache footprint is enormous: very long contexts sharply reduce achievable concurrency.
    • Middle-of-context information loss: when relevant information sits in the middle of a long context, model utilization may drop.

    Engineering options include:

    1. Retrieval augmentation: split, retrieve, and rerank very long inputs, and send only the most relevant fragments.
    2. Chunked prefill: split long inputs into chunks computed in stages, smoothing the memory peak and interleaving with decode requests.
    3. Context compression and summarization: periodically compress earlier history in multi-turn conversations to free cache.
    4. Prefix cache reuse: share system prompts, fixed documents, and history prefixes to avoid repeated prefill.

    7. Evaluation: Making Optimization Measurable and Repeatable

    The biggest risk in optimization is “it feels faster.” A reproducible evaluation system should include at least three layers:

    • Component level: numerical consistency checks for kernels and quantization accuracy.
    • Service level: load testing of throughput, latency, and error rate.
    • Quality level: automated and human evaluation of task success rate and output quality.

    7.1 Load Testing Script Example

    Before running, confirm that the target service supports streaming SSE, and check the Python version and timeout/concurrency settings, otherwise results may not be reproducible. Python 3.9 or newer is recommended; replace URL, MODEL, TIMEOUT, CONCURRENCY, and PROMPT with your actual environment values.

    1. import asyncio, json, time
    2. import urllib.request
    3. URL = "http://infer-gateway:8080/v1/chat/completions"
    4. MODEL = "default"
    5. TIMEOUT = 120
    6. CONCURRENCY = 32
    7. PROMPT = "Explain the CAP theorem in distributed systems in about 300 words."
    8. def build_payload(prompt: str) -> bytes:
    9. return json.dumps({
    10. "model": MODEL,
    11. "messages": [{"role": "user", "content": prompt}],
    12. "max_tokens": 256,
    13. "stream": True,
    14. }).encode("utf-8")
    15. def one_request(prompt: str) -> dict:
    16. req = urllib.request.Request(
    17. URL, data=build_payload(prompt), headers={"Content-Type": "application/json"}
    18. )
    19. start = time.perf_counter()
    20. ttft, tokens = None, 0
    21. with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
    22. for raw in resp:
    23. if not raw.startswith(b"data:"):
    24. continue
    25. chunk = raw[5:].strip()
    26. if chunk == b"[DONE]":
    27. break
    28. now = time.perf_counter()
    29. if ttft is None:
    30. ttft = now - start
    31. try:
    32. delta = json.loads(chunk)["choices"][0]["delta"]
    33. except (ValueError, KeyError, IndexError):
    34. continue
    35. tokens += bool(delta.get("content"))
    36. elapsed = time.perf_counter() - start
    37. tpot = (elapsed - ttft) / max(tokens - 1, 1)
    38. return {"ttft": ttft, "tpot": tpot, "tokens": tokens}
    39. async def worker(prompt: str, results: list):
    40. results.append(await asyncio.to_thread(one_request, prompt))
    41. async def bench(prompt: str, concurrency: int):
    42. results = []
    43. start = time.perf_counter()
    44. await asyncio.gather(*(worker(prompt, results) for _ in range(concurrency)))
    45. wall = time.perf_counter() - start
    46. ttfts = sorted(r["ttft"] for r in results)
    47. tpots = sorted(r["tpot"] for r in results)
    48. p95 = lambda xs: xs[min(len(xs) - 1, int(len(xs) * 0.95))]
    49. print(
    50. f"concurrency={concurrency:>3} | "
    51. f"throughput={sum(r['tokens'] for r in results)/wall:7.1f} tok/s | "
    52. f"TTFT P95={p95(ttfts)*1000:7.0f} ms | "
    53. f"TPOT P95={p95(tpots)*1000:5.1f} ms"
    54. )
    55. if __name__ == "__main__":
    56. asyncio.run(bench(PROMPT, CONCURRENCY))

    7.2 Signals to Read From Load Test Data

    • Throughput rises with concurrency while latency degrades at the same time: the system is approaching a compute or bandwidth ceiling.
    • Throughput stops growing but GPU utilization is not high: the bottleneck is likely scheduling, networking, queues, or KV cache management.
    • Noticeably long TTFT tail: usually long-input requests blocking others; consider chunked prefill or queue isolation.
    • Output rate drops markedly as context grows: decode-stage KV cache read bandwidth pressure is rising; consider KV quantization or multi-GPU parallelism.

    7.3 Load Test Record Template

    Model / version Input length Output length Concurrency TTFT P95 TPOT P95 Error rate Cost basis Conclusion
    TBD TBD TBD TBD TBD TBD TBD per 1K tokens TBD

    The template avoids the problem of being unable to compare results after optimization. Keep the model version, quantization method, batching configuration, hardware specification, and gateway configuration for every run, otherwise data from different rounds is not directly comparable.

    8. Observability and Capacity Planning

    8.1 Core Metrics to Watch

    After launch, continuously monitor:

    • Queue length and average wait time: reflect scheduling pressure.
    • KV cache utilization and preemption count: reflect memory bottlenecks.
    • Batch size distribution: reflect batching efficiency.
    • TTFT and output rate at P95/P99: reflect real user experience.
    • Error rate, timeout rate, and rate-limiting count: reflect system stability.
    • Resource consumption per 1K tokens: reflect cost efficiency.

    8.2 Alert Rule Example

    1. groups:
    2. - name: llm-inference
    3. rules:
    4. - alert: HighQueueWait
    5. expr: inference_queue_wait_seconds > 3
    6. for: 2m
    7. labels: {severity: warning}
    8. annotations: {summary: "Requests are queueing too long; consider scaling out or rate limiting"}
    9. - alert: HighKVCacheUsage
    10. expr: inference_kv_cache_used_bytes / inference_kv_cache_capacity_bytes > 0.9
    11. for: 5m
    12. labels: {severity: warning}
    13. annotations: {summary: "KV cache near capacity; preemption risk is rising"}
    14. - alert: HighTTFTP95
    15. expr: inference_ttft_seconds{quantile="0.95"} > 2
    16. for: 5m
    17. labels: {severity: critical}
    18. annotations: {summary: "TTFT over threshold; check for long-request blocking and batching config"}
    19. - alert: HighErrorRate
    20. expr: inference_requests_failed_total / inference_requests_total > 0.01
    21. for: 2m
    22. labels: {severity: critical}
    23. annotations: {summary: "Abnormal error rate; check backend health"}

    8.3 Autoscaling, Capacity Headroom, and Change Risk

    Capacity planning should be based on the “concurrency-latency” curve obtained from load testing, with headroom reserved for traffic spikes. If deployed on a mainstream cloud platform or an enterprise’s own platform, you can reuse existing container orchestration, logging, monitoring, alerting, and object storage, and focus your effort on inference scheduling and performance tuning.

    Autoscaling needs care:

    • Model loading and warm-up take a long time, so cold starts can cause jitter. Set a minimum replica count and warm up in advance.
    • Scaling signals should not rely on CPU utilization alone; queue length, cache utilization, and P95 latency are often closer to the actual bottleneck.
    • Offline batch jobs should be scheduled in off-peak windows, and when sharing resources with online serving, they need isolation and priority control.

    Change risk deserves special emphasis: before swapping out caches, scaling in, or restarting, preserve the current state, roll out gradually, and keep a rollback path, so requests are not interrupted and memory-resident data is not lost. Changes involving model versions, quantization configuration, batching strategy, or gateway routing should first be validated under shadow traffic or a small traffic slice. Logs and monitoring should mask sensitive fields so that user input, conversation content, or sensitive data is not written to disk in full.

    9. Phased Rollout Roadmap and Checklist

    Roll out inference serving in stages, following “measure first, then optimize, then verify” at every stage.

    Stage Goal Key actions Acceptance criteria Rollback trigger Owner
    Validate Prove the loop end to end Fixed evaluation set and load test baseline Features pass, baseline latency acceptable Core function fails or output quality degrades clearly Algorithm / backend
    Serve Usable externally Gateway integration, rate limiting, circuit breaking, logging, monitoring P95 latency and error rate meet targets Error rate spikes or queue keeps growing Backend / ops
    Optimize Lower cost, higher efficiency Continuous batching, quantization, prefix caching, chunked prefill Throughput up, cost down, no quality regression Quality regression or worse tail latency Performance / algorithm
    Scale Stable and controllable Queue isolation, autoscaling, multiple replicas, canary release Availability target met, scaling is smooth Scaling jitter, or scale-in interrupts requests Ops / SRE

    Go-live checklist:

    • Are SLOs and acceptance criteria defined?
    • Is there a reproducible load testing script?
    • Is the KV cache capacity estimated?
    • Are queue isolation, rate limiting, and circuit breaking configured?
    • Are canary release, rollback, and state-preservation mechanisms in place?
    • Are TTFT, TPOT, cache utilization, and error rate monitored?
    • Are logs and user inputs masked?

    Conclusion

    The complexity of LLM inference serving goes well beyond “load the model, receive a request, return a result.” From the computational characteristics of prefill and decode to paged KV cache management, from the dynamic scheduling of continuous batching to the precision-versus-speed trade-offs of quantization and speculative decoding, every optimization has to be grounded in metrics, load testing, and regression verification.

    What ultimately determines service quality is not whether any single technique is advanced, but whether the team has built clear metrics, reproducible evaluation, a rollback-capable change process, and a continuously iterating engineering loop. Break performance goals down, verify them, and extrapolate along that path, and inference serving can move from “it runs” to “it can be served, scaled, and operated over the long term.”