LLM Inference Serving: Optimizing Throughput, Latency, Quality, and Cost
As large language model capabilities grow, engineering teams must move from "it runs" to "it runs reliably, efficiently, and under control." This article lays out a practical engineering approach for serving general-purpose LLM inference. It covers how to turn performance expectations into measurable SLOs, how a request moves through the prefill and decode phases and where each phase bottlenecks, how to size and manage the KV cache (static allocation, paged attention, tiered swapping, prefix caching), how scheduling and continuous batching shape throughput versus tail latency, how quantization, kernel fusion, speculative decoding, and structured output constraints trade accuracy for speed, and how to benchmark, monitor, and alert on a live service. It also addresses long-context challenges and a phased rollout plan with acceptance criteria, rollback conditions, and a go-live checklist.
Introduction: From “It Runs” to “It Can Be Served”
As model capabilities keep improving, the real question for engineering teams is not just “what more can the model do,” but can it be run stably, efficiently, and under control?
Longer context windows, larger parameter counts, more tool calls, and multi-turn conversations all convert capability gains into pressure on GPU memory, bandwidth, scheduling, and operations. Without a serving layer, those capabilities rarely turn into a consistent user experience. The goal of inference serving is to turn “it can infer” into a measurable balance of throughput, latency, quality, and cost, and to give the system observability, rollback, and scale-out capabilities.
This article focuses on general-purpose LLM inference serving and lays out a practical engineering method: how to define targets, how to analyze the request lifecycle, how to manage the KV cache, how to schedule and batch, how to choose acceleration techniques, how to load test and monitor, and how to roll it out in stages.
1. Define Goals First: SLOs, Load Testing, and Choosing Optimizations
The first step in building a serving platform is not picking a framework, but turning “fast,” “stable,” and “cheap” into measurable numbers. Common questions in practice are: How should SLOs be defined? How should load testing be designed? How should quantization and speculative decoding be chosen? How should concurrency be estimated for long contexts?
LLM inference serving typically has to satisfy four goals at once:
- Throughput: tokens or requests processed per unit of time; reflects resource utilization.
- Latency: the speed users perceive, especially time to first token (TTFT) and output rate.
- Quality: accuracy, stability, and consistency of outputs; determines business results.
- Cost: compute resources consumed per unit of business volume; determines whether the service can scale.
These four goals constrain one another, forming the “impossible triangle” of inference serving. For example: a larger batch usually improves throughput but can lengthen queueing delays; more aggressive quantization can reduce memory usage and increase speed but may cost accuracy; longer contexts improve capability but significantly increase KV cache and scheduling pressure. Serving work should therefore start by defining SLOs from the business scenario, and only then select optimization techniques.
| Workload | Typical characteristics | Core metrics | Optimization focus |
|---|---|---|---|
| Online chat / assistants | Short inputs, longer outputs | TTFT, output rate | Scheduling latency, continuous batching |
| Long-document Q&A | Long inputs, short outputs | Prefill latency, memory usage | Context trimming, chunked prefill |
| Batch content generation | Offline jobs, no real-time requirement | Overall throughput, cost per unit | Batch size, utilization |
| High-concurrency API service | High volume, mixed lengths | P95/P99 latency, availability | Queue management, rate limiting, autoscaling |
For each scenario, define one primary goal and two constraints. For an online support assistant, for instance, make P95 TTFT the primary goal, while constraining output rate to stay above a threshold and error rate to stay below another. Once the metrics are fixed, capacity planning, load testing, and optimization all have a basis for judgment.
2. Anatomy of an Inference Request: Two Phases, Two Bottlenecks
Autoregressive LLM generation is usually divided into two phases:
- Prefill: all input tokens are processed in parallel, hidden states are computed for every position, and the KV cache is populated. This phase is dominated by matrix-matrix operations, is compute-bound, and directly determines TTFT.
- Decode: output tokens are generated one at a time. Each new token requires computing a single new query vector but reading the entire historical KV cache and the model weights. This phase is dominated by matrix-vector operations, is memory-bandwidth-bound, and determines output rate and total per-request time.
A simplified view of how the two phases differ computationally:
Arithmetic intensity ≈ FLOPs per parameter / bytes read per parameterPrefill (sequence length S): ≈ 2S (depends on input length)Decode (batch size B): ≈ B (depends on number of concurrent sequences)
Note that these arithmetic intensity figures are order-of-magnitude illustrations only. They ignore byte units, activations, temporary buffers, and framework overhead, and should not be used as exact formulas.
A few engineering rules follow from this:
- Increasing batch size mainly improves Decode efficiency and yields limited benefit for Prefill, where compute is already the bottleneck.
- When long-input and long-output requests share the same pool, they tend to block each other, and TTFT can be dragged up by long requests.
- Input length, output length, and concurrency have to be planned together rather than tuning a single variable.
3. The KV Cache: The First Constraint in the Long-Context Era
3.1 Estimating Memory Capacity
Inference memory consists of three main parts: model weights, the KV cache, and runtime overhead. The KV cache grows linearly with concurrency and sequence length, which makes it the key variable limiting service scale.
It can be approximated as:
KV bytes ≈ 2 × layers × KV heads × head_dim × sequence length × concurrency × bytes per element
For 32 layers, 8 KV heads, head dimension 128, stored in FP16:
2 × 32 × 8 × 128 × 2 ≈ 131072 bytes/token ≈ 0.125 MiB/token
That means roughly 128 MB of KV cache per 1K tokens per sequence. Estimates at different context lengths:
| Context length (single sequence) | KV cache per sequence | KV cache for 100 concurrent |
|---|---|---|
| 8K | ~1 GB | ~100 GB |
| 32K | ~4 GB | ~400 GB |
| 128K | ~16 GB | ~1.6 TB |
One point deserves emphasis: at a 128K context, even 100 concurrent sequences can push the KV cache into the terabyte range. Different layer counts, KV head counts, or precisions will change the numbers, but the order-of-magnitude reasoning stays the same. If the model uses an MoE structure, weights can be distributed across experts, but the KV cache usually still has to cover every decode layer and does not shrink automatically with expert sharding.
3.2 Three KV Cache Management Strategies
- Static reservation: pre-allocate contiguous memory for the maximum length. Simple to implement, but wasteful when lengths vary widely, and fragmentation caps concurrency.
- Paged management (PagedAttention): borrows the operating system paging idea by splitting the KV cache into fixed-size blocks allocated on demand. This substantially reduces internal fragmentation and raises both memory utilization and achievable concurrency.
- Tiered caching (swap/recompute): when memory runs short, less active cache entries are swapped out to host memory or remote storage, then swapped back or recomputed when needed. It buys concurrency at the cost of transfer or recomputation.
Prefix caching also matters. Shared prefixes such as system prompts, fixed documents, and multi-turn history can be reused across requests. On a prefix cache hit, the system can skip repeated prefill and lower TTFT.
3.3 A KV Cache Estimation Script
For capacity planning, start by estimating KV cache demand across context lengths and concurrency levels:
def kv_cache_gib(num_layers, num_kv_heads, head_dim,seq_len, concurrency, bytes_per_elem=2):"""Estimate KV cache memory usage in GiB; bytes_per_elem=2 means half precision."""per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_elemreturn per_token * seq_len * concurrency / (1024 ** 3)# Example: 32 layers, 8 KV heads, head_dim 128, half precision, 100 concurrentfor length in (8192, 32768, 131072):need = kv_cache_gib(32, 8, 128, length, concurrency=100)print(f"Context {length // 1024}K: 100 concurrent needs about {need:.1f} GiB of KV cache")
Comparing this against “total memory − weight footprint − runtime overhead” quickly tells you the concurrency ceiling your hardware can support, instead of discovering the shortfall after launch.
4. Scheduling and Batching: The Key to Throughput and Tail Latency
4.1 The Limits of Static Batching
The simplest batching approach is to “assemble a batch, compute it together, and only start the next batch after all sequences finish.” The problem is that once the short requests in a batch are done, they must wait for the longest sequence, while the GPU does wasted work for those few long sequences. The more dispersed the request length distribution, the greater the waste.
4.2 Continuous Batching
Continuous batching refines the scheduling granularity from “the whole request” to “each iteration.” The core idea: as soon as a sequence finishes, its resources are released immediately and a new request is admitted, keeping the GPU working as continuously as possible.
while not all_requests_done:# 1. Release cache blocks held by finished sequencesbatch.remove_finished()# 2. When under memory pressure, preempt some low-priority sequenceswhile memory_pressure(batch):victim = scheduler.select_preemptible(batch)victim.swap_out_or_recompute()# 3. Admit new requests from the waiting queuefor req in waiting_queue.pop_by_priority(batch.free_slots()):batch.add(req)# 4. Run one forward pass, producing one token for each unfinished sequencetokens = model.forward_step(batch)scheduler.commit(batch, tokens)
4.3 Preemption and Recovery
When active sequences approach the cache limit, the scheduler can preempt some of them. There are two common recovery strategies:
- Swap out / swap in: move the KV cache to host memory and bring it back on resume. Suited to sequences with long outputs that are expected to keep running for a while.
- Drop and recompute: free the KV cache and re-run prefill on resume. Simpler to implement, but incurs duplicate computation.
This is fundamentally a trade-off between “transfer cost” and “compute cost.” The scheduler should choose dynamically based on sequence length, expected remaining output, memory pressure, and hardware bandwidth.
4.4 Multiple Queues and Priorities
A single queue can rarely serve every request type well. A more robust approach is to split queues by business characteristics:
- Interactive request queue: high priority, short timeout, strict TTFT.
- Long-task queue: low priority, queuing allowed, optimized for throughput and cost.
- Batch queue: fully offline, scheduled flexibly during off-peak hours.
Combined with timeouts, rate limiting, circuit breaking, and input length validation, this prevents a single oversized input or abnormal output from taking down the whole service.
5. Compute Acceleration: Quantization, Kernel Fusion, and Speculative Decoding
5.1 Quantization
Decode is memory-bandwidth-bound, and reading weights usually dominates single-step time. Lowering weight precision reduces both memory footprint and bytes read.
- Weight quantization: compress weights into lower-precision formats such as INT8 or INT4.
- Activation and KV cache quantization: further reduce runtime and cache overhead, though outlier effects need attention.
- Group-wise / channel-wise quantization: determine quantization parameters over smaller groups to reduce the accuracy impact of outliers.
The gains from quantization must be grounded in evaluation. On a fixed regression set, compare task success rate, output consistency, and human review before and after quantization, rather than looking only at memory savings.
5.2 Kernel Fusion and Memory-Efficient Attention
A standard attention implementation explicitly materializes an attention matrix shaped “batch × heads × sequence length × sequence length,” whose memory footprint grows quadratically with context length. Memory-efficient fused attention kernels use tiled computation and online normalization to avoid materializing that large matrix, bringing attention memory overhead closer to linear growth while reducing memory accesses. For long-context scenarios, this kind of optimization is often a prerequisite for running the service stably at all.
5.3 Speculative Decoding
The core of speculative decoding is “guess first, verify after”:
- A lightweight draft model generates several candidate tokens.
- The main model performs one parallel forward pass over the candidates.
- An acceptance criterion guarantees the final output distribution matches decoding with the main model alone; rejected positions fall back to resampling.
Because one main-model forward pass can confirm multiple tokens, output speed increases, and in theory output quality is unchanged. The benefit depends on the draft hit rate: code, structured text, and templated content typically accelerate better.
5.4 Structured Output Constraints
When the business requires valid JSON or another fixed format, grammar constraints can be introduced during decoding to mask tokens that violate the syntax. This reduces format errors, downstream parsing failures, and retry requests, improving service stability.
5.5 Choosing an Acceleration Technique
| Technique | Suitable scenario | Main benefit | Cost and risk | How to validate |
|---|---|---|---|---|
| Continuous batching | High volume with varying lengths | Higher GPU utilization and throughput | Complex scheduling; long requests may interfere with short ones | Compare TTFT P95, throughput, preemption count |
| Quantization | Tight memory, bandwidth-limited | Less memory, higher speed, lower cost | Possible accuracy loss | Regression set, task success rate, human review |
| Speculative decoding | Predictable output, latency-sensitive | Higher output rate | Limited gains when draft hit rate is unstable | Compare TPOT, acceptance rate, quality regression |
| Prefix caching | Repeated system prompts, multi-turn history | Lower TTFT | Complex cache hit management | Hit rate, prefill time, memory usage |
6. Long Context: New Capability, New Challenges
Extending the context window improves long-document, multi-turn, and complex-instruction handling, but it also amplifies memory, bandwidth, and scheduling pressure in one place:
- Prefill cost grows with length: hundreds of thousands of input tokens can take seconds, easily blocking subsequent requests.
- Attention cost remains significant: even with memory-efficient implementations, long-context compute volume is still substantial.
- KV cache footprint is enormous: very long contexts sharply reduce achievable concurrency.
- Middle-of-context information loss: when relevant information sits in the middle of a long context, model utilization may drop.
Engineering options include:
- Retrieval augmentation: split, retrieve, and rerank very long inputs, and send only the most relevant fragments.
- Chunked prefill: split long inputs into chunks computed in stages, smoothing the memory peak and interleaving with decode requests.
- Context compression and summarization: periodically compress earlier history in multi-turn conversations to free cache.
- Prefix cache reuse: share system prompts, fixed documents, and history prefixes to avoid repeated prefill.
7. Evaluation: Making Optimization Measurable and Repeatable
The biggest risk in optimization is “it feels faster.” A reproducible evaluation system should include at least three layers:
- Component level: numerical consistency checks for kernels and quantization accuracy.
- Service level: load testing of throughput, latency, and error rate.
- Quality level: automated and human evaluation of task success rate and output quality.
7.1 Load Testing Script Example
Before running, confirm that the target service supports streaming SSE, and check the Python version and timeout/concurrency settings, otherwise results may not be reproducible. Python 3.9 or newer is recommended; replace URL, MODEL, TIMEOUT, CONCURRENCY, and PROMPT with your actual environment values.
import asyncio, json, timeimport urllib.requestURL = "http://infer-gateway:8080/v1/chat/completions"MODEL = "default"TIMEOUT = 120CONCURRENCY = 32PROMPT = "Explain the CAP theorem in distributed systems in about 300 words."def build_payload(prompt: str) -> bytes:return json.dumps({"model": MODEL,"messages": [{"role": "user", "content": prompt}],"max_tokens": 256,"stream": True,}).encode("utf-8")def one_request(prompt: str) -> dict:req = urllib.request.Request(URL, data=build_payload(prompt), headers={"Content-Type": "application/json"})start = time.perf_counter()ttft, tokens = None, 0with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:for raw in resp:if not raw.startswith(b"data:"):continuechunk = raw[5:].strip()if chunk == b"[DONE]":breaknow = time.perf_counter()if ttft is None:ttft = now - starttry:delta = json.loads(chunk)["choices"][0]["delta"]except (ValueError, KeyError, IndexError):continuetokens += bool(delta.get("content"))elapsed = time.perf_counter() - starttpot = (elapsed - ttft) / max(tokens - 1, 1)return {"ttft": ttft, "tpot": tpot, "tokens": tokens}async def worker(prompt: str, results: list):results.append(await asyncio.to_thread(one_request, prompt))async def bench(prompt: str, concurrency: int):results = []start = time.perf_counter()await asyncio.gather(*(worker(prompt, results) for _ in range(concurrency)))wall = time.perf_counter() - startttfts = sorted(r["ttft"] for r in results)tpots = sorted(r["tpot"] for r in results)p95 = lambda xs: xs[min(len(xs) - 1, int(len(xs) * 0.95))]print(f"concurrency={concurrency:>3} | "f"throughput={sum(r['tokens'] for r in results)/wall:7.1f} tok/s | "f"TTFT P95={p95(ttfts)*1000:7.0f} ms | "f"TPOT P95={p95(tpots)*1000:5.1f} ms")if __name__ == "__main__":asyncio.run(bench(PROMPT, CONCURRENCY))
7.2 Signals to Read From Load Test Data
- Throughput rises with concurrency while latency degrades at the same time: the system is approaching a compute or bandwidth ceiling.
- Throughput stops growing but GPU utilization is not high: the bottleneck is likely scheduling, networking, queues, or KV cache management.
- Noticeably long TTFT tail: usually long-input requests blocking others; consider chunked prefill or queue isolation.
- Output rate drops markedly as context grows: decode-stage KV cache read bandwidth pressure is rising; consider KV quantization or multi-GPU parallelism.
7.3 Load Test Record Template
| Model / version | Input length | Output length | Concurrency | TTFT P95 | TPOT P95 | Error rate | Cost basis | Conclusion |
|---|---|---|---|---|---|---|---|---|
| TBD | TBD | TBD | TBD | TBD | TBD | TBD | per 1K tokens | TBD |
The template avoids the problem of being unable to compare results after optimization. Keep the model version, quantization method, batching configuration, hardware specification, and gateway configuration for every run, otherwise data from different rounds is not directly comparable.
8. Observability and Capacity Planning
8.1 Core Metrics to Watch
After launch, continuously monitor:
- Queue length and average wait time: reflect scheduling pressure.
- KV cache utilization and preemption count: reflect memory bottlenecks.
- Batch size distribution: reflect batching efficiency.
- TTFT and output rate at P95/P99: reflect real user experience.
- Error rate, timeout rate, and rate-limiting count: reflect system stability.
- Resource consumption per 1K tokens: reflect cost efficiency.
8.2 Alert Rule Example
groups:- name: llm-inferencerules:- alert: HighQueueWaitexpr: inference_queue_wait_seconds > 3for: 2mlabels: {severity: warning}annotations: {summary: "Requests are queueing too long; consider scaling out or rate limiting"}- alert: HighKVCacheUsageexpr: inference_kv_cache_used_bytes / inference_kv_cache_capacity_bytes > 0.9for: 5mlabels: {severity: warning}annotations: {summary: "KV cache near capacity; preemption risk is rising"}- alert: HighTTFTP95expr: inference_ttft_seconds{quantile="0.95"} > 2for: 5mlabels: {severity: critical}annotations: {summary: "TTFT over threshold; check for long-request blocking and batching config"}- alert: HighErrorRateexpr: inference_requests_failed_total / inference_requests_total > 0.01for: 2mlabels: {severity: critical}annotations: {summary: "Abnormal error rate; check backend health"}
8.3 Autoscaling, Capacity Headroom, and Change Risk
Capacity planning should be based on the “concurrency-latency” curve obtained from load testing, with headroom reserved for traffic spikes. If deployed on a mainstream cloud platform or an enterprise’s own platform, you can reuse existing container orchestration, logging, monitoring, alerting, and object storage, and focus your effort on inference scheduling and performance tuning.
Autoscaling needs care:
- Model loading and warm-up take a long time, so cold starts can cause jitter. Set a minimum replica count and warm up in advance.
- Scaling signals should not rely on CPU utilization alone; queue length, cache utilization, and P95 latency are often closer to the actual bottleneck.
- Offline batch jobs should be scheduled in off-peak windows, and when sharing resources with online serving, they need isolation and priority control.
Change risk deserves special emphasis: before swapping out caches, scaling in, or restarting, preserve the current state, roll out gradually, and keep a rollback path, so requests are not interrupted and memory-resident data is not lost. Changes involving model versions, quantization configuration, batching strategy, or gateway routing should first be validated under shadow traffic or a small traffic slice. Logs and monitoring should mask sensitive fields so that user input, conversation content, or sensitive data is not written to disk in full.
9. Phased Rollout Roadmap and Checklist
Roll out inference serving in stages, following “measure first, then optimize, then verify” at every stage.
| Stage | Goal | Key actions | Acceptance criteria | Rollback trigger | Owner |
|---|---|---|---|---|---|
| Validate | Prove the loop end to end | Fixed evaluation set and load test baseline | Features pass, baseline latency acceptable | Core function fails or output quality degrades clearly | Algorithm / backend |
| Serve | Usable externally | Gateway integration, rate limiting, circuit breaking, logging, monitoring | P95 latency and error rate meet targets | Error rate spikes or queue keeps growing | Backend / ops |
| Optimize | Lower cost, higher efficiency | Continuous batching, quantization, prefix caching, chunked prefill | Throughput up, cost down, no quality regression | Quality regression or worse tail latency | Performance / algorithm |
| Scale | Stable and controllable | Queue isolation, autoscaling, multiple replicas, canary release | Availability target met, scaling is smooth | Scaling jitter, or scale-in interrupts requests | Ops / SRE |
Go-live checklist:
- Are SLOs and acceptance criteria defined?
- Is there a reproducible load testing script?
- Is the KV cache capacity estimated?
- Are queue isolation, rate limiting, and circuit breaking configured?
- Are canary release, rollback, and state-preservation mechanisms in place?
- Are TTFT, TPOT, cache utilization, and error rate monitored?
- Are logs and user inputs masked?
Conclusion
The complexity of LLM inference serving goes well beyond “load the model, receive a request, return a result.” From the computational characteristics of prefill and decode to paged KV cache management, from the dynamic scheduling of continuous batching to the precision-versus-speed trade-offs of quantization and speculative decoding, every optimization has to be grounded in metrics, load testing, and regression verification.
What ultimately determines service quality is not whether any single technique is advanced, but whether the team has built clear metrics, reproducible evaluation, a rollback-capable change process, and a continuously iterating engineering loop. Break performance goals down, verify them, and extrapolate along that path, and inference serving can move from “it runs” to “it can be served, scaled, and operated over the long term.”
