LLM Inference Performance Optimization: Balancing Throughput, Latency, and Cost
A practical guide to LLM inference performance. It explains the two core online-serving metrics (TTFT and TPS) and the inference stages behind them, breaks down the model architecture, inference framework, and infrastructure factors that determine speed, examines how reasoning models inflate cost through "overthinking," and offers guidance on tiered model selection and engineering best practices for balancing throughput, latency, and cost.
In real-world production, whether a model is “good enough” often has little to do with leaderboard scores and everything to do with its response speed and cost per unit of work. Many teams focus only on generation quality during selection, overlooking the fact that, at the same quality level, differences in inference performance can drive several-fold swings in cost. This article starts from measurable metrics, breaks down what determines LLM inference performance, and offers practical optimization and selection approaches.
1. Two Core Metrics You Can’t Avoid
Industry practice evaluates online inference services around two key metrics, each mapped to a distinct user experience:
1. TTFT (Time To First Token)
The time from when a user sends a request to when they receive the first token. It drives the first impression of whether the system feels “laggy,” and matters most for interactive scenarios (AI coding, chat assistants, Agent tool calls). As a rule of thumb, TTFT needs to stay within a few hundred milliseconds for users to feel the system is “keeping up with their thinking.”
2. TPS (Tokens Per Second)
After the first token arrives, how many tokens the model can generate per second. It determines the smoothness of long-form output. Once TPS reaches 100–200 or above, streaming output feels close to human reading speed, with almost no perceptible stutter.
It’s worth emphasizing that these two metrics correspond to two distinct phases of inference:
[Input Prompt] --Prefill--> [KV Cache] --Decode (token-by-token generation)--> [Output]↑ ↑mainly affects TTFT mainly affects TPS
- Prefill processes all input tokens in parallel. It is compute-intensive and mainly affects first-token latency.
- Decode generates tokens autoregressively, one at a time. It is memory-bandwidth-bound and mainly affects sustained throughput.
Understanding this is important: TTFT and TPS usually require different optimization techniques, and no single solution can cover both.
2. The Architecture and Infrastructure Behind Performance
When some models achieve low latency and high throughput, it is rarely the result of a single trick. It is usually the outcome of model architecture and inference infrastructure evolving together.
1. Model Architecture
- Sparse architectures (such as MoE, Mixture of Experts): Only a subset of parameters is activated per inference, preserving the capability that comes from a large total parameter count while significantly reducing per-call computation.
- Attention mechanism optimization: Techniques such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) compress the memory footprint of the KV Cache, enabling larger concurrent batches under the same GPU memory.
- Layered model structure design: Trade-offs across parameter count, number of layers, and context length for different capability tiers directly set the theoretical speed ceiling.
2. Inference Frameworks
Modern high-performance inference services generally rely on the following common techniques. They are not exclusive to any single vendor but are publicly documented engineering practices across the industry:
- Continuous Batching: Dynamically merges waiting requests into the current batch, greatly improving GPU utilization.
- Paged KV Cache management: Applies operating-system paging concepts to GPU memory management, reducing fragmentation and increasing concurrent capacity.
- Speculative Decoding: A smaller model or draft mechanism first “guesses” several tokens, which the large model then verifies in a batch, accelerating the Decode phase.
- Prefix Cache: Reuses computation results for repeated system prompts or context, reducing the overhead of repeated Prefill.
- Quantization and parallelism strategies: INT8/FP8 quantization, tensor parallelism, and pipeline parallelism compress compute and memory requirements while keeping accuracy under control.
3. Underlying Infrastructure
Achieving high TPS and low TTFT also depends on the overall scheduling of hardware and services:
- Compute supply: The GPU/NPU model, memory bandwidth, and inter-card interconnect together determine the memory-access ceiling of the Decode phase.
- Service elasticity: Container platforms, auto-scaling, and load balancing keep throughput stable during traffic peaks.
- Observability: Logging services and monitoring alerts continuously track metrics such as P99 latency, timeout rate, and queue time.
In short, a model that “keeps up with the times” in architecture and infrastructure can, compared with a model on an outdated architecture, offer a generational difference in cost-effectiveness even on the same task. This is the technical root of the phenomenon where “a new-architecture lightweight model outperforms an older, larger model” within a layered product lineup.
3. The “Overthinking” and Cost Trap of Reasoning Models
In recent years, reasoning models (which emphasize “thinking before answering”) have gained traction, improving accuracy on complex tasks through additional chain-of-thought reasoning. But this introduces a new problem: overthinking.
Why Overthinking Happens
A model may generate long chains of thought even for simple questions, stretching a task that should take a few seconds into something much longer. It shows up as:
- A noticeable “thinking pause” that stretches out TTFT;
- A number of reasoning tokens far exceeding what the task actually requires.
How Cost Gets Amplified
Here is a key but often overlooked fact: billing is typically based on the total of input and output tokens, and the tokens generated by chain-of-thought reasoning count as output-side consumption. This creates a tension:
Total cost ≈ (Input tokens + Output tokens) × unit price↑ ↑may decrease reasoning tokens drive it up
Even when a single response is generated fast (high TPS), the extra tokens from the chain of thought still push up the cost per task. This means “runs fast” and “costs little” are not equivalent: a model that can rapidly output large amounts of reasoning tokens may still fail to bring down the overall cost of an actual task.
How to Respond
- Differentiate by task complexity: Use lightweight, low-latency models that don’t force deep reasoning for simple tasks (classification, extraction, formatting); enable reasoning capability only for complex tasks (planning, reasoning, code generation).
- Control the thinking budget: Where supported, cap the maximum number of reasoning tokens, or choose an “on-demand thinking” strategy.
- Caching and batching: Use prefix caching for high-frequency repeated prompts, and use batch processing for offline tasks to improve throughput and amortize cost.
4. Tiered Selection: Positioning and Trade-offs Across Tiers
A mature LLM service system is usually layered: it has a “lightweight tier” focused on speed and cost, as well as a “flagship tier” focused on capability. Their positioning differs, so they cannot be compared head-to-head on capability alone.
| Dimension | Lightweight-tier model | Flagship-tier model |
|---|---|---|
| Primary goal | Low latency, high throughput, low cost | Capability ceiling on complex tasks |
| Use cases | High-concurrency online interaction, structured output | Deep reasoning, long-chain planning |
| Architecture orientation | Sparse design, optimized attention | Large parameters, strong reasoning |
| Cost profile | Cheap per token, suited to scale | High unit price, suited to critical paths |
A healthy decision framework looks like this:
- When the lightweight tier’s capability already meets most business needs, route traffic to the lightweight tier first to control overall cost;
- Only take on the extra cost of the flagship tier when its capability clearly pulls ahead and the business genuinely needs it;
- Avoid using the “flagship tier” to run simple tasks that the lightweight tier should handle—this is a hidden form of resource waste.
5. Practical Recommendations for Engineering Teams
- Build business-metric-driven evaluation: Don’t look only at leaderboards. Measure P50/P99 of TTFT, TPS, and the overall cost per thousand successful tasks under real traffic.
- Evaluate architecture and versions with foresight: Keep an eye on the performance leaps from new architectures and new inference frameworks, and avoid staying locked to older models on outdated architectures for too long.
- Factor the “thinking budget” into the cost model: Set clear token limits and fallback strategies for reasoning capabilities.
- Preserve the ability to switch across tiers: Through a unified service abstraction layer (rather than hard-coding a single model), support flexible routing between tiers to balance experience, cost, and stability.
Conclusion
Optimizing LLM inference performance is, at its core, a dynamic balance among throughput, latency, and cost. TTFT and TPS determine “how fast it feels,” architecture and infrastructure determine “whether it can be fast,” and the expansion of reasoning tokens determines “how expensive it is.” Understanding these three threads is what lets you make selection decisions—amid a vast and rapidly evolving supply of models—that both preserve user experience and control cost. That is far more pragmatic than chasing a single “most powerful model.”
