LLM Inference Performance Evaluation: Balancing Latency, Throughput, and Cost

    A practical guide to evaluating large language model (LLM) inference performance and balancing latency, quality, and cost. It breaks down the two core metrics—time to first token (TTFT) and generation throughput (TPS)—and shows how to measure them under realistic loads. It explains how model architecture and inference infrastructure (continuous batching, KV cache management, quantization, speculative decoding, and parallelism) shape performance, examines how reasoning tokens affect both user experience and billing, and lays out a tiered strategy for choosing between fast models and capability-oriented models. It closes with engineering practices for controlling inference cost.

    Introduction: When “Fast” and “Powerful” Become Two Mandatory Questions

    As large-scale LLM applications move into production, user expectations for interactive AI keep rising: answers must be accurate, and tokens must appear fast. Scenarios such as code completion, real-time Q&A, and intelligent customer service place nearly exacting demands on the responsiveness of inference services. At the same time, scenarios built around complex reasoning and long-chain tasks demand stronger thinking capabilities. These two needs largely conflict with one another, forming the core tension in engineering practice—a three-way trade-off among latency, quality, and cost.

    Recent industry discussions around “launching high-speed inference models while retiring older capability-oriented models” are essentially an externalization of this tension. This article is not limited to any specific product. Instead, from a general engineering perspective, it systematically reviews the methods, influencing factors, and optimization paths for evaluating LLM inference performance, helping developers make rational choices between fast models and capability-oriented models.

    1. How to Quantify Whether Inference Is “Good”: Two Key Metrics

    To evaluate an inference service, you first need to move beyond vague notions of “feeling fast or slow” and establish a quantifiable set of metrics. Among them, time to first token and generation throughput are the two most central dimensions.

    1. Time To First Token (TTFT)

    TTFT is the interval between sending a request and receiving the first token. It directly determines the user’s perceived responsiveness, and in streaming output scenarios, how quickly the first character appears is almost equivalent to the user’s first impression of the system’s agility.

    TTFT is mainly affected by the following factors:

    • Queue wait time: how long a request waits in the scheduling queue to be processed, which is directly related to load under high concurrency;
    • Prefill time: the time the model spends on a single forward pass over the input prompt, which is positively correlated with input length;
    • Network and transport overhead: the round-trip latency between client and server.

    Generally, a TTFT in the hundreds of milliseconds already delivers a smooth, “near real-time” experience. But when the input context is long, the prefill stage becomes the main bottleneck for TTFT.

    2. Generation Throughput (TPS)

    TPS is the number of tokens the model produces per second during the generation phase, and it determines how smoothly long text “streams out.” When TPS reaches 300 or more, human reading speed usually can’t keep up with generation, and users barely notice any “stuttering.” When TPS is low, even a fast first token is followed by generation that feels choppy.

    It’s worth emphasizing that TTFT and TPS optimize different phases: the former concerns “when it starts,” while the latter concerns “how fast it finishes.” A strong inference service needs to meet targets on both at the same time.

    3. End-to-End Latency and Measurement

    Beyond first-token and throughput, end-to-end total latency is equally important—it combines time to first token, the number of generated tokens, and throughput. You can use the following approach to establish a baseline measurement for a generic streaming inference API:

    1. import time
    2. def measure_inference(client, model, prompt):
    3. """Measure time to first token, generation throughput, and end-to-end latency (example: generic streaming API)"""
    4. start = time.perf_counter()
    5. first_token_time = None
    6. token_count = 0
    7. stream_resp = client.chat.completions.create(
    8. model=model,
    9. messages=[{"role": "user", "content": prompt}],
    10. stream=True,
    11. )
    12. for chunk in stream_resp:
    13. delta = chunk.choices[0].delta
    14. if delta.content:
    15. if first_token_time is None:
    16. first_token_time = time.perf_counter()
    17. token_count += 1
    18. end = time.perf_counter()
    19. ttft = (first_token_time - start) if first_token_time else None
    20. e2e = end - start
    21. tps = token_count / (end - first_token_time) if first_token_time else 0
    22. return {
    23. "ttft_ms": round(ttft * 1000, 1) if ttft else None,
    24. "total_tokens": token_count,
    25. "tps": round(tps, 1),
    26. "e2e_ms": round(e2e * 1000, 1),
    27. }

    It’s recommended to run multiple rounds of load testing under realistic business workloads across different input lengths and concurrency levels, and to record percentiles of TTFT and TPS (such as P50/P95/P99) rather than looking only at averages. Only this way can you accurately characterize the service’s true performance.

    2. Architecture and Inference Infrastructure: The Determinants Behind Performance

    Why do TTFT and TPS vary so widely across models? The reasons lie both at the model architecture level and at the inference infrastructure (Infra) level.

    1. Model Architecture Level

    • Parameter count and computational complexity: larger parameter scale means higher per-forward-pass computation, which usually translates to slower generation;
    • Attention mechanism efficiency: the complexity of attention computation and whether a more efficient attention implementation is used can significantly affect throughput over long sequences;
    • Built-in reasoning/thinking mechanism: some models generate an internal reasoning process before producing their final output, which adds extra computation and latency.

    2. Inference Infrastructure Level

    • Continuous batching: dynamically batching multiple concurrent requests significantly improves GPU utilization and overall throughput;
    • KV cache management: caching attention key-value pairs to avoid repeated computation is key to accelerating long-context scenarios, and introducing paged cache management can further reduce memory fragmentation;
    • Quantization: lowering the numerical precision of weights and activations (such as INT8 or FP8) reduces memory footprint and improves computational efficiency;
    • Speculative decoding: using a small model to “guess” several tokens in advance, then having the large model verify them in batch, can improve TPS with essentially no loss in quality;
    • Hardware and parallelism strategy: tensor parallelism, pipeline parallelism, and high-speed interconnect bandwidth determine the scaling ceiling for large-scale inference.

    It is precisely these engineering optimizations at the Infra level that enable a new generation of fast models to reach performance levels of hundreds of TPS and sub-second TTFT. Conversely, models with relatively older architectures fall noticeably behind on cost-performance, even if you simply throw more compute at them.

    3. The Cost of “Overthinking”: The Relationship Between Reasoning Tokens and Cost

    A defining characteristic of reasoning-oriented models is that, before giving a final answer, they unfold a relatively long internal “thinking” process (i.e., reasoning tokens). This mechanism genuinely improves accuracy on complex tasks, but it also brings two practical problems.

    1. Experience Level: Offset by High-Speed Generation

    When TPS is high enough (such as 300+), even if the thinking process generates many tokens, users barely notice the latency—because the streaming speed far exceeds reading speed. So high throughput to some extent “absorbs” the negative impact of overthinking on the interactive experience.

    2. Cost Level: The Hidden Overhead That Gets Billed

    What truly warrants caution is cost. The vast majority of inference services bill by total tokens, and thinking tokens are counted too. This means:

    • A task’s “visible output” may be only a few hundred tokens, but once the thinking process is added, the actually billed tokens can double or more;
    • For high-concurrency, high-volume calling scenarios, this hidden overhead is dramatically amplified;
    • Simply relying on a “faster model” does not reduce the total token consumption per task.

    Therefore, when evaluating inference cost, you shouldn’t look only at output length—you should factor thinking tokens into the overall accounting.

    4. Fast Models vs. Capable Models: A Tiered Selection Strategy

    Faced with changes like “launching high-speed models while retiring capability-oriented models,” the rational approach is to establish a clear tiered selection strategy rather than a one-size-fits-all rule.

    1. Positioning of the Two Model Types

    • Fast models: with advantages in high TPS, low TTFT, and low unit cost, they suit high-concurrency, latency-sensitive, relatively standardized scenarios such as code completion, real-time Q&A, content summarization, and intent classification;
    • Capable models: with advantages in stronger complex reasoning, long-chain task handling, and generation quality, they suit scenarios with extremely high correctness requirements and little room for error, such as complex analysis, multi-step reasoning, and high-difficulty creative work.

    2. Selection Decision Framework

    Dimension Prefer Fast Models Prefer Capable Models
    Latency sensitivity High (real-time interaction) Medium/Low (can tolerate longer waits)
    Task complexity Medium-low, standardized High, requires deep reasoning
    Concurrency scale High concurrency, high volume Relatively controlled
    Cost constraint Strict Relatively relaxed
    Quality tolerance Higher Very low

    3. Routing and Fallback Strategy

    In practice, you can adopt “intelligent routing” to direct requests to different models: first use a lightweight classifier to judge task difficulty, send simple requests to fast models, and escalate complex requests to capable models. You can also set up a fallback path—when a capable model is unavailable or heavily queued, automatically fall back to a fast model to safeguard service availability.

    5. Engineering Practices for Cost Optimization

    Regardless of which type of model you choose, the following strategies can effectively control system costs:

    1. Control the thinking budget: guide the model to “answer concisely” via prompts, or limit thinking length where supported, to avoid unnecessary over-reasoning;
    2. Trim the context: remove redundant conversation history and irrelevant documents to reduce input tokens, thereby lowering both prefill overhead and billing;
    3. Cache reuse: cache results for high-frequency, repeated requests so that a cache hit can skip inference entirely;
    4. Tiered rate limiting and monitoring: build a token-consumption dashboard to track TTFT, TPS, token usage, and cost per business line, catching abnormal growth early;
    5. Batching and asynchrony: use batch processing for non-real-time offline tasks to improve overall resource utilization and amortize unit cost.

    Conclusion: Returning to the Essential Trade-off

    Choosing an LLM inference service ultimately comes down to finding the balance point among latency, quality, and cost. Time to first token shapes the user’s first impression, generation throughput determines the smoothness of long outputs, inference infrastructure sets the performance ceiling, and thinking tokens reveal the hidden tension between performance and cost.

    Amid the ongoing iteration and replacement of fast models and capability-oriented models, what developers truly need to focus on is not the launch or retirement of any single version, but establishing a quantifiable evaluation system and a tiered selection mechanism: let data speak, route by scenario, and spend every bit of compute and cost where it’s genuinely needed. This is the most rational answer to “fast” and “powerful” from an engineering perspective.