LLM Inference Serving: Throughput, Time to First Token, and Cost Optimization in Practice

    This article turns LLM serving performance into three measurable numbers — time to first token (TTFT), generation throughput (TPS), and end-to-end latency — and explains which inference-engine techniques move each of them: KV cache and memory paging, continuous batching, quantization and kernel optimization, prefix caching, and parallelism. It also covers a cost driver teams routinely underestimate: reasoning models that "overthink" and inflate output token spend. Finally, it describes how to trade off between a fast tier and a capable tier, route requests between them, and apply practical cost controls such as input trimming, output limits, response caching, percentile-based capacity planning, and per-scenario cost attribution.

    1. Why Inference Performance and Cost Have to Be Evaluated Together

    The experience of an online LLM service is fundamentally a balance between two curves: response speed and cost per call. If responses are too slow, interactive scenarios such as chat, AI coding assistants, and real-time Q&A feel visibly laggy. But if you keep throwing compute at the problem to make it faster, the cost per request spirals out of control.

    Many teams fall into the same trap: they pick a model purely on capability benchmarks, then discover after launch that the high-capability tier comes with a longer inference path and heavier compute usage, so the concurrency the same budget can support shrinks dramatically.

    The professional approach is different: first decompose performance into quantifiable, measurable metrics and understand the system bottleneck behind each one. Then use that understanding to make engineering decisions about which capability tier to deploy. Finally, apply systematic techniques to bring cost down.

    2. The Core Metrics: Throughput, Time to First Token, and Generation Speed

    Before optimizing, you have to separate three metrics that are constantly confused:

    • Time to first token (TTFT): the time from sending the request to receiving the first output token. It is dominated by the prefill phase — the cost of computing the entire input prompt in one parallel pass. Longer inputs generally mean higher TTFT.
    • Generation speed / throughput (TPS, tokens per second): how many tokens per second are generated once output begins. This is determined by the token-by-token autoregressive work of the decode phase.
    • End-to-end duration: TTFT plus the full generation time. This is what actually shapes a user’s perception of “how long is this answer going to take.”

    The usual rule of thumb: interactive scenarios want TTFT in the hundreds of milliseconds and TPS in the tens to low hundreds of tokens per second to get a smooth typewriter effect. These two metrics are governed by different phase bottlenecks and require different optimizations — you cannot lump them together as “improve performance.”

    Here is a streaming call you can use to measure them:

    1. import time
    2. def measure_streaming(chat_stream_fn, prompt: str) -> dict:
    3. start = time.perf_counter()
    4. first_token_time = None
    5. n_tokens = 0
    6. for token in chat_stream_fn(prompt): # tokens returned incrementally
    7. if first_token_time is None:
    8. first_token_time = time.perf_counter()
    9. n_tokens += 1
    10. end = time.perf_counter()
    11. ttft_ms = (first_token_time - start) * 1000 if first_token_time else None
    12. total_s = end - start
    13. tps = n_tokens / total_s if total_s > 0 else 0
    14. return {"ttft_ms": ttft_ms, "total_s": total_s, "tps": round(tps, 1)}

    In production, this measurement should be wired into a unified monitoring and alerting system, reporting percentiles (such as P50/P95) for both TTFT and TPS rather than averages alone — tail requests are usually what wreck the user experience.

    3. What Really Sets the Performance Ceiling: Inference Architecture and Infrastructure

    At the same model size, engineering choices can produce several-fold differences in performance. The common techniques for high throughput and low latency are:

    1. KV cache and GPU memory management. Autoregressive generation reuses the keys and values of previous tokens. A larger cache saves compute, but consumes more GPU memory. Paged memory management can substantially reduce fragmentation and increase the number of concurrent requests.
    2. Continuous batching. The traditional “accumulate a batch, then compute” approach lets short requests get dragged down by long ones. Continuous batching allows requests to join or leave a batch dynamically mid-decode, greatly improving GPU utilization and overall throughput.
    3. Quantization and kernel optimization. Moving weights and activations from high precision down to INT8/INT4, combined with memory-access optimizations for attention kernels, can improve both TTFT and TPS while keeping accuracy within acceptable bounds.
    4. Prefix caching. In multi-turn conversations or scenarios with a fixed system prompt, reusing the computed result for an identical prefix significantly reduces the TTFT caused by repeated prefill.
    5. Parallelism strategies. Tensor parallelism and pipeline parallelism split the computation of a single layer across multiple GPUs, and are the infrastructure prerequisite for keeping latency acceptable on very large models.

    These are general-purpose inference engineering practices across the industry. Mainstream cloud platforms — with their container platforms, object storage, logging, and monitoring and alerting capabilities — can be combined to support the deployment and observability of a service like this.

    4. The Overlooked Cost Driver: “Overthinking” in the Inference Chain

    Reasoning models generate an internal chain of thought before producing a conclusion. A moderate amount of thinking improves accuracy on complex problems, but overthinking needlessly lengthens the output, with two direct consequences:

    • The generated thinking tokens are billed the same way and consume the same compute, so the total token cost for the actual task goes up rather than down.
    • Longer output directly lowers the effective information density in TPS terms, and users wait longer.

    The good news is that when generation is fast enough, the extra thinking tokens have little impact on perceived responsiveness — the real pain point is the bill. So engineering-wise, you should allow constraints on the reasoning budget (for example, capping maximum output length, or disabling or shortening reasoning for simple tasks). Whether to “think deeply” should be a tunable parameter, not a one-size-fits-all setting.

    5. Fast Tier vs. Capable Tier: How to Choose

    LLM product lines are generally tiered: one tier optimizes for maximum speed, low latency, and low unit price, while another targets the capability ceiling on complex tasks. The key question is not “which one is stronger” but “which constraint does your scenario actually hit”:

    • For high-concurrency, strongly interactive scenarios with relatively standardized tasks (customer support, summarization, code completion), prefer the fast tier and trade speed and cost for scale.
    • Only low-concurrency, high-difficulty tasks that are extremely sensitive to accuracy — offline or near-offline workloads — justify paying for the capable tier.
    • Legacy capable-tier models that have fallen behind on architecture have worse price-performance: they have no speed advantage and a high cost per unit of capability. Gradually retiring them and shifting traffic to faster tiers is a sound product decision.

    The pragmatic architecture is routing plus tiering: use rules or a lightweight classifier to judge request difficulty, send simple requests to the fast tier, and escalate complex ones to the capable tier.

    1. def route(prompt: str, difficulty_score: float) -> str:
    2. # difficulty_score is produced by lightweight rules/classifiers, range 0~1
    3. if difficulty_score < 0.5:
    4. return "fast_tier" # fast tier: low latency, low cost
    5. return "pro_tier" # capable tier: complex reasoning tasks

    6. Cost Controls You Can Actually Ship

    Beyond architecture and model selection, there is a whole set of cost-reduction levers on the engineering side:

    • Slim down the input: trim system prompts and remove redundant context to cut prompt tokens directly.
    • Reuse prefix caching: hand the fixed portion of the prompt to the prefix cache to avoid repeated computation.
    • Constrain the output: set a sensible maximum output length for tasks that do not need long-form reasoning, which suppresses overthinking.
    • Cache frequent requests: cache results for high-frequency questions with stable answers.
    • Percentile monitoring and capacity planning: scale based on P95 rather than the average, so you do not overpay for the tail.
    • Cost observability: tag token usage and spend by business line and scenario — know where the money goes before optimizing it.

    A simple usage and cost estimate helps you compare options during selection:

    1. def estimate_cost(input_tokens, output_tokens,
    2. price_in_per_1k, price_out_per_1k):
    3. return (input_tokens / 1000 * price_in_per_1k
    4. + output_tokens / 1000 * price_out_per_1k)

    7. Conclusion

    Optimizing LLM inference is never a single trick — it is a full chain. Quantify TTFT and TPS first, then find the bottleneck in the architecture and infrastructure, then control the hidden token cost of the reasoning chain, and finally manage experience and spend together with tiering, routing, and monitoring and alerting. Walk through those four steps and you can reliably match each scenario to a model tier that is good enough, fast enough, and cheap enough within a fixed compute budget.