LLM Inference Performance Optimization: Balancing Architecture, Latency, and Cost

    A practical guide to evaluating and optimizing large language model inference. It explains the three core metrics—TTFT, TPS, and TPOT—and how they define whether a service feels "fast." It then walks through the architectural techniques that raise throughput and cut latency (efficient attention, Mixture-of-Experts, quantization, and KV cache management), the serving infrastructure that turns architectural ceilings into real speed (continuous batching, paged KV cache, speculative decoding, and prefill/decode disaggregation), and the often-overlooked cost of "overthinking" in reasoning models. Finally, it clarifies the logic behind lightweight versus flagship model tiers and offers actionable advice for developers and enterprises on benchmarking, routing, and cost control.

    1. Why Inference Performance Became an Engineering Focus

    As large models move from experimental environments into large-scale online serving, user expectations for “response speed” keep rising. Interaction scenarios that once tolerated several seconds of waiting are now often required to return the first byte within sub-second latency and then keep streaming tokens at a pace close to human reading speed. Behind this experience demand sits an entire set of inference performance metrics, and it is the combined result of model architecture and infrastructure capabilities.

    It is worth stressing that inference performance is never a single-dimension contest. It is constrained simultaneously by model architecture, the inference engine, hardware resources, request scheduling strategies, and input/output lengths. A common product move in the industry is to “launch a new lightweight version and retire an older one.” This is essentially not a simple replacement of old with new, but an engineering decision made after the vendor re-finds a balance point among three constraints: performance, quality, and cost. Understanding this is the prerequisite for objectively evaluating any model iteration.

    2. First, Understand Three Key Performance Metrics

    To judge how fast an inference service is, you must first define the metrics clearly. The industry commonly focuses on the following three quantities:

    • TTFT (Time To First Token): The time from when a request is sent to when the first output token is received. It directly determines how strong the user’s “sense of waiting” is, and is especially critical for interactive scenarios such as chat and code completion.
    • TPS (Tokens Per Second): The speed at which the model continuously emits tokens, which determines the smoothness of long-text generation. Users typically read at a rate of a few tokens per second, so when TPS reaches the hundreds, output speed often already exceeds reading ability.
    • TPOT (Time Per Output Token): The average time needed to produce one token during the steady-state generation phase. It is the reciprocal of TPS.

    The following illustrative pseudocode shows how these three metrics are collected:

    1. def benchmark(model_client, prompt, max_new_tokens=512):
    2. start = time.perf_counter()
    3. response = model_client.stream(prompt, max_new_tokens)
    4. first_token_ts = None
    5. token_count = 0
    6. for token in response: # tokens streamed back one by one
    7. if first_token_ts is None:
    8. first_token_ts = time.perf_counter() # record the first-token timestamp
    9. token_count += 1
    10. end = time.perf_counter()
    11. ttft = first_token_ts - start # first-token latency
    12. tps = token_count / (end - first_token_ts) # steady-state throughput
    13. return {"ttft": ttft, "tps": tps, "total_tokens": token_count}

    Only when TTFT is low enough and TPS is high enough can an inference service be judged as “fast.” And whether these two metrics can be achieved together depends precisely on the design quality of the architecture and the infrastructure.

    3. Architectural Factors That Affect Inference Speed

    To achieve high-performance results such as “hundreds of TPS + sub-second-class TTFT,” simply piling on hardware is not enough—the model architecture itself often has to undergo substantial evolution. Commonly recognized technical approaches in the industry include:

    1. Efficient attention mechanisms. The computation and memory cost of full attention grow with the square of the context length, making it a bottleneck for long-sequence scenarios. As a result, sparse attention, grouped-query attention (GQA), and multi-query attention (MQA) are widely adopted. They reduce bandwidth pressure by cutting down redundant storage of the KV cache, thereby significantly improving decoding speed while preserving quality.

    2. Mixture-of-Experts (MoE) architecture. Through sparse activation, MoE lets each token invoke only a subset of parameters for computation, keeping the per-token compute cost low even when the total parameter count is very large. This is a common path to “a big model that is not slow to run.”

    3. Quantization and low-precision computation. INT8, FP8, and even lower-bit quantization—beyond FP16/BF16—can reduce memory usage and improve operator throughput while keeping precision loss under control. Combining weight quantization with KV cache quantization is a mainstream approach for improving concurrency and speed today.

    4. Context and KV cache management. The size of the KV cache directly determines how much memory a single request can occupy and how long a context it can support. The longer the context, the higher the TTFT and decoding cost. Reasonably designing the context window and reusing cached system prompts are both effective ways to reduce latency.

    When a new version delivers speed metrics that are significantly better than the old one, it usually means that the architectural reconstruction above has already landed; conversely, a model that has “fallen behind” architecturally will struggle to compete with next-generation models on cost-effectiveness, no matter how abundant its hardware is.

    4. Inference Infrastructure: Turning Architectural Advantages into Speed

    Architecture sets the upper bound of speed, while inference infrastructure determines whether that bound can be approached. Key engineering optimizations widely adopted in the industry include:

    1. Continuous batching. Traditional static batching requires all requests in a batch to start and finish together, which easily produces a lot of idle waiting. Continuous batching schedules dynamically at the token level: new requests can be inserted at any time, and finished requests can exit at any time, significantly improving GPU utilization and overall throughput.

    2. Paged KV cache (the PagedAttention idea). Drawing on the operating system’s paging management approach, the KV cache is split into fixed-size blocks that are allocated on demand, avoiding memory fragmentation and reservation waste, so the system can hold more concurrent requests at the same time.

    3. Speculative decoding. A lightweight draft model quickly generates candidate tokens, which the main model then verifies several at a time. If verification passes, it can “skip steps,” reducing decoding latency without changing the output distribution.

    4. Prefill/decode disaggregation. The prefill phase is compute-intensive, while the decode phase is memory-access-intensive. Deploying them into separate resource pools and optimizing each independently avoids mutual interference, letting TTFT and TPS both benefit.

    The following pseudocode illustrates the core idea of continuous batching scheduling:

    1. def continuous_batching_scheduler(request_queue, model, kv_cache):
    2. running = []
    3. while request_queue or running:
    4. # whenever there is a free slot, admit a new request (no need to wait for the same batch)
    5. while has_free_slot(kv_cache) and request_queue:
    6. running.append(request_queue.pop())
    7. # advance each still-running request by one decode step
    8. for req in running:
    9. if not req.is_finished():
    10. model.decode_one_step(req, kv_cache)
    11. else:
    12. release_slots(req, kv_cache) # release immediately and exit once finished
    13. yield collect_ready_tokens(running)

    It is precisely these infrastructure-level capabilities that let the “architectural ceiling” truly materialize as low latency and high throughput that users can perceive.

    5. The Overlooked Cost Variable: “Overthinking” in Reasoning Models

    In recent years, more and more models have introduced explicit reasoning capability—generating an internal thought process (Thinking) first, then giving the final answer. This design markedly improves performance on complex tasks, but it also brings an easily overlooked problem: overthinking.

    When a model generates large amounts of verbose thinking tokens even for simple questions, three costs appear:

    1. Higher cost. Billing is usually metered by the number of tokens actually generated and processed, and the thinking process is charged as well. Even if a single response is faster, verbose thinking tokens can make the “total token cost of completing the task” rise rather than fall.
    2. Longer latency. Thinking tokens delay the appearance of the final answer, which hurts in scenarios sensitive to perceived TTFT.
    3. Wasted resources. Simple tasks could be answered directly; over-reasoning is essentially trading compute for unnecessary intermediate output.

    A typical illustration of the trade-off is as follows:

    1. No thinking (direct output): input 800 tokens + output 200 tokens → 1000 tokens total, low TTFT
    2. Overthinking: input 800 tokens + thinking 1500 + output 200 → 2500 tokens total, high TTFT

    However, it is worth noting that when the base throughput is high enough (for example, reaching hundreds of TPS), the time cost of verbose thinking is greatly diluted, and the user’s perceived difference becomes noticeably smaller. At that point, the main problem of “overthinking” shifts to the cost side rather than the speed side. So when evaluating such models, the right approach is not to blindly pursue “faster,” but to include the “end-to-end token cost of completing the task” as an independent metric in the evaluation.

    6. Model Tiering: Why There Are “Lightweight” and “Flagship” Lines

    Once you understand the trade-off between performance and cost, it becomes clear why the model tiering strategy common in the industry is reasonable. Two positioning categories are usually defined:

    • Lightweight / high-speed line: Emphasizes low latency, high throughput, and controllable cost. It suits high-concurrency, real-time-sensitive online scenarios such as intelligent completion, customer-service Q&A, and batch data processing. The cost-effectiveness advantage of this line scales up with concurrency.
    • Flagship / capability line: Pursues the strongest quality on complex reasoning and long-chain tasks, at the cost of higher compute overhead. It is better suited to quality-first, cost-tolerant scenarios such as offline batch processing, complex analysis, and high-value generation.

    When a new version of the high-speed line is already strong enough in capability while the flagship line is not yet mature, a rational product strategy is often to first route traffic to the mature high-speed line, using large-scale real usage to refine its capabilities and balance compute overhead, then launch a higher-positioned flagship version when the time is right. For users, the key is not to chase the “latest version,” but to clarify priorities within the triangle of quality, latency, and cost based on your own scenario, and then make a selection.

    7. Practical Advice for Developers and Enterprises

    Based on the analysis above, here are some actionable recommendations:

    1. Build your own evaluation baseline. Do not rely on a single “overall score”; instead, target your real business distribution and measure TTFT, TPS, and the “total number of tokens to complete the task” at the same time, so you are not misled by a single metric.
    2. Route by scenario tier. Route simple, high-frequency requests to high-speed-line models and complex, low-frequency requests to flagship-line models, using the routing layer to balance overall cost and quality.
    3. Control thinking tokens. If your business is cost-sensitive, prefer models with controllable thinking, and suppress over-reasoning on simple tasks through prompt design, task classification, and similar techniques.
    4. Optimize inference infrastructure. In self-hosted serving scenarios, make good use of continuous batching, KV cache reuse, speculative decoding, and prefill/decode disaggregation to turn architectural advantages into real speed.
    5. Continuously track architectural evolution. A significant version upgrade often corresponds to a reconstruction of architecture and infrastructure; paying attention to these underlying changes helps you make rational selection decisions far better than blindly chasing version numbers.

    In summary, when evaluating a model iteration, you cannot look only at whether it is “new” or “fast.” Instead, you must weigh architectural capability, infrastructure quality, and end-to-end cost together on the same scorecard. Only by building a clear system for measuring performance and cost can you make engineering choices that genuinely fit business value—balancing performance, quality, and cost.