The Second Half of the LLM Race: Why Inference Speed Is Becoming a Core Capability

    As foundation models converge on similar capabilities, response speed is moving to the center of product competition. This article explains why latency, sustained generation rate, concurrency, and tail latency now shape user experience and cost, why decoding is hard to accelerate, which optimization paths are commonly used, and how to evaluate and deploy fast inference without sacrificing quality or reliability.

    For some time, the large language model industry has focused on parameter scale, context length, reasoning ability, and benchmark scores. Model launches have often centered on whether a system can answer harder questions, support longer context, or deliver stronger coding and multimodal capabilities. As foundation models grow closer in capability, another metric is quickly moving to the center of product competition: how fast a model can complete a response.

    This shift is not a simple chase for higher tokens per second. It is driven by real application needs. Online customer support, intelligent search, code completion, real-time productivity assistants, and voice interaction all require the model to respond quickly. Users generally do not want to wait until a full answer is generated before seeing anything. They want the first useful content in a very short time, followed by steady output. Speed has therefore moved from an infrastructure performance metric to a business capability that directly affects product experience.

    1. Model speed is more than “words generated per second”

    Evaluating large model speed requires at least three metrics.

    The first is time to first token: the time from request submission to the first useful token. It is affected mainly by request queuing, input processing, model computation, and network transfer. In conversation, search, and code completion, time to first token often influences subjective experience more than total completion time.

    The second is sustained speed during generation, usually measured as output tokens per second. This reflects computational efficiency in the decoding stage. Higher sustained speed returns long answers, generated code, and structured content faster.

    The third is overall throughput: the total number of tokens or requests the system can process per unit of time. High speed for a single user does not equal high concurrency. If a system reaches very high output speed only under low concurrency and then queues severely as requests increase, it still cannot support production.

    A more complete expression of performance is therefore:

    End-to-end response efficiency = time to first token + stable output speed + concurrent throughput + tail-latency control

    Tail latency deserves particular attention. A low average response time does not mean every user gets a stable experience. Organizations should watch high-percentile metrics such as P95 and P99 so that a small number of slow requests do not drag down the perceived quality of the service.

    2. Why speed is becoming a new focus of LLM competition

    2.1 Interaction is shifting from “waiting for a result” to real-time collaboration

    Traditional software usually completes work through buttons, pages, and fixed workflows. Generative AI is closer to a real-time collaborator. After a user enters a question, they expect the system to start responding immediately and to keep advancing the task with context.

    In coding, if a completion suggestion takes several seconds, the user may already have continued typing. In search, the earlier an answer appears, the more likely the user is to ask a follow-up. In voice interaction, high latency creates an obvious sense of pause. Speed determines whether interaction feels natural and whether users are willing to embed the model in everyday workflows.

    2.2 Capability gaps are narrowing while experience gaps are widening

    When several models can handle common question answering, summarization, translation, and code generation, simply adding parameters does not necessarily produce proportional product gains. Services that respond quickly, stay stable, and keep cost under control are more likely to differentiate.

    This is especially true in enterprise applications, where a model does not handle only one request. Systems often face high concurrency, long context, complex tool use, and strict service-level objectives. Higher speed means the same hardware can carry more requests and overall resource utilization improves.

    2.3 Faster inference can improve cost structure

    Inference cost is determined not only by the computation of a single request, but also by hardware occupancy time, concurrency, memory use, and scheduling efficiency. Higher generation speed means the same computing resources can finish more work in less time.

    Speed and cost are not simply linear, however. Higher speed may require higher-spec accelerators, greater parallelism, or more complex caching. The valuable optimization is not an unconditional pursuit of peak speed, but a balance among unit cost, target latency, and business quality.

    3. Why large model inference is hard to keep accelerating

    Large model inference is usually divided into prefill and decode.

    Prefill processes the user’s input context. Longer input generally means more computation, but this stage is highly parallel and suits the matrix-compute strengths of accelerators.

    Decode generates content token by token. Each new token must be computed from the preceding context, so the stage is inherently somewhat serial. Many online applications are limited precisely by decode efficiency.

    The system must also handle key-value cache, request scheduling, memory reads and writes, batch organization, and network transfer. Even if a model’s theoretical compute performance is high, weak cache management or poor request scheduling can still leave users with high latency.

    Inference acceleration therefore usually requires coordinated optimization at several layers:

    • Model: quantization, pruning, sparsity, or more efficient architectures to reduce computation and memory use
    • Compilation: operator fusion, memory-layout optimization, and computation-graph compilation to cut unnecessary overhead
    • Runtime: better batching, cache reuse, request scheduling, and dynamic resource allocation
    • Hardware: full use of parallel compute, memory bandwidth, and interconnects on accelerators
    • Serving: tiered scheduling by request length, priority, and business type

    4. Common paths for speeding up large models

    4.1 Quantization reduces compute and storage pressure

    Quantization is one of the more widely used inference optimizations. It represents weights or intermediate data at lower bit widths, reducing memory use and data movement.

    Quantization does not necessarily cause a clear drop in accuracy. Results depend on quantization granularity, calibration data, model structure, and task type. For tasks that demand factual accuracy, mathematical reasoning, or code generation, quality changes should be checked with offline evaluation rather than judged only by the speed gain.

    4.2 Continuous batching improves concurrency utilization

    Traditional batching often waits until a batch of requests is ready and then runs them together. Online requests arrive at different times and have different generation lengths, which can leave resources idle. Continuous batching lets the system add new requests dynamically and remove finished ones promptly, raising utilization.

    This is especially useful for online services whose request volume fluctuates. Larger batches are not always better. An oversized batch can increase waiting time and raise time to first token. The system needs to adjust dynamically between throughput and response speed according to live load.

    4.3 KV cache reduces repeated computation

    During token-by-token decoding, the model repeatedly uses historical context. Caching intermediate results that have already been computed avoids reprocessing the full context at every step.

    The key issues are memory management, cache reuse, how requests of different lengths are organized, and eviction policy. In long-context scenarios, the KV cache can consume a large amount of memory and even become the main limit on concurrency, so it needs techniques such as paging, compression, and tiered storage.

    4.4 Speculative decoding reduces serial waiting

    Speculative decoding typically uses a smaller draft model to propose several candidate tokens, which the target model then verifies in a batch. If the candidates match the target model, multiple tokens can be accepted at once, reducing the number of step-by-step decode passes.

    The gain depends on the draft model’s prediction accuracy, verification overhead, and task type. It may work better for more regular text generation, and less well for complex reasoning or tasks with large distribution shifts. It should be validated on real business data before deployment.

    4.5 Expert routing and sparse computation

    Some models use sparse activation, calling only part of the parameters for each request. This can keep a large total parameter count while lowering computation per request.

    Sparse designs also create routing, communication, and load-balancing problems. If many requests concentrate on a few compute units, local congestion can appear. Architecture choices therefore have to be designed together with runtime scheduling and hardware interconnect capacity.

    5. How to build a reliable speed evaluation system

    When comparing models or deployment options, organizations should not look only at advertised peak speed. An effective evaluation should at least include:

    1. Fixed input and output lengths, so different test conditions do not distort results
    2. Separate measurement of time to first token, average generation speed, and total completion time
    3. Coverage of single-request, low-concurrency, and high-concurrency scenarios
    4. P50, P95, P99, and other percentile latencies
    5. Memory use, device utilization, and unit cost recorded together
    6. Answer quality and stability checked on real business samples

    Overall efficiency can be expressed as:

    Unit-cost efficiency = effective tokens processed per unit time ÷ total compute and serving cost

    “Effective tokens” should not mean that more output is always better. If a model produces a large amount of redundant content, token speed may look high while the user receives no additional useful information. Evaluation should also consider task completion rate, answer usability, and the cost of the user’s next actions.

    6. Practical advice for deploying fast inference

    First, choose the optimization direction from the business goal. Real-time interaction should prioritize lower time to first token. Batch content generation should emphasize total throughput. Long-text processing should focus on cache and memory management.

    Second, use a tiered model strategy. Simpler tasks can use lighter models, and harder tasks can be handed to higher-capability models. Routing that avoids unnecessary computation is often more cost-effective than simply upgrading hardware.

    Third, build complete monitoring. Beyond request count and average latency, track queue time, input and output token counts, cache hit rate, device utilization, error rate, and tail requests. Only by relating these metrics can a team tell whether a bottleneck comes from the model, hardware, network, or scheduler.

    Finally, speed optimization must not come at the expense of reliability. Quantization, context truncation, and aggressive batching can all affect answer quality or stability. Organizations should use gradual rollout, regression evaluation, and automatic rollback so that optimizations remain controllable under real traffic.

    Conclusion

    In the second half of large model competition, simply scaling parameters is no longer the only direction. Faster time to first token, higher sustained output speed, more stable concurrent throughput, and lower unit cost are together forming a new competitive edge for model services.

    Future systems will be judged not only by how capable the model is, but also by whether that capability can be delivered to more users with lower latency, higher stability, and a more reasonable cost. For developers and organizations, speed should no longer be an add-on metric after deployment. It should be designed across model selection, system architecture, hardware planning, and business evaluation. Only when algorithms, compilation, runtime, and service governance work together can fast inference become durable product value.