Inference Speed Is Becoming a New Competitive Differentiator for Large Language Models

    As large language model competition moves into productization, parameter counts and benchmark scores no longer explain why users stay or leave. This article explains why inference speed has become a standalone product capability, how to define latency and throughput metrics, why identical model weights can differ by an order of magnitude across deployments, how to build a speed frontier, how to package speed as service tiers, and how to optimize and architect serving systems without sacrificing quality.

    After large language model competition entered the deep productization phase, comparing parameter scale or leaderboard scores alone can no longer explain why users stay or leave. Once interaction shifts from “ask a question and wait for one response” to multi-turn tool calling, long-context reading, and continuous generation, speed rises from a user-experience detail to a standalone capability: it determines task completion time, concurrency cost, and whether the system is stable and predictable.

    Why Speed Has Become a Standalone Product Capability

    Early model selection focused more on whether answers were correct. When most models can perform summarization, rewriting, code completion, and structured extraction, differentiation shifts to whether responses are fast enough, stable enough, and cheap enough. Speed changes at least three things.

    First, interaction expectations. If time to first token is too long, users assume the service is stuck; if output is too slow, long answers are abandoned midway. For a coding assistant, if no completion candidate appears within the completion window, the feature effectively does not exist.

    Second, agent economics. An agent loop is often a serial chain of “plan—call a tool—generate again,” and tokens per second at each step are amplified across the entire chain. A 200 ms delay per step becomes seconds or even tens of seconds across ten tool calls.

    Third, capacity and cost. With the same compute, faster single-stream generation usually means higher effective throughput, or lower cost per token at the same throughput. As a result, many services have begun to offer “high-speed tiers” as purchasable service levels: charging a higher unit price for a tighter latency target rather than putting all traffic on the same deployment.

    Define the Metrics First

    Before discussing speed, the metrics must be separated; otherwise numbers from different deployments are easily compared incorrectly.

    • Time to first token (TTFT): the time from request arrival to the first emitted token, affected by queueing, prefill length, and scheduling policy.
    • Time per output token (TPOT) or decoding speed: after decoding begins, the average interval between output tokens, closer to the “typing speed” users perceive.
    • Single-stream output speed: the generation rate of one request under exclusive or low-concurrency conditions, useful for measuring the upper bound of interactive experience.
    • System throughput: the total number of tokens completed per unit time, useful for capacity planning. High throughput does not mean fast single-stream speed; large batch merging often sacrifices per-request latency.
    • Goodput: throughput under a latency service-level objective (SLO), which is closer to real business conditions than peak throughput.

    In addition, input length, output length, concurrency, precision (such as weight quantization bit width), whether speculative decoding is enabled, and whether network round-trip time is included must all be fixed. Reporting only “tokens per second” without conditions produces conclusions that are almost impossible to reuse.

    Why the Same Model Can Differ by an Order of Magnitude

    A common phenomenon in practice is that the same weights can differ several-fold or even more than tenfold in single-stream speed when the deployment form changes. This is usually not because the model suddenly becomes smarter, but because the inference system reorganizes computation, memory access, and scheduling.

    The decoding stage is often memory-bound. Each generated token requires reading weights and KV cache associated with the number of layers, hidden dimension, and context length. If the batch is too small, arithmetic units are underutilized; if the batch is too large, queueing and tail latency increase. Therefore, an “official default service” that prioritizes high concurrency and low cost may deliver only tens of tokens per second for a single stream, while a deployment optimized for low concurrency, short queues, specialized kernels, and high-speed interconnect may push single-stream speed to hundreds or even thousands of tokens per second. The two have different optimization objectives and cannot substitute for each other.

    Main levers include:

    1. Matching compute and memory access. Operator fusion, paged KV cache, and continuous batching reduce kernel launches and memory fragmentation.
    2. Parallelism strategy. Tensor parallelism splits weights, pipeline parallelism splits layers, and expert parallelism serves sparse models. Communication topology and collective-communication efficiency determine whether scaling is worthwhile.
    3. Precision and kernels. Weight quantization, activation quantization, and KV-cache compression reduce bandwidth pressure, but quality regression must be validated separately rather than assuming that faster is always better.
    4. Speculative decoding and draft models. A small model drafts multiple tokens, and the target model verifies them in one pass, reducing decoding steps when the acceptance rate is high enough. The acceptance rate varies with task distribution and must be measured on the business corpus.
    5. Scheduling and isolation. Separate interactive traffic from offline batch processing so long prefills do not block short requests; use shallower queues and stricter timeouts for high-speed tiers.

    These are general industry techniques. Their effects depend heavily on model architecture, sequence length, and hardware form factor. There is no “universal speedup ratio” detached from the scenario.

    Build a Speed Frontier Rather Than Chasing a Single Peak

    By placing “published open-weight models that reach a certain parameter scale” on a parameter-count versus single-stream-speed plane, and connecting the highest speed at each scale, an empirical speed frontier can be obtained. Its purpose is not to award any particular solution, but to answer: at a comparable scale, how far is my deployment from the publicly visible engineering ceiling, and does the gap come from the model, the kernels, or the serving policy?

    When reading such charts, note sample bias. The models included are often from teams willing to publish their deployment plans; with the same total parameter count, active parameter count, attention implementation, and context length may be completely different; “top speed” often comes from a specifically optimized inference stack rather than a default reproducible environment. Therefore, a frontier is suitable as a reference coordinate system, not as a procurement commitment.

    A more reliable internal benchmark can be designed in four layers:

    • Model layer: fixed weights and tokenizer; record active parameters and context limit.
    • Runtime layer: batch size, parallelism, precision, KV strategy, and whether speculative decoding is enabled.
    • Serving layer: concurrency, queues, timeouts, and streaming protocol overhead.
    • Business layer: real prompt distribution, tool-call ratio, and acceptable quality threshold.

    Change only one factor at a time so that a “tenfold gap” can be attributed to actionable changes.

    Productization: Make Speed a Selectable Capability

    Speed should not exist only as extreme numbers in technical blog posts; it should map to service levels. A common approach is to provide a standard tier and a high-speed tier: the standard tier optimizes unit cost and overall throughput, while the high-speed tier optimizes TTFT and single-stream decoding speed, with limits on concurrency or more aggressive kernels and hardware allocation. Billing can use token-based pricing with a tier coefficient, but the actual price must be determined by the provider’s own cost model and should not copy external rumors.

    At launch, it is advisable to publish:

    • Target latency ranges (for example, P50 and P95 for TTFT and TPOT), not just averages.
    • Applicable input and output lengths; very long context should be degraded or separately priced.
    • Quality constraints: if the high-speed path changes precision or decoding strategy, provide sampled comparison results against the baseline.
    • Failure semantics: queue timeouts, overload rejection, and retry rules after a stream interruption.

    For enterprise users, speed tiers should be able to enter gateway policies: route by application, tenant, or scenario. For example, online chat and code completion can use the high-speed queue, while overnight knowledge-base rebuilding can use the throughput queue. Routing rules should be observable and reversible.

    A Practical Optimization Sequence

    Do not start by changing the model architecture. The recommended order is to measure first, then eliminate obvious waste, and only then apply higher-risk acceleration.

    Step 1: instrument. Record queue entry, prefill start, first token, every N tokens, and completion separately at the gateway, scheduler, and inference process. Distinguish client-side time from server-side time.

    Step 2: look at the queue. If most of TTFT is queueing, adding machines or splitting queues is more effective than changing kernels.

    Step 3: look at prefill. When long prompts saturate compute, cache common prefixes or separate static system prompts from user content to reuse KV.

    Step 4: look at decoding bandwidth. Confirm whether KV layout, batching strategy, and quantization make decoding bandwidth-bound.

    Step 5: evaluate speculative decoding, finer parallelism, or new hardware. Every step must retain a quality regression set: faithfulness, format compliance, code compile rate, and so on, to avoid optimizing only for speed.

    An illustrative single-stream benchmark can be very simple; the key is to lock down conditions and take percentiles over multiple trials:

    1. for each trial:
    2. t0 = now()
    3. stream = infer(prompt, max_new_tokens=N, temperature=0)
    4. t_first = time of first token
    5. t_end = time of last token
    6. ttft = t_first - t0
    7. decode_tps = (N - 1) / (t_end - t_first)
    8. report p50/p95 of ttft and decode_tps

    Temperature, stop sequences, and streaming packetization all change the results; comparison experiments must lock them down.

    How Architecture Should Support High-Speed Inference

    Without supporting architecture, high-speed-tier peaks will appear only in demo environments. Production requires at least: a unified model gateway for authentication, rate limiting, and routing; an inference service that supports streaming output and graceful traffic draining; a container or compute platform that scales based on queue depth rather than CPU alone; and logging and monitoring that place TTFT, TPOT, queue length, GPU utilization, and rejection rate in the same view. Object storage is suitable for weights and prompt-cache snapshots, and message queues are suitable for peak shaving, but the online decoding path itself should be as short as possible. Do not add extra hops merely for “architectural completeness.”

    If a team already deploys inference services in a cloud environment such as Baidu AI Cloud, it should prioritize connecting the general capabilities above—container orchestration, monitoring and alerting, log search, and autoscaling—to its own SLOs rather than assuming the platform will automatically provide a certain speed. The cloud provides resources and operations interfaces; speed still depends on the model, runtime, and traffic pattern.

    Conclusion

    From tens of tokens per second to hundreds or even thousands of tokens per second, the engineering space for inference efficiency has been opened, but the numbers themselves are not competitiveness. Competitiveness comes from defining experience with clear metrics, delivering speed through isolated service levels, constraining optimization with reproducible benchmarks, and reducing latency and cost without collapsing quality. In the next phase of large language model competition, what matters is not who reports the highest peak, but who can make speed a stable, purchasable, and verifiable product capability.