Output Speed Becomes a Key Metric in the Next Phase of LLM Competition
As large language models move from one-shot answers into real products, output speed is becoming a productized, priced, and SLA-bound capability. This article explains why speed now matters, how to separate single-stream latency from system throughput, and how model design, inference systems, deployment, and service governance must be optimized together.
For the past few years, competition in the large language model industry followed a relatively clear path: parameter scale, training compute, and benchmark scores. Scaling laws pushed model capability higher, leaderboard results became milestone markers, and the industry’s main question was how far a model could go. As models enter real business workloads, another metric that used to stay in the background is appearing more often: output speed. It is no longer only an internal parameter of the inference engine. It is becoming an independent capability that can be productized, priced, and constrained by service level agreements.
1. From “Can Answer” to “Can Interact”: What the Speed Race Is Really About
Speed has become central in this next phase first because the way models are used has changed. Early applications were mostly one-shot question answering. A user could submit a question and wait several seconds, or longer, and still find that acceptable. In coding assistants, real-time customer support, document assistants, agents, and multi-turn conversations, model output sits inside a continuous interaction loop. Waiting time turns directly into a worse experience.
In code completion, high time to first token interrupts a developer’s train of thought. In long-form generation, a slow single stream makes users feel that they are waiting token by token. In agent workflows, one task often includes multiple planning steps, tool calls, and result aggregation, so delay at each step is amplified across the chain. Speed then affects not only retention, but also task completion rate, concurrent capacity, and cost per request.
More importantly, speed is gaining commercial meaning. The industry now offers high-speed service tiers, dedicated inference pools, and pricing that varies by speed. “Faster” is moving from a back-end optimization goal to a front-end product capability. Speed is not a single metric, though, and higher is not always better. A speed capability that can actually be delivered must balance quality, cost, and stability.
2. Breaking Down Speed Metrics: A Fast Single Stream Is Not System Capacity
Before discussing speed, the metrics have to be separated. Otherwise it is easy to mistake “this model generates quickly on one stream” for “the whole service has high throughput.”
Common metrics include:
- Time to first token (TTFT): the time from sending a request to receiving the first token. It is affected by queuing, prefill computation, cache hits, and network access. It determines whether the response feels fast.
- Time per output token (TPOT), or inter-token latency: the average interval between adjacent tokens after the first token. It determines how fluid a long output feels.
- Single-stream output speed: usually measured in output tokens per second. It is related to TPOT, but the measurement definition matters.
- End-to-end latency: the time from sending the request to receiving the last token. It is approximately queue time plus prefill time plus output length multiplied by TPOT, plus network transfer time.
- System throughput: total output tokens produced by the system per unit of time. It depends on concurrency, batch size, memory bandwidth, and scheduling policy.
- Tail latency: P95 and P99 latency, along with timeout rate. In real workloads, tail latency often matters more to user experience than the average.
- Cost per token and cost per request: a speed gain that wastes resources is not commercially sustainable.
A simplified view is:
end-to-end latency ≈ queue time + prefill time + output length × TPOT + network timesystem throughput ≈ tokens processed per step / time per step
These formulas expose a common tradeoff. Larger batches can raise system throughput, but they may increase queueing and per-request latency. Pushing single-stream latency extremely low can sacrifice concurrent throughput. A mature service therefore does not chase one “fastest number.” It offers different tiers by scenario: interactive completion needs low time to first token, long-form generation needs stable TPOT, and offline batch processing cares more about throughput and cost.
3. What a “Speed Frontier” Shows: Joint Optimization of Model, System, and Deployment
Public discussion often uses a “speed frontier” view. Model scale is on the horizontal axis, single-stream output speed is on the vertical axis, and the highest observed speeds form an envelope. The point is not that bigger is faster. It is that, under the combined effect of model architecture, the inference system, and hardware deployment, speed has a boundary that can keep moving forward.
What deserves more attention is that the same model can differ in speed by an order of magnitude under different deployment choices. Public cases spanning from tens of tokens per second to the thousand-token-per-second range are not rare. The differences usually come from:
- Model structure: sparse activation, mixture of experts, attention design, vocabulary size, and how quantization-friendly the model is all affect decoding cost.
- Inference system: continuous batching, prefix caching, paged key-value cache management, speculative decoding, operator fusion, graph compilation, parallelism, and separation of prefill and decode.
- Hardware and interconnect: memory bandwidth, interconnect bandwidth, rack-level deployment, and thermal and power limits.
- Service policy: dedicated high-speed pools, request routing, rate limiting, cache warm-up, and elastic scaling.
Speed is therefore not decided by a single model parameter. It is a systems problem spanning the model, inference system, deployment design, and service policy. Competition in this phase is also shifting from point-model capability toward end-to-end delivery efficiency.
4. Technical Paths: Layered Optimization from Model to Service
4.1 Model Layer: Control Activation Cost and Improve Decoding Efficiency
Model-level optimization starts by distinguishing total parameters from activated parameters. Mixture-of-experts designs use sparse activation to limit computation per step, but routing balance, expert parallelism, and communication overhead affect real speed. Distillation and small-model collaboration can support a high-speed tier: a lightweight model handles low-complexity requests, while complex requests are routed to a stronger model. Quantization is a common method, but it needs calibration and accuracy regression. Its speed benefit is constrained by memory bandwidth and operator support. Attention and key-value cache optimization are equally important, including cache compression, tiered storage, sliding windows, and grouped-query attention. Speculative decoding uses a draft model to propose candidate tokens that the main model then verifies. The gain depends on the acceptance rate, so linear speedup should not be assumed.
4.2 Inference System Layer: Scheduling and Caching Shape the Real Experience
The core of the inference system is to keep hardware busy while controlling tail latency. Continuous batching and dynamic batching can merge requests and raise throughput, but batch size and waiting time must be limited. Prefix caching suits system prompts, long documents, and reused fixed context, but tenant isolation and invalidation must be handled carefully. Paged key-value cache management can reduce memory fragmentation and raise concurrency. Separating prefill and decode lets the first-token stage and the sustained generation stage use different resource pools. Parallelism and compilation must account for communication overhead; more parallelism is not automatically faster.
A simplified request-routing sketch is:
function route(request):if request.speed_tier == "ultra" and ultra_pool.has_capacity():return ultra_poolif request.prefix_cache_hit and request.max_latency_ms > threshold:return cached_poolif request.output_length_estimate > long_output_threshold:return throughput_poolreturn elastic_pool
The goal is not simply to push every request into the fastest pool. Routing should match tier, cache hit, output length, and deadline so that long tasks do not crowd out the high-speed pool.
4.3 Deployment and Service Layer: Speed Must Be Productized and Observable
Once speed is a product capability, the service layer needs tiered offerings, resource isolation, service level agreements, rate limits, and degradation policies. A high-speed tier cannot rely only on higher priority, or it will harm fairness for ordinary requests. Elastic scaling must account for model loading, cache warm-up, and cold starts. Multi-tenant environments need isolation. An observability system should at least cover time to first token, TPOT, P99 latency, queue length, cache hit rate, batch size, hardware utilization, and timeout rate.
Metric collection can be expressed as:
record(ttft_ms = first_token_time - request_time)record(tpot_ms = (last_token_time - first_token_time) / max(output_tokens - 1, 1))record(output_tps = output_tokens / max(last_token_time - first_token_time, 1))record(e2e_ms = last_token_time - request_time)record(queue_ms = start_time - request_time)
In production, general-purpose capabilities such as the container platform, monitoring and alerting, and logging services offered by Baidu AI Cloud can be used to build an elastic, observable foundation for inference services. The emphasis is on using these general capabilities for resource orchestration, metric collection, and a closed alerting loop, rather than depending on a single-point optimization.
5. Engineering Practice: Build an Evaluation and Optimization Loop
Speed optimization cannot be based on intuition. Start by defining scenario-level targets: time to first token, TPOT, P99 end-to-end latency, timeout rate, and cost per token. Load tests should cover output-length distribution, concurrency spikes, cold and warm caches, and mixed multi-tenant load, and should distinguish open-loop from closed-loop testing. A practical order is to optimize caching and routing first, then batching and quantization, and only then parallelism and hardware changes. Change one variable at a time, and watch speed, quality, and cost together.
Risks also need management. Quantization can reduce accuracy. Speculative decoding acceptance rates vary by task. Batching can raise tail latency. Caches can become polluted. Multi-tenant high-speed pools need isolation. Rollouts should use gradual release and A/B evaluation, maintain a quality regression set, and keep degradation and rollback options.
6. Conclusion
As large language model competition enters its next phase, speed is moving from a back-end metric to a front-end competitive advantage. Real speed capability is not a single number. It is a systems effort made up of metric definitions, model optimization, the inference system, deployment strategy, and service governance. Teams should build a complete speed metric set, optimize by layer and by scenario, use service tiers to balance experience and cost, and rely on observability and gradual release to protect stability. Those who can make speed deliverable, measurable, and governable as a product capability are better positioned for the next stage.
