From Parameter Race to Speed Race: System-Level Techniques for Faster LLM Inference

1. The Focus of LLM Competition Is Shifting

In the early days of large language models, industry attention centered on parameter scale, training data volume, and benchmark rankings. As foundation model capabilities gradually converge, the marginal returns from simply competing on parameter count have begun to decline. More and more teams have realized that the key factor affecting user experience and business deployment has shifted from “what a model can do” to “how fast a model can do it.”

This shift is especially pronounced in inference. In the past, users interacting with conversational or generative services often had to wait several seconds or even tens of seconds for a complete response. Today, models of comparable quality can produce output at speeds ranging from a few dozen tokens per second to over a thousand tokens per second. A change of this magnitude is not merely an experience improvement—it directly affects serving cost, concurrency capacity, and the range of viable applications.

Faster inference means the same hardware resources can handle more user requests, and the per-token serving cost drops accordingly. For enterprises, this is a critical precondition for moving from “can we use large models” to “can we use large models at scale.” Speed has become a new source of competitive advantage.

2. High-Speed Inference Is Not a Single Model Metric

Observations across the industry show that solutions at the speed frontier tend to share one common trait: they do not rely on a single model alone, but instead co-design model architecture with the inference system. For example, a model with roughly 300B total parameters, paired with a purpose-built high-speed inference system, can achieve single-stream output of approximately 2,000 tokens per second. The same class of model deployed without dedicated optimization might reach only a few dozen tokens per second—a gap of more than tenfold.

This demonstrates that inference speed is fundamentally a systems engineering problem, not an isolated model parameter problem. It is influenced by at least the following factors:

1. Model architecture and compute density: including layer count, hidden dimensions, number of attention heads, vocabulary size, activation function type, and whether sparse structures are used;
2. Inference engine and runtime optimization: including operator fusion, automatic kernel tuning, memory pool reuse, and dynamic-to-static graph conversion;
3. Hardware topology and communication patterns: including per-GPU compute throughput, memory bandwidth, inter-GPU interconnect, and inter-node network latency;
4. Serving-layer scheduling strategies: including request queuing, dynamic batching, preemptive scheduling, priority management, and cache hit policies.

Focusing only on a model’s floating-point operation count while ignoring system-level co-optimization makes it difficult to approach the true speed boundary.

3. System Optimization Is the Core Lever for Speed

3.1 Operator-Level Optimization in the Inference Engine

During LLM inference, attention computation, matrix multiplication, LayerNorm, Softmax, activation functions, and sampling operations account for most of the time. General-purpose deep learning frameworks perform well in usability and flexibility, but often introduce overhead for specific model architectures.

Inference engines apply targeted optimizations to these high-frequency operators. For example, fusing multiple small operators into a single kernel reduces the number of intermediate result reads and writes to memory; using tiling strategies for matrix multiplication improves data locality; and applying FlashAttention-style approaches to attention computation avoids explicitly constructing the full attention matrix, reducing memory usage and improving computational efficiency.

These optimizations typically do not change model output but can deliver several-fold speed improvements. For long-sequence scenarios, the gains are even more pronounced.

3.2 Dynamic Batching and Continuous Batching

In online serving scenarios, user request arrival times are unpredictable. If each request occupies its own forward pass, hardware utilization drops significantly, especially during periods of low request density.

Dynamic batching allows the system to merge multiple requests arriving within a short window into a single batch for computation. Unlike fixed batch sizes, dynamic batching adjusts batch size in real time based on current load, balancing throughput and latency.

Continuous batching further refines scheduling granularity. In the traditional approach, a batch must wait for all sequences to finish generating before releasing resources. Continuous batching allows completed sequences to exit early while new requests immediately fill the freed slots. As a result, hardware resources remain nearly saturated at all times, significantly improving overall throughput.

3.3 KV Cache Management and Reuse

During autoregressive generation, the model must compute key-value caches for historical tokens and reuse them when generating subsequent tokens. As sequence length grows, KV cache memory usage rises rapidly, becoming a key constraint on concurrency and long-context inference.

Optimization approaches include:

• KV cache quantization: compressing the cache from high-precision floating point to lower-precision representations, reducing memory usage and read bandwidth pressure;
• KV cache eviction strategies: dynamically retaining cache for important tokens based on attention distribution, lowering peak memory usage in long-sequence scenarios;
• Prefix cache reuse: for requests sharing a common prefix—such as system prompts or invariant portions of multi-turn conversations—caching the computed result once so subsequent requests can reuse it directly, avoiding redundant computation.

In real-world workloads, these strategies can significantly improve cache hit rates and reduce computational redundancy.

3.4 Quantization, Pruning, and Sparsification

Model compression techniques directly affect inference speed.

• Quantization: compressing weights and activations from FP16/BF16 to lower-precision representations such as INT8 or INT4, reducing memory bandwidth requirements and computational overhead. Most mainstream inference engines support mixed-precision strategies, retaining higher precision for sensitive layers and applying more aggressive quantization to redundant layers.
• Pruning: removing weights or structures that contribute little, reducing the number of parameters actually involved in computation.
• Sparsification: leveraging structured sparsity patterns to skip computation on zero or low-magnitude parameters. If hardware and operator libraries can efficiently support sparse matrix operations, inference can be further accelerated.

It is important to note that compression techniques require a trade-off between speed and output quality. A sound strategy combines precision sensitivity analysis with differentiated treatment across layers, rather than applying a one-size-fits-all approach.

3.5 Parallelism Strategies and Hardware Topology Matching

When model size exceeds what a single GPU’s memory can hold, strategies such as tensor parallelism and pipeline parallelism become necessary. The choice of parallelism strategy has a substantial impact on inference speed.

• Tensor parallelism: splitting computation within a single layer across multiple accelerators; involves frequent inter-GPU communication, making it suitable for high-speed interconnect environments;
• Pipeline parallelism: distributing different layers across different devices; involves less frequent communication but may introduce device idle waiting;
• Expert parallelism: for mixture-of-experts models, placing different experts on different devices and dispatching computation based on routing results.

In practice, the optimal parallelism combination must be selected based on hardware topology. For example, single-node multi-GPU environments can use tensor parallelism to reduce communication bottlenecks; cross-node environments should minimize inter-node communication frequency, favoring pipeline parallelism or data parallelism.

4. Deployment Form Has a Significant Impact on Inference Speed

The same model deployed on a general-purpose official service versus a purpose-optimized inference cluster can differ in speed by several-fold or even more than tenfold. Behind this phenomenon lies a difference in service objectives and resource investment.

General-purpose services prioritize stability and compatibility, typically adopting conservative batching strategies and default configurations to handle diverse and unpredictable user requests. Purpose-optimized deployments, by contrast, can adjust kernel parameters, pre-configure caches, optimize memory layouts, and select more suitable hardware combinations based on model characteristics.

For enterprise users pursuing high performance, targeted choices should be made in the following areas:

• Inference engine selection: evaluate measured throughput and latency of different inference engines on the target model;
• Hardware matching: select accelerator hardware with memory capacity, bandwidth, and compute throughput matched to model size and sequence length;
• Parameter tuning: systematically search over batch size, parallelism degree, quantization precision, and cache strategies;
• Load testing: test under request distributions close to real business workloads, rather than relying solely on idealized benchmark data.

5. From “Fast” to “Stable”: Engineering Considerations for High-Speed Inference

High-speed inference is not just a peak-speed problem. In production environments, stability, scalability, and cost control are equally important.

On one hand, high-throughput services require more refined resource management. For example, when request concurrency peaks, the system should have elastic scaling capabilities; when hardware failures occur, requests should be able to migrate smoothly. Therefore, inference services need to integrate well with container platforms, monitoring and alerting systems, logging services, and other infrastructure.

On the other hand, speed improvements should not come at the expense of output quality. Enterprises should establish a comprehensive evaluation system to compare output consistency between high-speed configurations and baseline configurations. For lossy optimizations such as quantization and cache reuse, their impact must be validated under target business scenarios.

From an industry trend perspective, model development efficiency itself is also improving. For example, incorporating automated code generation, experiment design, and system optimization methods can accelerate the iteration pace of inference systems, enabling teams to find optimal configurations for different hardware and scenarios more quickly. This path of “using AI to optimize AI systems” is becoming an important enabler for the continuous evolution of high-speed inference capabilities.

6. Summary

In the second half of LLM competition, speed is becoming a key variable. High-throughput, low-latency inference directly affects user retention, serving costs, and business expansion headroom. Achieving high-speed inference cannot rely solely on model improvements—it requires coordinated optimization across inference engines, scheduling strategies, cache mechanisms, compression techniques, parallelism approaches, and deployment configurations.

For developers, mastering systematic tuning methodologies has more long-term value than chasing a single performance number. For enterprises, building an observable, scalable, and cost-efficient inference serving system is what truly translates model capabilities into business advantage. As co-design of inference systems and model architectures continues to deepen, the efficiency frontier of large model services will continue to be pushed higher.