High-Speed Inference for Flagship Large Language Models: Key Technologies and Engineering Practices

    This article examines how flagship large language models can achieve high-speed inference through system-level engineering optimization. It explains key performance metrics such as TTFT, per-request token generation speed, and system throughput; reviews major acceleration techniques including KV cache paging, speculative decoding, continuous batching, quantization, prefix caching, chunked prefill, and parallel attention optimizations; and discusses how lower latency enables new application patterns and practical deployment choices.

    In recent years, the center of gravity in LLM competition has quietly shifted. In the past, model quality was judged mainly by correctness and writing quality—what people commonly refer to as “capability.” As model capabilities approach an acceptable baseline, user-perceived differences increasingly come down to “how fast it is.”

    A common assumption is that “fast” means “small”: lightweight models are easier to speed up, but they often sacrifice capability. Engineering practice is breaking this linear tradeoff. With a set of system-level inference optimizations, flagship-level LLMs can also deliver very high output speeds. From an engineering perspective, this article breaks down the key technologies behind high-performance inference and discusses how crossing a speed threshold changes application patterns.

    1. First, Define What Matters: Key Inference Performance Metrics

    To talk about “speeding up,” we first need to define what is being measured. For LLM inference, the industry generally focuses on two dimensions.

    The first dimension is time to first token, or TTFT. This is the time from when a request is sent to when the model emits its first token. TTFT is mainly affected by the prefill stage, where the model processes the complete input context in one pass. For interactive scenarios, TTFT determines how long users wait after clicking “Send.” Ideally, it should be controlled at the sub-second level.

    The second dimension covers generation throughput and per-request output speed. These are distinct concepts:

    • Per-request output rate: the number of tokens a single request can generate per second. This directly determines how smooth a single answer feels and is usually measured in tok/s.
    • System throughput: the total number of tokens produced by the service across all concurrent requests in a given period. This reflects how efficiently hardware resources are utilized.

    A human blink takes about 0.3 seconds. In the same time, a high-speed inference service can generate more than a hundred tokens. When per-request output speed reaches several hundred tokens per second, generation becomes fast enough to feel almost like real-time output. This is especially important for the application patterns discussed later.

    2. The Core of Speed: The Technology Stack for Inference Acceleration

    High-speed inference is not the result of any single optimization. It comes from coordinated hardware and software improvements. The following sections outline the main technologies, ordered by their importance to performance.

    1. KV Cache and PagedAttention

    During autoregressive generation, every new token requires the Key and Value tensors of all previous tokens. Recomputing these values each time would cause overhead to grow quadratically with sequence length. Engineering practice therefore caches these K/V tensors in the KV Cache.

    The challenge is that sequence lengths vary greatly across requests. Reserving a fixed-size, contiguous memory block for each request leads to serious memory fragmentation and waste. PagedAttention borrows the idea of virtual memory paging from operating systems. It divides the KV Cache into fixed-size blocks and allocates them on demand in non-contiguous memory. This significantly improves memory utilization, allows more concurrent requests to be served, and indirectly increases system throughput.

    2. Speculative Decoding

    Large-model token-by-token generation is inherently sequential, which is a major source of latency. The core idea of speculative decoding is simple: let a small model guess, and let the large model verify.

    The process works as follows. A lightweight draft model quickly generates several candidate tokens in sequence. The large model then verifies those candidates in parallel during a single forward pass. Accepted tokens are adopted directly, and if an error occurs, generation rolls back to the first point of divergence. Because one forward pass of the large model can verify multiple tokens, and verification is much cheaper than generating tokens one by one, overall output speed improves significantly. In theory, this does not change the final output distribution.

    The following pseudocode illustrates the idea:

    1. def speculative_decode(target_model, draft_model, prompt):
    2. state = prefill(target_model, prompt) # Prefill the target model once
    3. draft_state = prefill(draft_model, prompt)
    4. while not done:
    5. # The draft model proposes gamma tokens in sequence
    6. draft_tokens = draft_model.generate(draft_state, gamma)
    7. # The target model verifies the candidates in one parallel pass
    8. logits = target_model.verify(state, draft_tokens)
    9. accepted = accept_prefix(logits, draft_tokens)
    10. yield from accepted # Emit multiple verified tokens at once
    11. state = advance(state, len(accepted))

    3. Continuous Batching

    Traditional batching requires requests in a batch to proceed in lockstep. Resources are not released until the entire batch finishes generating, so completed requests can occupy memory while waiting unnecessarily. Continuous batching refines scheduling to each iteration. As soon as one request finishes, a newly arrived request can fill the freed slot. This is similar to an elevator not waiting for everyone to get on or off before closing its doors. By keeping GPU compute units busy, continuous batching pushes system throughput close to the hardware limit.

    4. Quantization

    When model weights are stored in FP16/BF16, memory usage is high and bandwidth pressure is significant. Quantization compresses weights and activations into lower-bit representations, such as FP8, INT8, or even INT4. Lower-bit formats not only reduce memory usage but also reduce memory-access bandwidth demand. Since the decoding stage is often memory-bandwidth bound, quantization can directly lower per-token generation latency. Modern quantization schemes—such as quantizing only weights while preserving high-precision activations, or using group scaling factors—can substantially reduce memory requirements while keeping accuracy loss within an acceptable range. This makes it possible to be “fast without becoming blurry.”

    5. Prefix Caching and Prefix Reuse

    Many requests share the same prefix, such as a unified system prompt, few-shot examples, or a long document. Prefix caching stores the already-computed KV Cache for that shared prefix. When a new request matches the same prefix, it does not need to repeat prefill and can reuse the cached values directly. For scenarios such as coding assistants with fixed context templates, prefix caching can significantly reduce TTFT.

    6. Chunked Prefill

    The prefill stage is compute-intensive. If an extremely long input is processed all at once, it can occupy resources for a long time and block other requests from decoding, increasing their TTFT. Chunked prefill splits long prefill work into smaller chunks and interleaves them with decoding requests. This creates a smoother latency balance between long inputs and interactive short requests.

    7. Parallelism and Attention Operator Optimization

    For extremely large models, tensor parallelism and pipeline parallelism are needed to split single-layer computation across multiple devices. Efficient attention operators, such as attention implementations that reduce memory reads and writes through chunking, further lower the time and memory overhead of attention computation. Together, these low-level optimizations and upper-level scheduling form the physical foundation for flagship models to be both large and fast.

    3. After Crossing the Speed Threshold, Application Patterns Are Redefined

    Technical metrics ultimately matter because of their impact on product experience. When output speed jumps from tens of tokens per second to hundreds of tokens per second, many product forms that were previously infeasible due to latency suddenly become practical.

    1. Real-Time Streaming Code Generation

    In coding assistance scenarios, slow output means the model may still be halfway through writing while the user waits, creating a disjointed interaction. Once output enters the high-speed range, a complete HTML, CSS, and interaction logic structure can be streamed within seconds. Users can almost synchronously watch code grow in the editor and preview the result immediately. The speed difference directly magnifies into an experience difference: with the same capability, the faster side may deliver in one minute, while the slower side may take several times longer.

    2. Interactive Real-Time Creation

    Consider a scenario where a user enters a natural-language description and the model generates shader code in real time, while a rendering window updates the image synchronously. Every new instruction receives an almost immediate response, forming a real-time creative loop where “what you say is what happens.” Converting a hand-drawn sketch into a complete front-end page, or using natural language to adjust the colors and elements of a three-dimensional scene in real time, both belong to this “modify while speaking, view while modifying” pattern. These forms depend on sufficiently low output latency.

    3. Multi-Agent Collaboration and Aggregate Throughput

    When dozens of agents work in parallel on the same task and their results must be aggregated, system throughput becomes as important as per-request latency. With continuous batching and parallelism optimizations, a service can maintain acceptable per-request experience while pushing aggregate throughput far beyond single-request speeds. This is especially valuable for batch code review, parallel generation of multiple solutions, and large-scale data synthesis.

    4. Engineering Implementation: Bringing High-Speed Inference to Your Own Service

    If you want to build such a high-performance inference service, you can start from the following directions and amplify the benefits by combining them with general cloud capabilities.

    • Selection and deployment: Prefer inference frameworks or inference services that support features such as continuous batching and paged caching, so that optimizations are available out of the box rather than requiring you to build a scheduler from scratch.
    • Elasticity and concurrency: Use container services and auto-scaling capabilities on cloud platforms to adjust the number of inference instances dynamically based on request peaks, balancing cost and throughput.
    • Context reuse: For scenarios with fixed templates, enable prefix caching to amortize the prefill cost of common prompts.
    • Precision and speed tradeoff: Based on the output-quality requirements of your business, choose an appropriate quantization level to find a balance among speed, memory usage, and accuracy.
    • Observability: Connect logging services and monitoring alerts to continuously track key metrics such as TTFT, output speed, concurrency, and memory usage, and use data to drive tuning.

    On general cloud capabilities such as model inference, containers, and monitoring offered by platforms such as Baidu AI Cloud, developers can focus more on business logic and experience design, leaving heavy scheduling, scaling, and operations work to managed platforms. This makes it easier to approach the goal of “flagship capability plus extreme speed.”

    5. Conclusion

    LLM competition is moving from a single capability dimension to a two-dimensional contest of “capability × speed.” Mature technologies such as KV cache paging, speculative decoding, continuous batching, quantization, and prefix reuse jointly support high-performance inference for flagship models. More importantly, once output speed crosses a critical threshold, new product forms—real-time streaming generation, interactive creation, and multi-agent collaboration—open up. For developers, the key to turning the slogan of “having both” into reality is to understand the tradeoff logic behind these technologies and apply general cloud capabilities appropriately in engineering practice.