Reducing LLM Inference Costs: Quantization, KV Cache, and Prefill–Decode Disaggregation

    A practical guide to evaluating and reducing large language model inference costs through consistent benchmarking, quantization, fused attention operators, paged KV cache management, prefill–decode disaggregation, request routing, prefix caching, and asynchronous batching. The article also covers deployment trade-offs, enterprise platform selection, and the operational risks that should be addressed before production rollout.

    1. Establish Consistent Measurement Standards

    Before comparing deployment options, teams should agree on a consistent set of metrics. Unit cost can be calculated as follows:

    Unit cost (CNY per million tokens) = Hourly price per GPU ÷ Output tokens per GPU hour × 1,000,000

    For example, if one GPU costs CNY 10 per hour and produces 2,000 output tokens per second, its hourly output is 7,200,000 tokens. The resulting unit cost is approximately CNY 1.39 per million tokens. This is only an example; actual pricing and benchmark results should be substituted.

    Throughput alone can hide latency problems. Teams should also record time to first token (TTFT) and time per output token (TPOT). Different solutions should be compared using the same hardware and the same request distribution. Benchmark reports should include the model version, input and output length distributions, and concurrency level. Otherwise, the results cannot be reproduced reliably.

    2. Quantization: Trading Precision for Cost

    Quantization reduces the numerical precision of weights and activations to lower memory consumption and improve computational throughput. Common approaches include INT8, FP8, and weight-only quantization.

    Weight quantization generally reduces memory usage. Its impact on throughput depends on the hardware and inference engine. Methods that also quantize activations, such as W8A8, can provide more significant computational gains but are typically more sensitive to accuracy degradation.

    Evaluating Quantization Accuracy

    Quantization quality should be evaluated with real business data rather than general-purpose leaderboards alone. Create a fixed regression set covering at least three task categories:

    • Complex instruction following
    • Code generation
    • Mathematical reasoning

    Compare the original and quantized models with the same decoding parameters, and measure task success rates. Acceptance thresholds should be defined according to business risk. For example, a team may require the degradation on critical tasks to remain below a predefined limit.

    Fine-grained quantization, such as per-channel quantization, is often more stable than applying one scale uniformly to an entire layer. However, it increases metadata and implementation complexity. The final choice should be based on accuracy regression results, not theoretical compression ratios alone.

    3. Fused Attention Operators

    Standard attention computation includes the QKᵀ matrix multiplication, scaling, Softmax, and multiplication with V. If every step writes intermediate results back to GPU memory, memory traffic can become substantial.

    Operator fusion combines these steps into a single GPU kernel. The intermediate attention matrix does not need to be fully written back to memory. FlashAttention is a representative implementation of this approach.

    The benefits of fused operators depend on the hardware architecture, data type, and sequence length. They also require support from the inference framework or underlying libraries. With FP16 or BF16, the numerical results before and after fusion may differ slightly. These differences should be verified before production deployment to ensure that they do not affect business metrics.

    4. KV Cache Memory and Paged Management

    During autoregressive generation, the Key and Value states for every generated token must be stored in the KV cache. The approximate memory required per token can be calculated as follows:

    Bytes per token = 2 × Number of layers × Number of KV heads × Head dimension × Bytes per element

    For a model with 32 layers, 8 KV heads, a head dimension of 128, and FP16 values, where each element occupies 2 bytes:

    2 × 32 × 8 × 128 × 2 = 131,072 bytes

    This is approximately 128 KiB per token. A 4,096-token sequence therefore requires about 512 MiB.

    Memory planning should be based on the peak sequence length rather than the average length. Otherwise, the service may run out of memory during traffic spikes.

    Paged management, similar to PagedAttention, divides the KV cache into fixed-size blocks and allocates them on demand. This reduces memory fragmentation. The following example omits block release and block-table mapping:

    1. class PagedKVCache:
    2. def __init__(self, num_blocks, block_size):
    3. self.block_size = block_size # Example value; adjust for the model and available memory
    4. self.block_pool = [Block(block_size) for _ in range(num_blocks)]
    5. self.free_blocks = self.block_pool.copy()
    6. def allocate(self, seq_len):
    7. num_required = (seq_len + self.block_size - 1) // self.block_size
    8. if num_required > len(self.free_blocks):
    9. raise OutOfMemoryError(
    10. f"{num_required} blocks required, but only "
    11. f"{len(self.free_blocks)} remain"
    12. )
    13. return [self.free_blocks.pop() for _ in range(num_required)]

    The example checks the number of free blocks before allocating them all at once. This prevents a partial allocation failure from leaving blocks that cannot be reclaimed. A production implementation must also handle block reclamation when requests finish and use locking or another synchronization mechanism for concurrent access.

    5. Prefill–Decode Disaggregation and Cluster Scheduling

    The prefill stage processes the complete input and is primarily compute-bound, making it suitable for high-compute nodes and batch execution. The decode stage generates tokens one at a time and is more memory-bandwidth-bound.

    Separating prefill and decode onto different nodes allows the two resource profiles to scale independently.

    However, disaggregation introduces additional costs. The KV cache produced by the prefill nodes must be transferred to the decode nodes, so network bandwidth and transfer time become part of the latency budget. This architecture is more suitable for workloads with long inputs and high concurrency. When inputs are short, transfer overhead may exceed the performance benefits.

    Evaluation should include KV transfer time in TTFT and compare the result with a combined deployment using equivalent hardware.

    6. Three Business-Level Practices

    6.1 Request Routing

    Use different model instances for simple tasks, such as summarization and sentiment classification, and complex tasks, such as advanced code generation and mathematical modeling.

    A lightweight classifier can estimate task complexity. The system can then implement an escalation mechanism: when the lower-cost model produces an output with insufficient confidence, the request is forwarded to a more capable model.

    Before deploying routing rules, evaluate the misrouting rate on manually labeled samples. A team may initially use a misrouting rate below 5% as a starting target and adjust it based on business outcomes.

    6.2 Prompt Prefix Caching

    Many requests share the same system prompt and background knowledge. An inference framework can cache the KV cache for this shared prefix.

    A cache hit requires the prefix to match token by token. Fixed content should therefore appear at the beginning of the prompt, while dynamic content, such as user input, should be placed later.

    vLLM supports Automatic Prefix Caching. During load testing, teams can measure how the cache hit rate affects TTFT. In multi-tenant environments, caches should be isolated by tenant to prevent sensitive prefix content from being shared across customers.

    6.3 Asynchronous Batch Processing

    Non-real-time workloads, such as overnight data cleaning and bulk document summarization, can combine requests into larger batches to improve GPU utilization.

    Batch processing should define both a maximum waiting time and a maximum batch size. Start with a small waiting interval and tune it while ensuring that P99 latency remains within the service-level target.

    Batching increases queueing latency and is therefore unsuitable for interactive services with strict real-time requirements.

    7. Enterprise Integration and Platform Selection

    Organizations that do not want to build and operate their own inference clusters can access model services through platforms such as Baidu AI Cloud’s Qianfan Large Model Platform.

    The platform provides capabilities related to compute resources, monitoring and alerting, logging, and container infrastructure. During evaluation, confirm the billing model—whether usage is charged by token or by allocated resources—as well as concurrency limits, service-level agreements, and data storage regions.

    Specific terms should be verified in the official documentation and applicable contract.

    8. Risks and Boundaries

    • Quantization may cause noticeable degradation on long-chain reasoning or multilingual tasks. Business-specific regression tests are required.
    • Memory planning must cover peak sequence length and peak concurrency. Otherwise, the service may run out of memory during traffic spikes.
    • Prefix caching depends on an exactly matching prefix. Changing even one character in the system prompt can invalidate the cache.
    • Shared caches in multi-tenant environments may create data-isolation risks and must be addressed at the architecture level.
    • Asynchronous batching increases queueing time and should not be used for services with strict real-time requirements.

    Conclusion

    First, use consistent benchmarks before deployment. Record unit cost, TTFT, TPOT, and business accuracy metrics before deciding whether to adopt quantization or fused operators.

    Second, classify requests by task complexity and include prefix-cache hit rate and routing errors in routine monitoring.

    Third, run regression tests for every change to quantization, operators, or memory configuration, and keep the previous version available for rapid rollback.