Long-Context Inference Cost Reduction: Prefill Acceleration and KV Cache Compression in Practice

    As agents take on ever-longer contexts, long-context inference has become a major cost driver. This article breaks down where those costs originate—Prefill computation and KV cache memory—and organizes mainstream optimizations into four orthogonal compression strategies: low-bit KV quantization, sparse attention, cross-layer KV sharing, and asymmetric Prefill. It shows how these techniques compound multiplicatively rather than additively, maps them to common deployment scenarios, and covers the evaluation, serving, and cost-control practices needed to make long-context agents economically viable at scale.

    1. Introduction: Long Context Is Becoming the Primary Source of Inference Cost

    As agent capabilities grow, models must process increasingly long contexts. A single complex software engineering task may involve hundreds of tool calls, hundreds of thousands of tokens of terminal output, and the complete context of a code repository. Deep research, GUI agents, and enterprise workflows all need to persist and repeatedly reuse lengthy interaction histories. Context has shifted from a “one-time input” to a “long-lived computational state.”

    Traditional decoder-only Transformers face two main costs in these scenarios.

    1. Prefill computation cost. Suppose a model has L layers and a prompt length of N. Every layer must process the full prompt; this work is roughly O(NL) and scales with the model’s parameter count. The longer the context, the slower the first token is produced (TTFT).

    2. KV cache cost. Traditional models typically store K and V separately for each layer. As the context grows, this cache continuously occupies GPU HBM, reducing the number of requests that can run concurrently. In multi-turn conversations and distributed deployments, it also incurs transfer overhead across host memory, SSD, and inter-node networks.

    Therefore, the goal of long-context inference optimization is not just “cheaper per token,” but simultaneously compressing four quantities:

    • The storage size of each KV entry;
    • The number of historical positions that actually participate in attention;
    • The KV that is redundantly stored across layers;
    • The number of network layers that must fully execute during Prefill.

    The sections below use these four threads as a framework, analyze the benefits and costs of mainstream optimization approaches one by one, and offer engineering recommendations.

    2. Quantifying First: Where Do Long-Context Inference Costs Come From

    2.1 Prefill Stage: Computation Grows Linearly with Prompt Length

    Ignoring the attention term, the number of floating-point operations needed for a forward pass over N tokens can be approximated as:

    1. Prefill_FLOPs ≈ 2 × N × P

    where P is the average number of parameters activated per token. In dense models, P equals the total parameter count; in MoE (Mixture of Experts) models, P is typically far smaller than the total parameter count.

    Actual latency also depends on memory bandwidth. Taking P = 20B and N = 100K as an example:

    Metric Estimate
    Forward pass computation ~4 PFLOPs
    Data to read weights (BF16) ~40 GB
    Theoretical floor with 3 TB/s HBM bandwidth ~13 ms
    Compute time with 1 PFLOPS FP8 per card ~4 s

    This reveals three points:

    • Compute grows faster than memory bandwidth, but weight reading always has a lower bound on time;
    • The main Prefill bottleneck is often compute;
    • Decode, which generates token by token, is mainly bandwidth-bound—the two have different optimization priorities.

    Actual performance also depends on hardware specs, parallelism strategies, and framework implementation. The values in the table are theoretical estimates meant only to illustrate orders of magnitude.

    2.2 KV Cache: Memory Grows Linearly with Context and Concurrency

    The number of bytes needed to cache K and V per token, per layer is:

    1. bytes_per_token = 2 × L × H_kv × d_head × b / 8

    where:

    • 2 accounts for one copy each of K and V;
    • H_kv is the number of KV heads. With GQA (Grouped-Query Attention) or MQA (Multi-Query Attention), H_kv is smaller than the number of query heads;
    • d_head is the dimension per head;
    • b is the number of bits per element.

    The table below shows sample calculations for typical configurations:

    Configuration Per layer per token Per token (61 layers) 100K context, single request 8K context × 64 concurrency
    8 KV heads, head dim 128, BF16 4 KB ~244 KB ~23.8 GiB ~122 GiB
    Same, FP8 2 KB ~122 KB ~11.9 GiB ~61 GiB
    4 KV heads, head dim 128, BF16 2 KB ~122 KB ~11.9 GiB ~61 GiB

    An 80 GB accelerator card, under BF16 with 8 KV heads, can hold roughly 327 concurrent 8K sessions from KV cache alone—before accounting for weights, activations, and framework overhead. Stretch the context by another order of magnitude and the gap widens further.

    For a single request, the KV cache cost extends beyond memory:

    • With paged KV management, attention must read the cache page by page, and read cost grows with context and concurrency;
    • With prefix caching or context offloading, the cache must transfer across instances and nodes, adding bandwidth and serialization overhead;
    • Compared to the text itself, the KV cache balloons faster and more easily becomes the storage bottleneck in multi-turn conversations.

    3. Optimization Landscape: Four Compression Threads

    Combining the formulas from 2.1 and 2.2, feasible optimization directions reduce to four:

    1. Compress individual KV entries: lower the storage bit width, e.g., from 16 bits to 8 or 4 bits;
    2. Reduce historical positions in attention: use sparse attention to compute only selected portions;
    3. Eliminate cross-layer redundancy: adjacent layers share the same KV, or reuse selection results across layers;
    4. Reduce layers executed in Prefill: let most prompt tokens pass through only the front half of the network.

    These four approaches act on different factors in the formulas and are largely orthogonal, so they can be stacked. Common engineering trade-offs:

    • Low-bit quantization: low implementation cost and the most direct benefit, but long-context accuracy must be validated;
    • Sparse attention: significant reduction in memory and decode read cost, but adds selection-logic overhead;
    • Cross-layer sharing: high compression ratio, but changes the model’s expressive power and must be designed in the training phase;
    • Asymmetric structure: directly cuts Prefill compute, but is an architecture-level change requiring supporting attention design and training strategies.

    The table below compares these four approaches qualitatively from an engineering perspective:

    Approach Main benefit Main cost Typical application stage
    Low-bit KV quantization Lower memory and read cost; negligible effect on Prefill compute Quantization error may accumulate with context KV storage and read/write path
    Sparse attention Lower decode read cost for long contexts Selection logic has its own overhead; may miss key information Long-context decode
    Cross-layer KV sharing Storage and read cost drop roughly proportionally Limited expressive power; requires training support Model architecture design
    Asymmetric Prefill Directly cuts Prefill compute Large architecture change; high training and evaluation cost Prompt encoding stage

    4. Thread One: Making Prefill Compute Less

    4.1 Approach: Process Inputs and Outputs Asymmetrically

    In conventional decoder-only models, input and output share the same full network, and every token in the prompt is computed layer by layer.

    The asymmetric design (which can be called a Causal Encoder-Decoder structure) works as follows: most prompt tokens pass through only the front half of the network during Prefill, acting as a causal encoder. The intermediate representations they produce are cached for reuse during generation. Meanwhile, key tokens such as task instructions and the tokens being generated still pass through the full back half of the network.

    Sample code:

    1. def prefill_model(
    2. hidden_states,
    3. final_token_mask,
    4. encoder_layers, # front half: processes all inputs
    5. decoder_layers, # back half: processes only key positions
    6. ):
    7. # Stage 1: all tokens pass through the front half
    8. for layer in encoder_layers:
    9. hidden_states = layer(hidden_states)
    10. # Stage 2: only key tokens continue through the full network
    11. keep = final_token_mask # shape: [seq_len], boolean mask
    12. decoded = decoder_forward(decoder_layers, hidden_states[keep])
    13. # Illustrative logic for writing back to the KV cache
    14. write_kv_cache(cache, hidden_states, keep)
    15. return decoded

    Letting H and D denote the number of layers in the front and back halves, the Prefill compute of such a structure can be estimated as:

    1. Conventional: N × H × p_layer + N × D × p_layer
    2. Asymmetric : N × H × p_layer + K × D × p_layer
    3. Savings ≈ (N − K) / (N + N×H/D)

    where p_layer is the compute per layer, and K is the number of tokens that must pass through the full network. When the prompt is long, the back half is not too large a share, and few tokens need full processing, the compute can approach being halved.

    4.2 Supporting Design: Which Attention Fits This Structure

    Prompt tokens pass through only the front half, yet generation must read their representations. So the back half’s attention must directly access the front half’s cache. Two common designs:

    • Shared global state: the back half shares the same global KV as the front half, which outputs the final cache at its last layer without extra memory;
    • Separate caches: the back half maintains its own KV and passes summary information from the front half between layers.

    4.3 Benefits and Costs

    Benefits:

    • Prefill compute drops significantly. The longer the context, the more obvious the savings;
    • Long system prompts and full repositories consist mostly of low-information-density content, so this design cuts compute effectively while having relatively limited impact on retaining task-critical information.

    Costs:

    • It is an architecture-level change requiring supporting training data and processes;
    • If the task requires every token to pass through the full network (e.g., character-by-character verification), the benefits diminish;
    • In evaluation, don’t rely only on short-prompt leaderboard performance; focus on long-context, multi-hop reasoning, and tool-call success rates.

    5. Thread Two: Compressing the KV Cache

    5.1 Low-Bit Quantization: From 16 Bits to 8 and Then 4

    The storage bit width of the KV cache is the most direct factor in compression cost: from BF16 to FP8 halves storage; further to FP4, in theory compressing to a quarter of the original.

    Typical steps:

    1. Compute scale factors per group of 128 elements, or per head;
    2. Map values to a finite integer range;
    3. Restore with the scale factors.
    1. import torch
    2. def quantize_kv(x, bits=4, group_size=128):
    3. """Grouped symmetric quantization; returns quantized values and per-group scale"""
    4. qmax = 2 ** (bits - 1) - 1
    5. x = x.float().reshape(*x.shape[:-1], -1, group_size)
    6. scale = x.abs().amax(dim=-1, keepdim=True).clamp_min(1e-6) / qmax
    7. q = torch.clamp(torch.round(x / scale), -qmax - 1, qmax).to(torch.int8)
    8. bits_per_element = bits + 16 / group_size # extra 16-bit scale per group
    9. bytes_per_token = x.numel() / group_size * bits_per_element / 8
    10. return q, scale, bytes_per_token

    Three points to watch during quantization:

    • Only storage bit width is reduced: attention typically dequantizes back to higher precision before computing, so compute precision need not drop;
    • Error accumulates: in the decode phase, newly generated K and V are continuously appended to the cache, and quantization error accumulates. Whether the error is biased in the position distribution matters more than its average magnitude;
    • Validation: use perplexity and other general metrics for quick regression, then validate with business evaluation sets such as long-context QA and multi-hop reasoning.

    Researchers are also exploring further compressing some tokens’ cache to low bits or even 1-bit extreme narrow quantization, as a complement to sparsification.

    5.2 Sparse Attention: Compute Only Selected Historical Positions

    Sparse attention replaces full computation with “a local window plus selected historical fragments.” In decode, the slope of KV read volume as context grows slows markedly; in Prefill, it can also combine with scheduling mechanisms like chunked prefill to reduce the compute pressure of long inputs.

    The key to sparsification is selecting “positions worth attending to” at low cost. Common approaches:

    • A fixed sliding window covering recent local context;
    • Retaining a few global tokens, such as system instructions;
    • Using a lightweight index module to score historical positions and select the Top-K highest-scoring ones.

    Sparse attention suits scenarios that frequently browse long contexts, such as code repository retrieval and long-document QA. But it relies on the empirical assumption that “attention is sparse.” Tasks requiring line-by-line verification of long-text details should separately evaluate the risk of missed recall.

    5.3 Cross-Layer Sharing: Compute KV and Selection Once

    In conventional structures, every layer generates independent K and V, and adjacent layers’ caches are often highly similar. The cross-layer sharing idea: only some layers generate and maintain K and V, while the rest reuse them. If the selection indices for “which positions” are also shared, Top-K scoring can be computed once instead of repeated at every layer.

    1. def forward_with_shared_cache(h, layer_idx, cache, share_groups):
    2. """
    3. share_groups: {group id: layer index of that group's first layer}
    4. Generate KV only at the group's first layer; reuse it in other layers of the group
    5. """
    6. group = layer_idx // 4 # illustrative: share one group per 4 layers
    7. owner = share_groups[group] # layer responsible for generating KV in this group
    8. if layer_idx == owner:
    9. k, v = compute_kv(h)
    10. cache.store(group, k, v)
    11. else:
    12. k, v = cache.load(group)
    13. # Share the Top-K selection within the group to avoid repeated scoring per layer
    14. topk_idx = cache.get_topk(group) if layer_idx != owner else select_topk(q, k)
    15. return attention_with_indices(h, k, v, topk_idx)

    The compression from cross-layer sharing is roughly proportional to “how many layers share one group.” The cost is reduced expressive power: different layers were originally good at capturing different semantic relationships, and after forced sharing, the model must learn to work within the shared structure during training.

    5.4 Serving Layer: Paged Management and Prefix Reuse

    In-model compression solves “how much to store”; the serving layer decides “when to store and for how long.” In inference serving, several general practices apply:

    • Paged KV management: split the cache into fixed-size blocks, allocated and released on demand to reduce memory fragmentation;
    • Prefix cache reuse: in agent multi-turn conversations, system prompts and historical messages are heavily repeated, and cache hits can skip that Prefill portion entirely;
    • Cache tiering: after a session pauses, migrate inactive sessions’ cache from HBM to host memory or local SSD, and swap it back on resume;
    • Cache-aware scheduling: route requests with the same prefix to the same instance to improve hit rates.

    Suppose in a 100K-token request, 80K tokens are a cached historical prefix; Prefill compute can drop to about 20% of the original. As context grows further, the newly added cost falls mainly on the uncached incremental portion. This explains why some teams optimize only “average input length” yet fail to cut costs: what really determines the bill is often the cache hit rate, not the total context length.

    6. Thread Three: Decoupling Knowledge Capacity from Context

    Not all tokens in a context need to participate in computation. Information such as system prompts, product docs, and historical conversations can be split into two types:

    • Resident knowledge: low change frequency, used repeatedly;
    • Task state: changes dynamically with interaction.

    Decoupling knowledge memory from the main computation is analogous to how modern processors distinguish registers from main memory:

    • The backbone network acts as the compute unit, keeping only necessary computation state;
    • Knowledge is stored in an independent memory structure, tapped on demand at a smaller activation cost.

    Corresponding engineering practices include:

    • Using retrieval to “invite” knowledge into the context, injecting only the fragments truly needed for the current turn;
    • After a session ends, writing long-lived conclusions, preferences, and decisions to external memory rather than piling them into the context;
    • Using summaries or compressed representations to replace the raw long text in subsequent reasoning.

    Two metrics commonly measure such designs:

    • Activation overhead: how many extra parameters or cache the model activates per unit of knowledge capacity added;
    • Effective knowledge utilization rate: the proportion of retrieved knowledge that the model actually uses.

    Note that decoupling does not mean long context is unnecessary. Tasks that truly require all information reasoned over together (e.g., cross-file refactoring) still cannot replace long context. The point of decoupling is to separate “resident” from “temporary” so the resident part is no longer billed repeatedly by context length.

    7. Comprehensive Evaluation: How to Judge Whether a Solution Pays Off

    7.1 Benefits Multiply Rather Than Add

    The four threads act on different factors, so their combined effect is closer to multiplication than simple addition. Using the illustrative 4KB/layer/token config (61 layers, ~244KB per token), starting from a BF16 baseline:

    1. Share one KV group per 4 layers (cross-layer sharing) and reuse selection results: ~61KB;
    2. Quantize the cache to 4 bits (4 bits per element, plus a 16-bit scale per 128-element group, effectively ~4.125 bits): ~17KB;
    3. Retain only half of all layers in the cache: ~8.5KB.

    Overall compression is about 28×, consistent with the expected “1/4 × 1/4 × 1/2.” Conversely, if you only compress inside the model without prefix reuse, the cache hit rate stays unchanged and first-token latency improvements will be limited.

    7.2 Scenario Benefit Matrix

    The table below gives qualitative benefits for four typical scenarios. These are not precise numbers; actual benefits need to be validated through evaluation.

    Scenario Main bottleneck Optimization combination with highest benefit
    Single-turn long-document QA Prefill compute, long-context accuracy Asymmetric Prefill + low-bit quantization + sparse attention
    Multi-turn agent (software engineering, GUI operation) Memory capacity, cache hit rate, decode read cost Cross-layer sharing + quantization + prefix caching + tiered storage
    High-concurrency knowledge QA Memory and throughput, request queue latency Cross-layer sharing + sparse attention + knowledge retrieval decoupling
    Ultra-long context (million-token scale) End-to-end cost, including memory and inter-node transfer Combine all optimizations, with cache sharding and quantized transfer

    8. Engineering Deployment: Evaluation, Serving, and Monitoring

    8.1 Three-Layer Evaluation Checklist

    Architecture-level optimizations can’t be validated by general leaderboards alone. Cover at least three layers:

    1. Basic regression: use metrics like perplexity to quickly judge whether the compressed model degrades overall;
    2. Capability evaluation: cover complex reasoning, code generation and repair, long-context QA, multi-hop retrieval, and more;
    3. Online validation: use business metrics such as tool-call success rate, task completion rate, and human adoption rate.

    For sparsification and quantization solutions, construct targeted comparison cases:

    • Place key information at different positions in a long context and observe whether a “lost in the middle” phenomenon appears;
    • Build cases requiring repeated verification of numbers and identifiers to test restoration accuracy under low-bit quantization.

    8.2 Serving Layer Design Points

    • Cache lifecycle management: set reasonable expiration times and eviction policies to avoid bandwidth waste from frequent swapping;
    • Cache-aware routing: let requests with the same prefix land on the same instance as much as possible;
    • Observability: monitor first-token latency, per-token output latency, prefix cache hit rate, and memory watermark together.
    1. class InferenceMetrics:
    2. """Key metrics for long-context inference serving (illustrative)"""
    3. def record(self, request):
    4. self.ttft_ms.append(request.first_token_latency) # first-token latency
    5. self.tpot_ms.append(request.per_output_token_latency) # per-token output latency
    6. self.cache_hit_ratio = request.cached_tokens / max(request.prompt_tokens, 1)
    7. self.kv_bytes = request.kv_cache_bytes # cache usage for this request
    8. if self.cache_hit_ratio < 0.5:
    9. # A low hit rate usually means prefix routing or eviction policy needs tuning
    10. self.alert_low_cache_hit(request)

    If serving across multiple instances or regions, capabilities like cache sharding, prefix routing, and request scheduling are usually implemented through a cloud platform’s general infrastructure. For example, on Baidu AI Cloud, you can use container services to host inference instances, object storage to hold model parameters and long-session snapshots, message queues to smooth traffic peaks, and logging and monitoring/alerting systems to track the metrics above. This avoids re-implementing these underlying capabilities in business code.

    8.3 Cost Control Points

    • Choose tiers by task: use low-bit or low-activation configs for simple tasks, high-precision configs for complex tasks;
    • Context governance: periodically clean up ineffective historical turns and move long-term knowledge to external memory;
    • End-to-end accounting: convert memory, bandwidth, storage, and network transfer into unit cost together, rather than looking only at the price per million tokens;
    • Continuous regression: after each model or inference framework upgrade, rerun accuracy and cost baselines.

    9. Conclusion

    Optimizing long-context inference cost is essentially systematic governance of four factors: the size of individual KV entries, the number of historical positions participating in attention, the KV redundantly stored across layers, and the number of layers Prefill must fully execute.

    For developers, the key takeaways are:

    • Build a cost model first: use the 2.1 and 2.2 formulas to estimate your own business scale, then decide which direction to invest in;
    • Prioritize low-risk items: low-bit quantization and serving-layer cache management are relatively low-cost to implement and should be landed first;
    • Approach structural changes cautiously: sparse attention, cross-layer sharing, and asymmetric Prefill are structural changes that need a supporting evaluation system;
    • Combination beats single points: only by stacking multiple optimizations can you achieve breakthroughs in memory, compute, and knowledge capacity simultaneously.

    When the storage cost of each token, the attention cost of each position, and the Prefill compute cost of each round can all be measured and optimized individually, long-context agents truly gain the economics for large-scale deployment.