Hybrid Attention LLMs and KV Cache Prefix Reuse: Why a Hit Doesn't Mean Correct State
Hybrid attention architectures now power million-token context windows and cross-request reuse in LLM inference, but the long-context metrics and cache hit rates in official reports measure how much compute is saved, not whether the output is correct. This article explains why a KV cache prefix hit does not guarantee valid cache state: a single token now spans multiple heterogeneous cache objects. It breaks down the five cache object types—compressed KV, uncompressed tail, SWA window, indexer state, and on-disk prefix—along with their lifecycles and recovery conditions, then presents a four-layer hit verification process and four consistency invariants for the cache allocator. Engineering guidance covers three-stage hit semantics, regression cases, and tiered caching. The core takeaway: under hybrid attention, prefix cache hit rate is a performance metric, not a correctness metric.
Introduction: A Prefix Hit Doesn’t Guarantee Correct Cache State
New-generation hybrid attention LLMs have been open-sourced, and their million-token long context plus cross-request reuse bring clear benefits to inference serving. However, the long-context metrics and cache hit rates reported in the official technical report measure “how much compute was saved,” not “whether the results are correct.” In engineering practice, a previously overlooked boundary has emerged: even when the prefix cache hits, the KV cache state may not line up.
To understand this boundary, we first need to see the implicit assumption behind traditional inference serving.
Why traditional paged caching was “good enough”
In mainstream inference frameworks’ KV cache management, KV is sliced into fixed-size pages, with a block table mapping them to physical KV blocks. This approach originates from PagedAttention and has been used across various online inference systems for a long time. Two requests sharing the same prefix can share the same set of physical blocks, eliminating the repeated prefix prefill computation.
This mechanism has only two core actions:
- Allocation: pick a free block for new tokens and write the KV;
- Prefix matching: hash tokens to look up the longest common prefix in the cache tree and record reference counts.
It is reliable because of a rarely-stated premise: each token has only one cached form across all layers, and “write complete” equals “immediately usable.” At this point, a hash hit is essentially equivalent to content being ready.
Hybrid attention breaks the premise
Hybrid attention mixes different attention mechanisms across different layers of the same model, commonly:
- full-history compressed attention;
- sliding window attention (SWA);
- sparse routing attention with a lightweight indexer.
A token thus no longer maps to a single KV entry, but must be represented across multiple objects. For two requests with the same prefix, a hash hit only means “the token sequence is identical,” not that these objects are all ready, still valid, or correctly restorable.
The library analogy
Cache reuse in ordinary models is like finding the shelf number in a library and taking the book directly. Cache reuse under hybrid attention adds a layer of verification:
- Is the book already on the shelf (compressed KV)?
- Are the latest few pages already bound (uncompressed tail)?
- Are the lending rules still valid (SWA window)?
- Is the index card in sync with the content (indexer state)?
- Is the book still in transit from an off-site depot (on-disk prefix)?
Only when all five are confirmed can you truly “continue reading.”
The core conclusion
Once cache objects become heterogeneous, the allocator can no longer just find free GPU memory blocks; it must also record the current form of each token segment. Otherwise, you get cases where prefix matching reports a hit by hash, but at execution time the tail is missing, the SWA window is missing, or the index state is missing, and the generated output is immediately wrong.
This is the core argument of this article: under hybrid attention architectures, prefix cache hit rate is a performance metric, not a correctness metric. The text below uses abbreviations for five states to refer to the five categories of cache objects:
- compressed KV: the compressed KV representation;
- uncompressed tail: the token tail not yet settled;
- SWA window: in-window data for sliding-window layers;
- indexer state: the index state for sparse routing;
- on-disk prefix: prefixes residing outside the GPU.
Why “Same Prefix, So Reusable” Fails
In traditional inference systems, a cache hit usually means the following path:
Prefix matching (hash tokens to look up the longest common prefix)→ hit blocks are attached to the new request's block table→ only the unmatched suffix is prefilled→ enter token-by-token decoding
This path almost always holds on standard full-attention models, for two reasons:
- Dependencies are monotonic: the K/V of the i-th token depends only on tokens 0 through i.
- The cache is homogeneous: all layers use the same cache layout, and any block is either fully readable or not yet written.
Hybrid attention weakens both premises, and the weakening comes from the superposition of three directions.
(1) Architecture level: the same prefix has different “identities” in different layers
Within a hybrid attention layer:
- compressed attention layers need the compressed low-rank latent vectors;
- sliding-window layers keep only the in-window KV, and out-of-window history is discarded;
- sparse layers additionally need the lightweight scoring state produced by the indexer.
These objects differ entirely in write timing, residency location, and validity duration. Sticking with the binary “the whole prefix either hits or misses” judgment will inevitably treat unusable objects as usable.
(2) Scale level: million-token context exceeds single-GPU memory
KV memory usage is proportional to sequence length, and at million-token context, a single request can exceed GPU memory capacity. The system must tier storage and use compressed representations and sparse selection to reduce computation.
This means the “hit” objects may reside in different storage tiers at different readiness stages. “Found” is not the same as “reachable.”
(3) Reuse level: cross-request, cross-session sharing
System prompts, long documents, and multi-turn conversation histories are often shared by many requests. To reuse across nodes and sessions, the cache needs to be persisted outside the GPU, and even distributed across multiple instances. Once the reuse scope crosses memory tiers and node boundaries, it inevitably introduces version, consistency, and visibility issues.
With these three inputs superimposed, cache state is pushed into a situation that “a single rule cannot cover.” Handled poorly, two typical failures occur:
- A hash hit but actually unusable; at execution time, gaps or stale data are read;
- A hash miss but actually restorable; the long prefix is needlessly recomputed, and time-to-first-token (TTFT) stays high.
The Five Cache Objects: Lifecycle and Recovery Conditions
Understanding these five objects comes down to answering three questions separately: When is it produced? When is it valid? How should it be handled after a hit?
| Cache object | Typical content | Main storage location | Lifecycle | Meaning of a hit | Correct action |
|---|---|---|---|---|---|
| compressed KV | low-rank latent vectors | mainly GPU memory | resident with the corresponding tokens, subject to eviction policy | representation is generated | reuse directly after confirming boundary alignment and consistent layout |
| uncompressed tail | tail KV not yet written or not yet settled | GPU memory | within one scheduling round | content is not yet determined | always recompute; never counted as reuse |
| SWA window | K/V of in-window tokens | GPU memory | fixed window width, invalidated as the window slides | the window has not slid past this segment | reuse only the last in-window segment |
| indexer state | scoring keys, block-level indices | mainly GPU memory | synchronized with the corresponding KV | the index is in sync with the KV | reuse the index, re-run sparse selection |
| on-disk prefix | complete prefix evicted from the GPU | host memory, local disk, object storage | long, subject to capacity-based eviction | a backup has been located | fetch back asynchronously and verify first, then reuse |
3.1 compressed KV: The representation is generated, but must be boundary-aligned
Compressed attention layers don’t store the full K/V for each token directly; they project K/V into a lower-dimensional latent space and perform attention computation in that latent space. This has two direct consequences:
- the cache stores the compressed representation, not the raw K/V;
- restoration must use it in the same compressed way; it cannot be handed to other operators as ordinary K/V.
Scenarios to watch for
- Block-level compression: each cache unit represents a segment of tokens. Reusing only part of the tokens in that segment breaks the integrity of the compression unit.
- Layout across stages: if the prefill and decode stages use different operator implementations, layout details such as latent vector dimension order and whether positional encoding is fused may differ.
- Precision changes: if the compressed vector is saved in low precision, there may be numerical differences between precision-based reconstruction and direct reading. These differences are usually small, but they should be included in the comparison baseline during evaluation to avoid misjudgment.
Correct approach
- Use the compression unit as the minimum reuse unit; non-splittable compression units don’t participate in prefix reuse;
- Record the operator layout version that produced each cache, and don’t reuse across layouts.
3.2 uncompressed tail: the part whose content is not yet determined
In each scheduling cycle, the last few tokens being prefilled or just decoded may have KV that isn’t fully written or not yet settled. Reasons include:
- Fused operators: the attention operator that writes KV only produces the K/V for the last position in the current computation;
- Speculative decoding: multiple draft tokens are verified at once, and the results are not yet determined;
- Chunked prefill: a long prompt is split into several chunks, and the tail state of a chunk is not yet settled.
The tail usually lives for only one scheduling cycle, but it determines two things:
- the reusable prefix boundary;
- when a prefix segment can be shared with other requests.
If another request registers reuse just after the previous request wrote but before it settled, it will read a half-finished product.
Engineering advice
- Define “write complete” and “state settled” as two separate events. Only after settling is that segment of tokens visible externally;
- Define the reusable boundary uniformly as “the settled token sequence”; the tail doesn’t participate in any prefix matching.
3.3 SWA window: history outside the window is no longer read
Sliding-window layers’ attention only covers the most recent W tokens. Once the window slides past a position, the K/V at that position no longer affects any subsequent computation. This leads to a counterintuitive conclusion:
For sliding-window layers, “reusing the entire prefix” and “reusing only the last W tokens” produce exactly the same computation result.
Therefore:
- If you keep the entire prefix of sliding-window layers resident, you’re just wasting GPU memory;
- If you “hit” data that slid out of the window long ago and try to rely on it, you’ll either read blocks that have been reclaimed or overwritten, or incur unnecessary swap-in overhead.
Sliding-window layers also have three easily overlooked details:
- Hybrid configuration: sliding windows are usually used only for some layers, while full-history layers still need the complete prefix, so the reuse policy must be made per layer.
- Attention sinks: some implementations keep a few tokens at the very front of the sequence (attention sink), forming a “front fixed segment + sliding window” non-contiguous layout. Swap-in/swap-out can’t treat it as a contiguous range.
- Effective reuse length: let the reusable length of sliding-window layers be U (n·W when there are n sliding-window layers) and the reusable length of non-sliding-window layers be V (m·L when there are m non-sliding-window layers), with U < V. If a new request reuses the KV blocks of the previous request, writing to these blocks will overwrite in-window data that the previous request’s subsequent steps still need to read. In this case, you should copy the U-length sliding-window KV and reuse only the V − U length. Otherwise, a hard-to-locate “write-after overwrite” error occurs.
3.4 indexer state: the index is ready, but selection must be redone
Sparse attention first uses a lightweight indexer to score each query and pick a small number of historical blocks or tokens to participate in the computation. The index state usually includes a lightweight scoring key for each historical position (and block-level indices in some implementations).
Its key property is: the index state depends only on keys, independent of the current query. So it can be cached and reused along with the prefix. But the “which blocks to select” computed from it depends on the current query and must be recomputed every time. The most common misunderstanding here is conflating the two:
- using the selection result computed from an old query to serve a new query;
- reusing the KV but not reusing the index state in sync, causing missing scores.
Both approaches change the actual set of tokens participating in attention, thereby changing the output.
Moreover, sparse routing often combines with sliding windows and front fixed segments to form an attention range of “fixed part + dynamic selection.” The reuse strategy must model the three ranges separately; it can’t simply understand “sparse” as “compute only the selected part.”
3.5 on-disk prefix: found, but still in transit
Prefixes that don’t fit in the GPU are evicted to host memory, local disk, or object storage. For such prefixes, “hit” is only a locating result; to be truly reusable, the following conditions must be met:
- fully fetched back into GPU memory;
- version verification passes;
- the target GPU block is available.
This gives rise to two specific problems:
- Fetch-back latency: cold-loading shared prefixes brings a noticeable TTFT increase. You can prefetch asynchronously ahead of time based on request priority and remaining budget.
- Partial fetch-back: after a fetch-back is interrupted, it can’t simply be discarded as a “miss.” The correct approach is to record the fetched-back range and resume from the breakpoint next time, avoiding repeatedly starting from zero.
Four-Layer Hit Verification: Confirming “we can continue computing directly”
The five objects have different lifecycles. If you only do token prefix matching, you’ll miss a gap in any of these states. It’s recommended to split hit determination into four progressive layers of verification: if any layer fails, degrade to partial recompute or fallback recompute, rather than continuing execution with an incomplete state.
Layer 1: Full-Unit Alignment — hits must land on whole-block boundaries
For “non-splittable” units like compression units, block-level indices, and pages, the reusable length must be floored to the maximum size of these units. A hit whose offset lands inside a compression unit can only fall back to the boundary before that unit.
Layer 2: Intersection of multiple cache group states — the weakest link determines the reuse length
A request contains multiple cache groups, for example:
- the full-history compressed layer group;
- the sliding-window layer group;
- the sparse layer group;
- an optional state-space layer group, whose state is a fixed-length vector independent of sequence length and needs to be saved and restored as a whole per request.
Each group has its own reusable length, and the overall reusable length takes the minimum across groups.
Layer 3: Data readiness and layout consistency
Readiness needs to confirm two things:
- the data of the corresponding physical block has been fetched back into GPU memory;
- the layout is consistent with the current operator, including precision, dimension order, whether positional encoding is fused, etc.
If any item doesn’t match, that range is treated as a miss.
Layer 4: Byte-level hardware alignment
Page size must match the alignment requirements of GPU pages and the memory allocator, and in-page data must also satisfy the alignment requirements for vectorized loading; otherwise, rebuilding the block table produces wrong addressing or extra out-of-bounds reads.
The following illustrative code shows how to chain the four layers of verification together:
# Illustrative code: expresses only the decision logic, not any specific implementationfrom dataclasses import dataclass@dataclassclass CacheGroup:kind: str # full | swa | sparse | ssmunit_size: int # non-splittable unit size (in tokens)window: int | None # sliding window width; None for non-sliding-window groupsresident: list[bool] # whether each unit's data is ready in the target storagelayout_ok: list[bool] # whether each unit's layout matches the current operatordef floor_align(length: int, unit: int) -> int:"""Layer 1: floor to a full-unit boundary"""return length // unit * unitdef usable_prefix(groups: list[CacheGroup], settled_len: int,old_owner_refs: list[int] | None = None) -> int:"""Return the prefix length that can be safely reused; four layers of checks progressively narrow it"""usable = []for g in groups:length = min(settled_len, g.unit_size * len(g.resident))# Layer 2: sliding-window groups allow reuse only of the window-covered partif g.kind == "swa":length = min(length, g.window)# Layer 3: units with unready data or inconsistent layout don't participate in reuseunits = length // g.unit_sizefor i in range(units):if not g.resident[i] or not g.layout_ok[i]:length = i * g.unit_sizebreakusable.append(length)reuse = floor_align(min(usable), max(g.unit_size for g in groups))# If shared blocks will be overwritten by new token writes (e.g., the sliding window moves forward with new tokens), exclude themif old_owner_refs is not None:for ref in old_owner_refs:reuse = min(reuse, floor_align(ref, 1))return max(0, reuse)
Four Consistency Invariants: the allocator state machine must not violate them
With the above determination in place, the cache manager must also stay consistent throughout concurrent scheduling and swap-in/swap-out. The following four should be written into the state machine as hard constraints.
1. Admission control
The GPU memory budget must include “the part to be written in the future,” including the KV of new tokens and the draft tokens of speculative decoding. It’s recommended to also set a cap on the number of concurrent sequences to prevent the admission count from exceeding the number of physical blocks. Otherwise, under overload it will only cause frequent cache thrashing, offsetting the reuse benefit with swap-in/swap-out.
2. Five-state routing
After a hit, handle by state separately:
- ready, layout-consistent compressed KV is attached directly to the block table;
- the tail and unsettled ranges are always recomputed;
- the SWA window reuses only the last window-covered segment;
- the indexer state is reused with the prefix, but selection is redone for every query;
- the on-disk prefix is fetched back asynchronously and verified first, then registered as a hit after completion.
Any hit that “can’t be routed” should be downgraded to fallback recompute; it can’t continue executing as a hit.
3. Time-of-check-to-time-of-use (TOCTOU) race
Between “determining usable” and “actually reading,” the cache block may have been swapped out, overwritten, or freed by another request. It’s recommended to:
- put the hit check and reference count increment in the same critical section, or complete them with atomic operations;
- hold the reference throughout the whole process from hit check to invocation;
- use the same generation number for the state metadata of each cache group and the block table references.
Otherwise, a mismatch of “reference counts line up but state metadata doesn’t” occurs, which is hard to locate by looking at the block table alone.
The following illustrative code gives a simple “check and lock” implementation:
import threadingclass CacheTable:"""Illustrative code: use a lock to combine 'check available' and 'increment reference' into one atomic action"""def __init__(self):self.lock = threading.Lock()self.generation = 0 # generation shared by metadata and block tableself.refs: dict[int, int] = {} # block_id -> reference countself.ready: dict[int, bool] = {} # block_id -> whether data is readydef acquire(self, block_ids: list[int]) -> bool:with self.lock:if not all(self.ready.get(b, False) for b in block_ids):return Falsefor b in block_ids:self.refs[b] = self.refs.get(b, 0) + 1return Truedef release(self, block_ids: list[int]) -> None:with self.lock:for b in block_ids:self.refs[b] -= 1if self.refs[b] == 0:del self.refs[b]
4. Static execution topology
Static execution approaches with fixed shapes and fixed addresses (such as graph capture) require a stable cache address space. Dynamic swap-in/swap-out and on-demand growth break the static topology. A common approach is to reserve a fixed cache pool and reuse it by slot, ensuring address stability. For branches like “hit / partial hit / miss,” it’s recommended to reserve slots in the topology, or use uniform bucketing to keep shapes consistent, avoiding re-capture on every hit.
Engineering Implementation Advice
Establish three-stage hit semantics
Break hits into three layers to measure and count:
- metadata hit: the prefix exists in the index;
- restorable hit: passes the four-layer verification;
- actual reuse: computation is actually saved.
Only the third represents a benefit. The differences among the three precisely indicate the direction for the next phase of optimization.
Keep regression cases on hand
- Output comparison between warm cache and fully cold cache;
- Cases where the hit position lands on unit boundaries, window boundaries, block boundaries, and page boundaries;
- Speculative decoding and chunked prefill scenarios;
- Reuse after eviction then swap-in;
- Reuse of old cache after model or operator version upgrades.
Supporting capabilities for tiered caching
When GPU capacity is limited, prefixes are usually stored by temperature in tiers: hot data in GPU memory, warm data in host memory, and cold data handed to object storage. Invalidation broadcasts can leverage message queues, and runtime state connects to logging services and monitoring/alerting, observing the three metrics “metadata hit, restorable hit, actual reuse” separately. Platforms like Baidu AI Cloud also provide this kind of general capability combination. It needs to be emphasized: a high hit rate only indicates the index is effective, not that the output is correct; correctness can only be proven by the output itself.
Conclusion
Prefix caching is the performance cornerstone of long-context inference serving, but under hybrid attention architectures it changes from the binary judgment of “whole-segment hit is usable” into a collaborative problem of “multiple objects, multiple states, multiple stages.”
In engineering, it boils down to three points:
- the allocator needs to record the form of each token segment;
- hit determination needs to confirm restorability object by object;
- consistency invariants need to be enforced at the state-machine level.
Grasping these three lets you get reuse benefits in million-token contexts and cross-request reuse, without trading correctness for it.
