Hybrid Attention LLMs and KV Cache Prefix Reuse: Why a Hit Doesn't Mean Correct State

    Hybrid attention architectures now power million-token context windows and cross-request reuse in LLM inference, but the long-context metrics and cache hit rates in official reports measure how much compute is saved, not whether the output is correct. This article explains why a KV cache prefix hit does not guarantee valid cache state: a single token now spans multiple heterogeneous cache objects. It breaks down the five cache object types—compressed KV, uncompressed tail, SWA window, indexer state, and on-disk prefix—along with their lifecycles and recovery conditions, then presents a four-layer hit verification process and four consistency invariants for the cache allocator. Engineering guidance covers three-stage hit semantics, regression cases, and tiered caching. The core takeaway: under hybrid attention, prefix cache hit rate is a performance metric, not a correctness metric.

    Introduction: A Prefix Hit Doesn’t Guarantee Correct Cache State

    New-generation hybrid attention LLMs have been open-sourced, and their million-token long context plus cross-request reuse bring clear benefits to inference serving. However, the long-context metrics and cache hit rates reported in the official technical report measure “how much compute was saved,” not “whether the results are correct.” In engineering practice, a previously overlooked boundary has emerged: even when the prefix cache hits, the KV cache state may not line up.

    To understand this boundary, we first need to see the implicit assumption behind traditional inference serving.

    Why traditional paged caching was “good enough”

    In mainstream inference frameworks’ KV cache management, KV is sliced into fixed-size pages, with a block table mapping them to physical KV blocks. This approach originates from PagedAttention and has been used across various online inference systems for a long time. Two requests sharing the same prefix can share the same set of physical blocks, eliminating the repeated prefix prefill computation.

    This mechanism has only two core actions:

    • Allocation: pick a free block for new tokens and write the KV;
    • Prefix matching: hash tokens to look up the longest common prefix in the cache tree and record reference counts.

    It is reliable because of a rarely-stated premise: each token has only one cached form across all layers, and “write complete” equals “immediately usable.” At this point, a hash hit is essentially equivalent to content being ready.

    Hybrid attention breaks the premise

    Hybrid attention mixes different attention mechanisms across different layers of the same model, commonly:

    • full-history compressed attention;
    • sliding window attention (SWA);
    • sparse routing attention with a lightweight indexer.

    A token thus no longer maps to a single KV entry, but must be represented across multiple objects. For two requests with the same prefix, a hash hit only means “the token sequence is identical,” not that these objects are all ready, still valid, or correctly restorable.

    The library analogy

    Cache reuse in ordinary models is like finding the shelf number in a library and taking the book directly. Cache reuse under hybrid attention adds a layer of verification:

    • Is the book already on the shelf (compressed KV)?
    • Are the latest few pages already bound (uncompressed tail)?
    • Are the lending rules still valid (SWA window)?
    • Is the index card in sync with the content (indexer state)?
    • Is the book still in transit from an off-site depot (on-disk prefix)?

    Only when all five are confirmed can you truly “continue reading.”

    The core conclusion

    Once cache objects become heterogeneous, the allocator can no longer just find free GPU memory blocks; it must also record the current form of each token segment. Otherwise, you get cases where prefix matching reports a hit by hash, but at execution time the tail is missing, the SWA window is missing, or the index state is missing, and the generated output is immediately wrong.

    This is the core argument of this article: under hybrid attention architectures, prefix cache hit rate is a performance metric, not a correctness metric. The text below uses abbreviations for five states to refer to the five categories of cache objects:

    • compressed KV: the compressed KV representation;
    • uncompressed tail: the token tail not yet settled;
    • SWA window: in-window data for sliding-window layers;
    • indexer state: the index state for sparse routing;
    • on-disk prefix: prefixes residing outside the GPU.

    Why “Same Prefix, So Reusable” Fails

    In traditional inference systems, a cache hit usually means the following path:

    1. Prefix matching (hash tokens to look up the longest common prefix)
    2. → hit blocks are attached to the new request's block table
    3. → only the unmatched suffix is prefilled
    4. → enter token-by-token decoding

    This path almost always holds on standard full-attention models, for two reasons:

    1. Dependencies are monotonic: the K/V of the i-th token depends only on tokens 0 through i.
    2. The cache is homogeneous: all layers use the same cache layout, and any block is either fully readable or not yet written.

    Hybrid attention weakens both premises, and the weakening comes from the superposition of three directions.

    (1) Architecture level: the same prefix has different “identities” in different layers

    Within a hybrid attention layer:

    • compressed attention layers need the compressed low-rank latent vectors;
    • sliding-window layers keep only the in-window KV, and out-of-window history is discarded;
    • sparse layers additionally need the lightweight scoring state produced by the indexer.

    These objects differ entirely in write timing, residency location, and validity duration. Sticking with the binary “the whole prefix either hits or misses” judgment will inevitably treat unusable objects as usable.

    (2) Scale level: million-token context exceeds single-GPU memory

    KV memory usage is proportional to sequence length, and at million-token context, a single request can exceed GPU memory capacity. The system must tier storage and use compressed representations and sparse selection to reduce computation.

    This means the “hit” objects may reside in different storage tiers at different readiness stages. “Found” is not the same as “reachable.”

    (3) Reuse level: cross-request, cross-session sharing

    System prompts, long documents, and multi-turn conversation histories are often shared by many requests. To reuse across nodes and sessions, the cache needs to be persisted outside the GPU, and even distributed across multiple instances. Once the reuse scope crosses memory tiers and node boundaries, it inevitably introduces version, consistency, and visibility issues.

    With these three inputs superimposed, cache state is pushed into a situation that “a single rule cannot cover.” Handled poorly, two typical failures occur:

    • A hash hit but actually unusable; at execution time, gaps or stale data are read;
    • A hash miss but actually restorable; the long prefix is needlessly recomputed, and time-to-first-token (TTFT) stays high.

    The Five Cache Objects: Lifecycle and Recovery Conditions

    Understanding these five objects comes down to answering three questions separately: When is it produced? When is it valid? How should it be handled after a hit?

    Cache object Typical content Main storage location Lifecycle Meaning of a hit Correct action
    compressed KV low-rank latent vectors mainly GPU memory resident with the corresponding tokens, subject to eviction policy representation is generated reuse directly after confirming boundary alignment and consistent layout
    uncompressed tail tail KV not yet written or not yet settled GPU memory within one scheduling round content is not yet determined always recompute; never counted as reuse
    SWA window K/V of in-window tokens GPU memory fixed window width, invalidated as the window slides the window has not slid past this segment reuse only the last in-window segment
    indexer state scoring keys, block-level indices mainly GPU memory synchronized with the corresponding KV the index is in sync with the KV reuse the index, re-run sparse selection
    on-disk prefix complete prefix evicted from the GPU host memory, local disk, object storage long, subject to capacity-based eviction a backup has been located fetch back asynchronously and verify first, then reuse

    3.1 compressed KV: The representation is generated, but must be boundary-aligned

    Compressed attention layers don’t store the full K/V for each token directly; they project K/V into a lower-dimensional latent space and perform attention computation in that latent space. This has two direct consequences:

    • the cache stores the compressed representation, not the raw K/V;
    • restoration must use it in the same compressed way; it cannot be handed to other operators as ordinary K/V.

    Scenarios to watch for

    • Block-level compression: each cache unit represents a segment of tokens. Reusing only part of the tokens in that segment breaks the integrity of the compression unit.
    • Layout across stages: if the prefill and decode stages use different operator implementations, layout details such as latent vector dimension order and whether positional encoding is fused may differ.
    • Precision changes: if the compressed vector is saved in low precision, there may be numerical differences between precision-based reconstruction and direct reading. These differences are usually small, but they should be included in the comparison baseline during evaluation to avoid misjudgment.

    Correct approach

    • Use the compression unit as the minimum reuse unit; non-splittable compression units don’t participate in prefix reuse;
    • Record the operator layout version that produced each cache, and don’t reuse across layouts.

    3.2 uncompressed tail: the part whose content is not yet determined

    In each scheduling cycle, the last few tokens being prefilled or just decoded may have KV that isn’t fully written or not yet settled. Reasons include:

    • Fused operators: the attention operator that writes KV only produces the K/V for the last position in the current computation;
    • Speculative decoding: multiple draft tokens are verified at once, and the results are not yet determined;
    • Chunked prefill: a long prompt is split into several chunks, and the tail state of a chunk is not yet settled.

    The tail usually lives for only one scheduling cycle, but it determines two things:

    • the reusable prefix boundary;
    • when a prefix segment can be shared with other requests.

    If another request registers reuse just after the previous request wrote but before it settled, it will read a half-finished product.

    Engineering advice

    • Define “write complete” and “state settled” as two separate events. Only after settling is that segment of tokens visible externally;
    • Define the reusable boundary uniformly as “the settled token sequence”; the tail doesn’t participate in any prefix matching.

    3.3 SWA window: history outside the window is no longer read

    Sliding-window layers’ attention only covers the most recent W tokens. Once the window slides past a position, the K/V at that position no longer affects any subsequent computation. This leads to a counterintuitive conclusion:

    For sliding-window layers, “reusing the entire prefix” and “reusing only the last W tokens” produce exactly the same computation result.

    Therefore:

    • If you keep the entire prefix of sliding-window layers resident, you’re just wasting GPU memory;
    • If you “hit” data that slid out of the window long ago and try to rely on it, you’ll either read blocks that have been reclaimed or overwritten, or incur unnecessary swap-in overhead.

    Sliding-window layers also have three easily overlooked details:

    1. Hybrid configuration: sliding windows are usually used only for some layers, while full-history layers still need the complete prefix, so the reuse policy must be made per layer.
    2. Attention sinks: some implementations keep a few tokens at the very front of the sequence (attention sink), forming a “front fixed segment + sliding window” non-contiguous layout. Swap-in/swap-out can’t treat it as a contiguous range.
    3. Effective reuse length: let the reusable length of sliding-window layers be U (n·W when there are n sliding-window layers) and the reusable length of non-sliding-window layers be V (m·L when there are m non-sliding-window layers), with U < V. If a new request reuses the KV blocks of the previous request, writing to these blocks will overwrite in-window data that the previous request’s subsequent steps still need to read. In this case, you should copy the U-length sliding-window KV and reuse only the V − U length. Otherwise, a hard-to-locate “write-after overwrite” error occurs.

    3.4 indexer state: the index is ready, but selection must be redone

    Sparse attention first uses a lightweight indexer to score each query and pick a small number of historical blocks or tokens to participate in the computation. The index state usually includes a lightweight scoring key for each historical position (and block-level indices in some implementations).

    Its key property is: the index state depends only on keys, independent of the current query. So it can be cached and reused along with the prefix. But the “which blocks to select” computed from it depends on the current query and must be recomputed every time. The most common misunderstanding here is conflating the two:

    • using the selection result computed from an old query to serve a new query;
    • reusing the KV but not reusing the index state in sync, causing missing scores.

    Both approaches change the actual set of tokens participating in attention, thereby changing the output.

    Moreover, sparse routing often combines with sliding windows and front fixed segments to form an attention range of “fixed part + dynamic selection.” The reuse strategy must model the three ranges separately; it can’t simply understand “sparse” as “compute only the selected part.”

    3.5 on-disk prefix: found, but still in transit

    Prefixes that don’t fit in the GPU are evicted to host memory, local disk, or object storage. For such prefixes, “hit” is only a locating result; to be truly reusable, the following conditions must be met:

    • fully fetched back into GPU memory;
    • version verification passes;
    • the target GPU block is available.

    This gives rise to two specific problems:

    • Fetch-back latency: cold-loading shared prefixes brings a noticeable TTFT increase. You can prefetch asynchronously ahead of time based on request priority and remaining budget.
    • Partial fetch-back: after a fetch-back is interrupted, it can’t simply be discarded as a “miss.” The correct approach is to record the fetched-back range and resume from the breakpoint next time, avoiding repeatedly starting from zero.

    Four-Layer Hit Verification: Confirming “we can continue computing directly”

    The five objects have different lifecycles. If you only do token prefix matching, you’ll miss a gap in any of these states. It’s recommended to split hit determination into four progressive layers of verification: if any layer fails, degrade to partial recompute or fallback recompute, rather than continuing execution with an incomplete state.

    Layer 1: Full-Unit Alignment — hits must land on whole-block boundaries

    For “non-splittable” units like compression units, block-level indices, and pages, the reusable length must be floored to the maximum size of these units. A hit whose offset lands inside a compression unit can only fall back to the boundary before that unit.

    Layer 2: Intersection of multiple cache group states — the weakest link determines the reuse length

    A request contains multiple cache groups, for example:

    • the full-history compressed layer group;
    • the sliding-window layer group;
    • the sparse layer group;
    • an optional state-space layer group, whose state is a fixed-length vector independent of sequence length and needs to be saved and restored as a whole per request.

    Each group has its own reusable length, and the overall reusable length takes the minimum across groups.

    Layer 3: Data readiness and layout consistency

    Readiness needs to confirm two things:

    • the data of the corresponding physical block has been fetched back into GPU memory;
    • the layout is consistent with the current operator, including precision, dimension order, whether positional encoding is fused, etc.

    If any item doesn’t match, that range is treated as a miss.

    Layer 4: Byte-level hardware alignment

    Page size must match the alignment requirements of GPU pages and the memory allocator, and in-page data must also satisfy the alignment requirements for vectorized loading; otherwise, rebuilding the block table produces wrong addressing or extra out-of-bounds reads.

    The following illustrative code shows how to chain the four layers of verification together:

    1. # Illustrative code: expresses only the decision logic, not any specific implementation
    2. from dataclasses import dataclass
    3. @dataclass
    4. class CacheGroup:
    5. kind: str # full | swa | sparse | ssm
    6. unit_size: int # non-splittable unit size (in tokens)
    7. window: int | None # sliding window width; None for non-sliding-window groups
    8. resident: list[bool] # whether each unit's data is ready in the target storage
    9. layout_ok: list[bool] # whether each unit's layout matches the current operator
    10. def floor_align(length: int, unit: int) -> int:
    11. """Layer 1: floor to a full-unit boundary"""
    12. return length // unit * unit
    13. def usable_prefix(groups: list[CacheGroup], settled_len: int,
    14. old_owner_refs: list[int] | None = None) -> int:
    15. """Return the prefix length that can be safely reused; four layers of checks progressively narrow it"""
    16. usable = []
    17. for g in groups:
    18. length = min(settled_len, g.unit_size * len(g.resident))
    19. # Layer 2: sliding-window groups allow reuse only of the window-covered part
    20. if g.kind == "swa":
    21. length = min(length, g.window)
    22. # Layer 3: units with unready data or inconsistent layout don't participate in reuse
    23. units = length // g.unit_size
    24. for i in range(units):
    25. if not g.resident[i] or not g.layout_ok[i]:
    26. length = i * g.unit_size
    27. break
    28. usable.append(length)
    29. reuse = floor_align(min(usable), max(g.unit_size for g in groups))
    30. # If shared blocks will be overwritten by new token writes (e.g., the sliding window moves forward with new tokens), exclude them
    31. if old_owner_refs is not None:
    32. for ref in old_owner_refs:
    33. reuse = min(reuse, floor_align(ref, 1))
    34. return max(0, reuse)

    Four Consistency Invariants: the allocator state machine must not violate them

    With the above determination in place, the cache manager must also stay consistent throughout concurrent scheduling and swap-in/swap-out. The following four should be written into the state machine as hard constraints.

    1. Admission control

    The GPU memory budget must include “the part to be written in the future,” including the KV of new tokens and the draft tokens of speculative decoding. It’s recommended to also set a cap on the number of concurrent sequences to prevent the admission count from exceeding the number of physical blocks. Otherwise, under overload it will only cause frequent cache thrashing, offsetting the reuse benefit with swap-in/swap-out.

    2. Five-state routing

    After a hit, handle by state separately:

    • ready, layout-consistent compressed KV is attached directly to the block table;
    • the tail and unsettled ranges are always recomputed;
    • the SWA window reuses only the last window-covered segment;
    • the indexer state is reused with the prefix, but selection is redone for every query;
    • the on-disk prefix is fetched back asynchronously and verified first, then registered as a hit after completion.

    Any hit that “can’t be routed” should be downgraded to fallback recompute; it can’t continue executing as a hit.

    3. Time-of-check-to-time-of-use (TOCTOU) race

    Between “determining usable” and “actually reading,” the cache block may have been swapped out, overwritten, or freed by another request. It’s recommended to:

    • put the hit check and reference count increment in the same critical section, or complete them with atomic operations;
    • hold the reference throughout the whole process from hit check to invocation;
    • use the same generation number for the state metadata of each cache group and the block table references.

    Otherwise, a mismatch of “reference counts line up but state metadata doesn’t” occurs, which is hard to locate by looking at the block table alone.

    The following illustrative code gives a simple “check and lock” implementation:

    1. import threading
    2. class CacheTable:
    3. """Illustrative code: use a lock to combine 'check available' and 'increment reference' into one atomic action"""
    4. def __init__(self):
    5. self.lock = threading.Lock()
    6. self.generation = 0 # generation shared by metadata and block table
    7. self.refs: dict[int, int] = {} # block_id -> reference count
    8. self.ready: dict[int, bool] = {} # block_id -> whether data is ready
    9. def acquire(self, block_ids: list[int]) -> bool:
    10. with self.lock:
    11. if not all(self.ready.get(b, False) for b in block_ids):
    12. return False
    13. for b in block_ids:
    14. self.refs[b] = self.refs.get(b, 0) + 1
    15. return True
    16. def release(self, block_ids: list[int]) -> None:
    17. with self.lock:
    18. for b in block_ids:
    19. self.refs[b] -= 1
    20. if self.refs[b] == 0:
    21. del self.refs[b]

    4. Static execution topology

    Static execution approaches with fixed shapes and fixed addresses (such as graph capture) require a stable cache address space. Dynamic swap-in/swap-out and on-demand growth break the static topology. A common approach is to reserve a fixed cache pool and reuse it by slot, ensuring address stability. For branches like “hit / partial hit / miss,” it’s recommended to reserve slots in the topology, or use uniform bucketing to keep shapes consistent, avoiding re-capture on every hit.

    Engineering Implementation Advice

    Establish three-stage hit semantics

    Break hits into three layers to measure and count:

    • metadata hit: the prefix exists in the index;
    • restorable hit: passes the four-layer verification;
    • actual reuse: computation is actually saved.

    Only the third represents a benefit. The differences among the three precisely indicate the direction for the next phase of optimization.

    Keep regression cases on hand

    • Output comparison between warm cache and fully cold cache;
    • Cases where the hit position lands on unit boundaries, window boundaries, block boundaries, and page boundaries;
    • Speculative decoding and chunked prefill scenarios;
    • Reuse after eviction then swap-in;
    • Reuse of old cache after model or operator version upgrades.

    Supporting capabilities for tiered caching

    When GPU capacity is limited, prefixes are usually stored by temperature in tiers: hot data in GPU memory, warm data in host memory, and cold data handed to object storage. Invalidation broadcasts can leverage message queues, and runtime state connects to logging services and monitoring/alerting, observing the three metrics “metadata hit, restorable hit, actual reuse” separately. Platforms like Baidu AI Cloud also provide this kind of general capability combination. It needs to be emphasized: a high hit rate only indicates the index is effective, not that the output is correct; correctness can only be proven by the output itself.

    Conclusion

    Prefix caching is the performance cornerstone of long-context inference serving, but under hybrid attention architectures it changes from the binary judgment of “whole-segment hit is usable” into a collaborative problem of “multiple objects, multiple states, multiple stages.”

    In engineering, it boils down to three points:

    • the allocator needs to record the form of each token segment;
    • hit determination needs to confirm restorability object by object;
    • consistency invariants need to be enforced at the state-machine level.

    Grasping these three lets you get reuse benefits in million-token contexts and cross-request reuse, without trading correctness for it.