Hybrid Attention LLMs: The Hidden Boundary of KV Prefix Cache Hits

    Prefix caching is a standard feature of mainstream LLM inference engines, built on one assumption: if the prefix tokens are identical, the cached state can be reused directly. This article shows how hybrid attention architectures quietly break that assumption. Prefix hashes still hit, yet generation becomes wrong, because the cached object itself has become heterogeneous. It explains why hybrid attention breaks the "same prefix means reusable" rule, which cache states an inference service must explicitly track, what changes allocators need, and how engineers can verify that a prefix hit is genuinely usable. Organized around five cache objects—compressed KV, the uncompressed tail, the sliding-window (SWA) window, indexer state, and on-disk prefixes—it reframes prefix caching from a hit-rate metric into a verifiable correctness guarantee.

    Introduction: A high hit rate does not mean a correct result

    Recently, several large open-source models have adopted hybrid attention architectures, where different layers of the same model use full-history compression, sliding windows, and sparse routing respectively. The cost of long-context inference serving is dominated by the KV Cache, so the industry widely uses paged KV management: it splits the attention key-value (KV) history into fixed-size blocks (pages), uses a block table to map logical positions to physical GPU-memory blocks, and lets requests with identical prefixes share the same set of blocks. This mechanism follows the naive assumption of “identical prefix, direct reuse,” and has become a standard feature of mainstream inference engines.

    But on hybrid attention models, this assumption quietly fails: the prefix hash still hits, yet the generated output is already wrong. The problem is not in the hashing algorithm, but in the cached object itself having become heterogeneous. A prefix hit is only a reuse candidate; whether the cache state is correct is the precondition for reuse.

    This article addresses four questions:

    • Why does hybrid attention break “identical prefix means safe reuse”?
    • Which cache states must inference services explicitly track?
    • What changes do allocators need?
    • How can engineers verify that a prefix hit is actually usable?

    1. The implicit premise of classic prefix caching and why it breaks

    1.1 What paged KV management solves

    In autoregressive decoding, every newly generated token must attend to the KV of all previous tokens. If history were recomputed at each step, complexity would grow sharply with sequence length, so the mainstream approach keeps historical KV in GPU memory for repeated use—this is the KV Cache.

    Prompts from different requests often share large common parts, such as system prompts, tool definitions, and multi-turn conversation history. If every request repeated the prefill pass (a full forward computation over the input to produce the initial KV), it would waste both compute and bandwidth. Paged management borrows the idea of the page table from operating-system virtual memory:

    • Chunking: KV is stored in fixed-size blocks, each covering the positions of a number of tokens;
    • Mapping: a block table maps logical positions to physical blocks, so a logically contiguous sequence can be physically non-contiguous;
    • Sharing: the prompt token sequence is hashed into a key; requests with identical prefixes point to the same set of physical blocks, and reference counting tracks the sharing relationships.

    The correctness of this design relies on an implicit premise: the KV at a given position is determined solely by the token sequence that precedes it, regardless of “how it was computed.” Under this premise, “same prefix tokens → same prefix KV → can be reused directly” is a rigorous inference.

    1.2 How hybrid attention breaks this premise

    Hybrid attention combines three kinds of mechanisms across different layers, each imposing different constraints on caching.

    • Full-history compression: some layers no longer keep the full KV of every historical token, but instead compress several consecutive tokens into a few “representative” entries at a fixed ratio. Inputs like code and structured text often contain long, highly repetitive passages—for example, JSON field names, repeated configuration items, and similar code blocks. Such prefixes can be compressed many times over, and the result is determined almost entirely by the token sequence. But compression introduces two constraints:
      • Keeping only the “compressed state” cannot generate new tokens; an uncompressed tail must be stored separately;
      • Neither the prefix hash nor the compressed-block hash can capture differences in the compression-algorithm version or quantization format. Compressed representations produced by different versions cannot be mixed.
    • Sliding window: some layers only look at the most recent fixed-length tokens, and earlier history is naturally truncated. These layers have a bounded GPU-memory footprint, but introduce new consistency problems:
      • Cached content becomes invalid as the decoding position slides, so the window’s coverage must be confirmed first;
      • Shared blocks from other requests may still be being written back, creating a risk of conflict with paged reuse.
    • Sparse routing: some layers first use a lightweight indexer to score all history, then attend only to the few top-scoring tokens. The correctness of such layers depends not only on the KV itself but also on the indexer state. When parameters such as the scoring key, query-projection weights, and Top-K count differ, even identical KV will select different sets of tokens.

    More importantly, all three mechanisms coexist in the same model, and each can also be split across all attention heads or only some of them. This means a single request may simultaneously hold several cache forms.

    1.3 Three typical triggering scenarios

    The following three scenarios amplify the problem:

    1. Intra-layer mixing: when different attention heads within a layer, or different layers, use different cache types, “hit determination” turns from a single hash comparison into a multi-dimensional joint decision.
    2. Context exceeding GPU memory: with million-scale contexts, a single request’s KV can exceed a single GPU’s memory capacity, forcing part of the prefix to be swapped out to host memory or even disk.
    3. Cross-request reuse: when requests sharing a system prompt or conversation history arrive concurrently, one request evicting or writing back a shared block can instantly invalidate another request’s cache state.

    Combined, these make it impossible for any single rule to cover all cases.

    1.4 A different angle: what a cache stores is not just “books”

    A more intuitive analogy is “retrieving archived files.” In an ordinary model, once you find the block number it is like finding a shelf number in a library—you just go and take the book. In a hybrid attention model, finding the block number is only the first step; you must then confirm five things:

    • Is the book on the shelf (is the compressed KV complete)?
    • Are the latest few pages bound (is the uncompressed tail fully written)?
    • Has the borrowing card expired (is the sliding window still valid)?
    • Has the index been synchronized (is the indexer state consistent)?
    • Has the book been sent to a suburban warehouse and not yet returned (has the on-disk prefix been swapped in)?

    Only when all five are confirmed can you truly continue reading. Throughout the rest of this article, we use the following terms for these five kinds of objects: compressed KV, uncompressed tail, SWA window, indexer state, and on-disk prefix.


    2. The five cache objects: lifecycle, recovery conditions, and hit determination

    Cache object Typical source Lifecycle characteristics Must confirm after a hit Typical misjudgment
    compressed KV Full-history compression layers A segment is long-term stable once generated; can be shared across requests The segment is complete, and its version and quantization format match the current one Comparing only the segment hash while ignoring the compression version
    uncompressed tail The near-end uncompressed segment of a compression layer Grows continuously with decoding; usable only after fully written Writing is complete; length and offset are aligned Using a tail that is only half written
    SWA window Sliding-window layers Continuously invalidated as decoding advances The window covers the current position, and shared blocks are not being concurrently written back Reusing history outside the window
    indexer state Sparse-routing layers Attached to its corresponding KV and evicted with it The index is from the same source as the KV, with consistent parameters KV hits but the index is missing, forcing a fallback
    on-disk prefix Long-prefix swap-out Determined by the swap-out policy; costly to reclaim Metadata and checksums are valid, and swap-in can happen in time Counted as a logical hit, but recovery cannot finish in time

    2.1 Compressed KV: stable, but strongly version-dependent

    Lifecycle: during request processing, the part of the prefix that reaches the compression granularity is compressed into several compressed segments. Once generated, they stay unchanged for a long time, making them well suited for cross-request sharing.

    Recovery conditions: the entries within a segment are complete, there are no coverage gaps between segments, and the compression-algorithm version and quantization format match the current request.

    Hit determination: both the segment identifier and the segment attributes must be compared. A more robust approach is to write the version and format into the metadata when the segment is created, and validate them along with the segment, rather than comparing only the token hash.

    Note: a compressed representation is approximate information, and the decoding model has already adapted to this approximation, so reuse itself introduces no new numerical error. The problem lies in a version or format mismatch, not in compression itself.

    2.2 Uncompressed tail: always growing, must be “fully written before read”

    Lifecycle: the compression granularity is often larger than the page size, so a remainder of tokens smaller than one compression unit may be left at the end of a compressed segment. This remainder is usually stored uncompressed and grows block by block as decoding proceeds.

    Recovery conditions: all blocks within the target range have been written. A half-written tail cannot count as a “hit,” or attention will read uninitialized GPU memory.

    Hit determination: judge jointly by “length + offset,” requiring the hit length to fall on a completed block boundary; the remainder must be recomputed.

    1. def tail_reusable(hit_end: int, written_len: int, page_size: int) -> int:
    2. """
    3. Return the number of reusable tail tokens; return 0 if not reusable.
    4. Only fully written pages count as a hit; any unwritten remainder must be recomputed.
    5. """
    6. completed = (written_len // page_size) * page_size
    7. return min(hit_end, completed) if completed > 0 else 0

    2.3 Sliding window: an always-moving validity period

    Lifecycle: the cache of a sliding-window layer only needs to cover a range backward from the current position; the earlier the content, the sooner it expires. Every step forward in decoding moves the window boundary accordingly.

    Recovery conditions: the distance from the hit range to the current position is within the window, and there are no gaps within the covered range.

    Hit determination: you must also distinguish “this history lies within the visible range of the sliding-window layer” from “it is shared with other requests and still being written back.” The former can be reused directly; the latter requires confirming whether the write-back has completed, and copying a private copy if necessary.

    2.4 Indexer state: meaningful only paired with the KV

    Lifecycle: the indexer state shares its source and lifecycle with the KV it serves. When the KV is evicted, the corresponding index must also be invalidated; otherwise the index would point to already-freed positions.

    Recovery conditions: the index structure corresponds to the current KV content, and the indexer version, Top-K parameters, and scoring logic match the request configuration.

    Hit determination: it is recommended to manage the KV and the index as one integrated object, referenced by a single handle, to avoid hitting only half of it.

    Cost: a missing index does not mean the prefix is entirely invalid; the index can be rebuilt on the fly, at the cost of additional computation time. Whether to rebuild depends on the payoff: the longer the hit prefix, the more worthwhile the rebuild.

    2.5 On-disk prefix: a hit does not mean it can be used in time

    Lifecycle: when the context exceeds GPU-memory capacity, cold prefixes can be swapped out to host memory, local disk, or a shared cache layer. Such prefixes remain logically valid for a long time, but whether they can be used depends on recovery resources.

    Recovery conditions: metadata is complete and checksums match; the swap-in process can obtain the required bandwidth and space exclusively and can be admitted by the current scheduler. If the swap-in has not completed, the request should be judged as an “unusable hit,” not a “partial hit.”

    Hit determination: the recovery cost must be factored into the decision. Count a segment as a hit only if it is expected to recover on time; treat the rest as missing and route it to recomputation or a swap-in queue.


    3. Allocator and prefix matching: from “finding free blocks” to a “state machine”

    3.1 The traditional approach: two parallel actions

    In traditional implementations, a KV Cache manager usually contains two non-interfering actions:

    • Allocation: finding free GPU-memory blocks for new tokens;
    • Prefix matching: looking up reusable blocks in a block table or hash table by token hash.

    The correctness of both rests on the assumption that “the block is everything.”

    3.2 After heterogeneity: both actions must be state-aware

    Once the cache objects become heterogeneous, allocation is no longer just “finding a free block”; it must also record the current form of each token segment. Otherwise the most dangerous class of error appears: prefix matching reports a hit by hash, but at actual read time the tail is missing, the window coverage is missing, or the index routing is missing, ultimately producing wrong output. The “five things” mentioned earlier are essentially answering the same question: can this cache be recovered?

    A workable approach is to model each cache segment as a small state machine:

    1. # Illustrative code: model each cache segment as a state machine (field names are illustrative)
    2. from enum import Enum, auto
    3. class SegState(Enum):
    4. EMPTY = auto() # not yet written
    5. BUILDING = auto() # being built / swapped in
    6. READY = auto() # safe to read
    7. EXPIRED = auto() # logically invalid (e.g., the SWA window has slid past)
    8. EVICTED = auto() # already evicted; can only be recomputed or swapped in
    9. class CacheSegment:
    10. tokens: range # the token range this cache corresponds to
    11. kind: str # compressed / tail / swa / indexer / on_disk
    12. version: str # compression or index algorithm version
    13. state: SegState = SegState.EMPTY
    14. ref_count: int = 0
    15. write_pos: int = 0 # length written so far; READY is allowed only when equal to the token length

    On this basis, the allocation logic turns from “find a free block” into “make a recovery plan based on the missing states”:

    1. def build_plan(request, cache):
    2. hit = cache.prefix_match(request.tokens) # get candidate hits by hash
    3. gaps = [] # gaps that need recovery
    4. for seg in hit:
    5. # returning None means the segment cannot be recovered directly
    6. reason = seg.check_reusable()
    7. if reason is None:
    8. continue
    9. if reason == "recompute": # missing, expired, or half written
    10. gaps.append(("recompute", seg.tokens))
    11. elif reason == "restore": # on disk, needs swap-in
    12. gaps.append(("restore", seg.tokens))
    13. elif reason == "rebuild_index": # index invalidated, rebuild it
    14. gaps.append(("rebuild_index", seg.tokens))
    15. plan = Plan(steps=gaps, budget=estimate_cost(gaps))
    16. # any step may fail, so a capacity admission check is required
    17. if not cache.admit(plan):
    18. return fallback(request) # fall back to conventional scheduling without prefix sharing
    19. return plan

    Key points of this illustrative code:

    • A hit is not the endpoint: every candidate segment must pass a recoverability check;
    • Gap classification: distinguish three kinds of gaps—recompute, swap-in, and index rebuild—whose resource costs are entirely different;
    • Admission up front: determine whether the plan is feasible before actually allocating GPU memory, avoiding “discovering it does not fit halfway through allocation.”

    3.3 Write-back, sharing, and early release

    Heterogeneous caches also bring three easily overlooked semantic problems:

    • Write-back timing: the uncompressed tail and the sliding-window content are both changing. If a block is marked “shareable” before the write-back completes, later requests will read incomplete data.
    • Copy-on-write: when multiple requests share the same prefix segment, private parts (such as the tail) must use copy-on-write, or they will corrupt the prefix that other requests are using.
    • Early release: invalidation timing differs greatly across cache types. Compressed segments can be kept long-term; sliding-window content can be reclaimed immediately once it slides out of range; indexes must be invalidated together with the KV. A unified “expire and clean up together” policy would cause either GPU-memory waste or dangling references.

    4. Validation before reuse: can a prefix hit really be extended directly

    A prefix-hash hit only says that “the token sequence looks the same.” What truly determines whether you can continue computing directly is the following four layers of validation. This order also echoes the “retrieving archived files” analogy: once the shelf number is right, you still have to confirm each item.

    4.1 Block alignment: the least-common-multiple constraint

    Attention computation is usually performed in units of fixed-size data blocks (tiles), while compression, paging, and sliding windows each have their own granularity. The three must satisfy a divisibility relationship: the least common multiple (LCM) of any two granularities should equal the larger one, and the page size is usually an integer multiple of the attention-block size.

    Otherwise, a page boundary would fall in the middle of an attention block, and KV concatenated across requests would look like an illegal shape to the kernel. In engineering, this set of constraints should be statically validated at service startup, rather than discovered at runtime.

    4.2 Kernel ABI compatibility: interchangeable only when homogeneous

    The same KV may have completely different memory layouts across different kernel implementations. For example, multi-head attention KV can be arranged “head-major” or “position-major,” and whether certain instruction-set features are used can also change the internal layout. If the hash covers only the token sequence, blocks produced by two implementations could substitute for each other.

    A workable constraint is to explicitly include a “layout / kernel ABI identifier” in the cache key, or to guarantee that only one layout is used globally.

    4.3 State intersection across multiple cache groups

    Hybrid attention models usually maintain several cache groups at once, for example a full-attention group, a sliding-window group, and an index group. Reuse determination should take the intersection of “all required groups being ready,” not the hit of any single group.

    1. def all_ready(groups, required):
    2. # reuse is allowed only when all required groups are READY
    3. return all(groups[g].state == SegState.READY for g in required)

    You can liken this to a natural join in relational algebra: the join result holds only when the join keys are complete and their values are consistent.

    4.4 Byte-level hardware alignment

    Swap-in and swap-out usually happen at a fixed byte granularity and require the start address to be aligned to a certain number of bytes (for example, to a cache-line or memory-page size). If the block size does not match the alignment granularity, a single copy could read or write past the boundary into adjacent blocks, causing data corruption that is hard to locate.

    Therefore, the allocator should ensure at block-selection time that “the block size is an integer multiple of the alignment granularity,” rather than padding at copy time.


    5. Four consistency checks: rules the state machine must not violate

    5.1 Admission cap

    Before accepting a request, estimate the GPU memory, bandwidth, and compute budget based on the recovery plan:

    • Admission should be based on the “total plan cost,” not on currently free GPU memory;
    • Reserve some headroom for the tail and sliding window currently being written back;
    • Set queuing and timeouts for the recovery of very large contexts, so that a few requests do not monopolize the swap-in channel.

    5.2 Five-state routing

    Different states correspond to different recovery paths and must never be handled by a single generic path:

    • compressed KV missing: can only be re-prefilled; no local repair is possible;
    • uncompressed tail missing: can be recomputed from the last complete block;
    • SWA window expired: only the part within the window range needs recomputation;
    • indexer state missing: when the KV is available, only the index needs rebuilding;
    • on-disk prefix: go through the swap-in queue and count against the recovery-time budget.

    Misclassifying a state results in performance degradation at best and wrong generation at worst.

    5.3 TOCTOU and state_data_ptrs

    TOCTOU (Time-Of-Check to Time-Of-Use) refers to the time gap between “check time” and “use time,” during which the state may change. A typical scenario:

    1. Prefix matching decides a block is reusable;
    2. The scheduler evicts the block while it is queued;
    3. By the time attention actually executes, the block already points to different data.

    Countermeasures include:

    • Using a version number (epoch) to verify the block was not modified after the check;
    • Adding a reference count or “pinning” the block during reuse to forbid eviction;
    • Passing a stable handle rather than a raw pointer, and re-resolving it at the execution entry.

    For structures that hold data pointers (such as a “state → data address” mapping like state_data_ptrs), the mapping should be updated synchronously on every state change and used together with the version check above.

    5.4 Address stability under static graph capture

    To reduce kernel launch overhead, many inference engines capture and fix the compute graph in advance. In a static execution topology like graph capture (for example, CUDA Graph), memory addresses, pointers, and shapes are fixed at capture time. If swap-in/out, reallocation, or block migration occurs at runtime, replay may access stale addresses.

    Common practices include:

    • Pre-allocating a pool of blocks at fixed addresses and managing offsets within the pool, instead of frequently allocating and freeing GPU memory;
    • Performing swap-in/out and state updates outside the graph and guaranteeing ordering through synchronization points;
    • Centrally validating handles shared between the inside and the outside of the graph.

    6. From “hit rate” to a “verifiable correctness rate”

    6.1 Split the hit rate into two kinds of metrics

    Traditional prefix caching usually only looks at the “logical hit rate,” that is, the proportion of prompt tokens that hit by hash. Under heterogeneous caching, it is recommended to also track:

    • Effective hit rate: the proportion of hits that actually pass the recoverability check;
    • Recovery failure rate: the proportion that pass the check but fail during recovery (swap-in timeout, version mismatch);
    • Per-state hit rate: the hit situation of each of the five states, used to locate bottlenecks;
    • Recovery latency distribution: the cost of tail recomputation, index rebuild, and swap-in separately.

    When the logical hit rate is high but the effective hit rate is noticeably low, it means a large number of “hits” ultimately degrade into recomputation and the system is doing useless work.

    6.2 Guard correctness with consistency tests

    Three kinds of tests are recommended:

    • Reuse vs. no-reuse comparison: for the same batch of requests, enable and disable prefix sharing separately, and compare outputs under fixed random seeds and sampling parameters to ensure token-by-token consistency;
    • Boundary cases: cover compression-granularity boundaries, page boundaries, the moment a sliding window slides out, index invalidation and rebuild, and on-disk swap-in timeouts;
    • Out-of-order and concurrency: shuffle the arrival order of requests and randomly evict caches to verify that version checks and reference counting take effect.

    6.3 Observability and degradation strategies

    Wiring the cache-state validation results, recovery-plan execution results, and version-conflict counts into logging services and monitoring alerts allows anomalies to be detected before they spread. It is also recommended to set degradation strategies: when the recovery cost exceeds a threshold, proactively give up the shared prefix and execute with conventional scheduling. Trading a little performance for correctness is far more worthwhile than returning wrong results.


    7. Conclusion

    Prefix caching itself is not wrong; the mistake is treating “same prefix” as “same state.” Hybrid attention splits the single KV array into objects with multiple forms and lifecycles: compressed segments need version validation, tails need write-completeness validation, sliding windows need validity-period validation, indexes need same-source validation, and on-disk prefixes need recoverability validation.

    For engineering teams, the key implementation points can be summarized in three items:

    1. Explicitly model the state and recovery conditions for each type of cache object;
    2. Have the allocator complete a feasibility judgment before allocating;
    3. Have every reuse go through validation and leave observable evidence.

    By doing these three things, prefix caching can evolve from a “hit-rate metric” into a “verifiable correctness guarantee.”

    When building inference services on cloud platforms such as Baidu AI Cloud, you can also leverage general cloud-product capabilities like object storage, message queues, monitoring alerts, and logging services to support the persistence and tiering of cold prefixes, the queuing and scheduling of recovery tasks, and the observability and analysis of cache states—turning the principles above into a concrete system.