Long-Context Inference: Why Prefix Cache Hits Don't Guarantee Correctness

    For the past two years, paged KV management has been the de facto standard for long-context LLM serving, resting on a simple assumption: an identical prefix means the cached KV can be reused as-is. Hybrid attention, where different layers use different attention strategies, breaks that assumption. This article explains why cache hit rate is not a correctness metric, describes the five cache objects that must be tracked explicitly (compressed KV, uncompressed tail, sliding window state, indexer state, and on-disk prefix), and lays out the four layers of hit validation and the four consistency checks an allocator state machine needs. It closes with practical guidance for teams serving million-token contexts.

    1. An Assumption Trusted for Two Years That Suddenly Stopped Holding

    Over the past two years, paged KV-cache inference frameworks have become the de facto standard for long-context serving. The idea is elegant: slice the continuously growing token history into fixed-size pages, map logical positions to physical KV blocks through a page table, and when multiple requests share a prefix, reuse the existing blocks instead of recomputing the forward pass.

    This mechanism rests on an assumption that almost nobody questioned:

    Same prefix ⇒ hit ⇒ cache can be reused directly.

    Once a model adopts hybrid attention — that is, different layers use different attention strategies — this assumption shows its first systemic boundary. One point deserves emphasis: prefix cache hit rate is not a correctness indicator. No matter how high the hit rate is, it only tells you what you reused, not whether you reused the right thing.

    2. How Hybrid Attention Breaks “Same Prefix Means Reusable”

    In conventional dense attention, every position attends over the entire history, and the KV layout is single-shaped and homogeneous. That is why “same prefix → same KV → direct reuse” holds.

    Hybrid attention splits layers into three or more shapes used side by side:

    • Full-history compression: some layers compress very long histories, so what is stored is compressed KV rather than per-token raw KV.
    • Sliding window: some layers attend only within a recent fixed window; KV outside the window no longer participates in the computation and may already have been reclaimed.
    • Sparse routing: some layers use an indexer to selectively access a small number of tokens, and their validity depends on whether the routing state is aligned with the current request.

    These three shapes have different lifecycles, different recovery conditions, and different hit criteria. As a result, “the prefix is the same” no longer implies “the cache state is consistent.”

    An analogy: in a conventional model, hitting a block is like finding a shelf number in a library and walking over to pick up the book. In a hybrid-attention model, after locating the block you still have to confirm five things — whether the book is still on the shelf (is the compressed KV usable?), whether the latest pages have been bound (is the tail ready?), whether the borrowing card is still valid (does the sliding window cover this position?), whether the index catalogue has been synchronized (is the indexer state current?), and whether this book was shipped to a suburban depot and hasn’t come back yet (has the disk prefix been migrated back?). Only when all five confirm recoverability can you talk about “reading on from here.”

    3. Five Cache Object Types: Shape, Lifecycle, and Recovery Rules

    Turning that library intuition into engineering means explicitly tracking five classes of cache objects:

    1. Compressed KV: carries the compressed representation used by full-history layers. On a hit, verify that the compression granularity and compression version match the current block; otherwise compression error propagates through the deeper layers.
    2. Uncompressed tail: the most recent tokens are usually kept in raw, uncompressed form to preserve local precision. Before reuse, confirm that this portion is “bound and ready,” or generation will start with holes in it.
    3. Sliding window (SWA) state: responsible only for the most recent fixed length. The window belonging to a logically matching block may already have slid out of range, and reusing it is then simply wrong.
    4. Indexer state: the routing table / index structure that sparse routing depends on. If the index has not been synchronized with this request, routing points at the wrong tokens.
    5. On-disk prefix: history evicted to a slower storage tier because of insufficient GPU or host memory. A hit here means the block matched, but the data has not been migrated back and you still cannot continue computing from it.

    The key difference among these five objects is that their invalidations are independent of each other. Compressed KV hitting does not mean the tail is ready; a ready tail does not mean the SWA window is valid; a valid window does not mean the index is synchronized. Any single missing piece is enough to turn an apparent hit into an actual error.

    4. Extending the Allocator’s Job: From Finding Free Blocks to Tracking Shape

    In a conventional design, the allocator does one thing: pick a free block for a new token and write into it. In a heterogeneous-cache setting, that responsibility has to expand — the allocator must not only allocate physical blocks but also record the current shape of each token range (compressed, raw tail, inside the window, or already evicted).

    Without that, you get the most typical “hit but wrong” failure: the downstream prefix matcher reports a hit based on hash, and the allocator skips recomputation accordingly, but at actual read time the tail is missing, or the window, or the indexer routing — and generation collapses. The following sketch illustrates the per-state checks that must follow a hash hit:

    1. # A hash hit is only the starting point; every state must be validated before reuse
    2. def decide_reuse(cached_block, request):
    3. # 1. Does the compressed KV cover the target range?
    4. if not cached_block.compressed_kv.covers(request.range):
    5. return RECOMPUTE
    6. # 2. Is the uncompressed tail ready?
    7. if not cached_block.uncompressed_tail.ready(request.tail_len):
    8. return RECOMPUTE
    9. # 3. Is the sliding window still valid at the current position?
    10. if not cached_block.swa_window.valid_at(request.pos):
    11. return RECOMPUTE
    12. # 4. Is the indexer routing aligned with the current head?
    13. if cached_block.indexer_state.stale(request.head):
    14. return RECOMPUTE
    15. # 5. Has the on-disk prefix finished migrating back?
    16. if cached_block.on_disk_prefix.pending():
    17. return RECOMPUTE
    18. return REUSE # Only when all five states pass can we continue directly

    Moving the correctness criterion from “hashes are equal” up to “all five states are satisfied” is the dividing line that determines whether this mechanism can run reliably.

    5. Four Layers of Hit Validation: Proving You Can Continue Directly

    Even with all five states present, a hit still has to pass four layers of validation before the cache can be considered safe to resume from:

    • Block-level integrity check: validate completeness at block granularity to avoid a “half-block hit.” If any part is incomplete, fall back to recomputation for the entire block.
    • Compute kernel consistency (kernel ABI): the layout of the reused KV must match the memory layout, data types, and alignment that the current compute kernel expects; otherwise the values are present but their semantics are wrong.
    • Intersection of multi-cache-group states: under hybrid attention, several cache groups coexist (compression group, window group, index group, and so on), so a hit must be the intersection of all group states — if any group fails, the result is a miss.
    • Byte-level hardware alignment: when reusing across devices or precisions, byte-level layout must align with the hardware access granularity, so that boundary reads cannot overrun and precision cannot be misaligned.

    All four layers serve the same purpose: replace “hit rate,” a surface metric, with “safe to resume” as the correctness criterion.

    6. Four Consistency Checks: Red Lines the Allocator State Machine Cannot Cross

    Under concurrency, swap-in/swap-out, and graph execution, four consistency checks must run throughout to protect the allocator’s state machine:

    1. Admission cap: set an upper bound on cache occupancy for concurrent requests, preventing over-commitment that would leave later requests without usable cache and force constant recomputation.
    2. Five-way state routing: on the allocation and reclamation paths, explicitly separate the five streams — compressed KV, uncompressed tail, window, index, and disk prefix — instead of handling heterogeneous objects with one uniform policy.
    3. TOCTOU race between validation and use, and state pointers: a time window exists between hit validation and actual read, so state pointers (state_data_ptrs) must pin the state version as of validation time, preventing the cache from being rewritten “after validation, before use.”
    4. Static compute graph topology: on graph-captured static execution paths, memory addresses and topology are fixed at capture time, so any dynamic reuse must not break the already-captured static topology, or the execution phase will read misaligned addresses.

    7. Engineering Recommendations

    For teams already serving million-scale contexts, the conclusions above can be narrowed to a few actionable rules:

    • Move the correctness criterion from hit rate to “all five states recoverable.” Monitoring should surface the state of each of the five object classes, not a single hit-rate number.
    • Establish a strong consistency contract between prefix matching and the allocator. A hash hit must trigger per-state validation, and failed validation must fall back to recomputation rather than silently reusing the cache.
    • Design separate routing and lifecycle management for heterogeneous cache. Avoid “one policy for everything.”
    • Make the four validation layers and the four consistency checks part of the pre-launch test checklist, especially the multi-cache-group intersection, TOCTOU, and static graph topology — the three places most likely to expose defects under load.

    It is worth noting that the high concurrency, observability, and stability demands of this kind of inference service are precisely what general-purpose cloud capabilities typically cover, such as containerized orchestration, monitoring and alerting, log services, and elastic scaling — the general engineering practices on platforms like Baidu AI Cloud are one example. Even so, the correctness design of the underlying framework still has to be completed by the serving side at the usage layer; general platform capabilities cannot replace the model’s own explicit tracking of cache consistency.

    Conclusion

    The capability leap that hybrid attention brings comes at the cost of a prefix-caching reuse assumption that has become brittle. Real progress is not pushing the hit rate higher; it is admitting honestly that a hit is only a candidate, and correctness has to be endorsed jointly by five objects: compressed KV, the uncompressed tail, the sliding window, indexer state, and the on-disk prefix. Whoever manages the lifecycle and recovery conditions of these five states clearly will be able to run both fast and correctly in the long-context era.