Designing Consistency for Mixed-Attention Long-Context Cache Reuse
This article explores the engineering challenges of reusing prefix caches in long-context inference when models employ mixed attention mechanisms. It explains why prefix hash hits no longer guarantee safe cache reuse, introduces five types of cache objects with distinct recovery semantics, and proposes a four-layer validation and four consistency-check framework to ensure that cached KV data can be reliably resumed. The goal is to help system designers move from “logical hit” to “verified resumable” cache reuse.
Introduction: New Boundaries Brought by Long Context and Mixed Attention
As model contexts reach the million-token scale, the core challenge for inference services is no longer just “can we compute it,” but “how to stably reuse previously computed results under limited GPU memory and bandwidth.” Prefix caching has therefore become a key optimization for long-context inference: for identical or partially identical prefixes, the server aims to directly reuse cached KV data to avoid recomputing attention keys and values, reducing time-to-first-token and overall compute cost.
Traditional page-based KV cache management is built on a simple assumption: token history is divided into fixed-granularity pages, and a page table maps them to physical KV blocks. As long as two requests share the same prefix, the same blocks can be found via hash matching and directly reused. This works well under a single attention structure, because the same token sequence typically corresponds to a consistent cache shape and lifecycle across all layers.
Mixed attention changes this assumption. To balance long-history modeling and computational efficiency, a model may use different attention mechanisms across layers: some layers compress the full history, some only retain fine-grained KV within a sliding window, and some use sparse routing to select a few historical fragments. For the same token prefix, different layers and different cache groups may hold it in different states: compressed, recent-tail-only, sliding-window-constrained, routing-index-dependent, or even swapped out to disk.
Consequently, the rule “same prefix → direct cache reuse” is no longer sufficient. A prefix hash hit only indicates that the token sequence is identical; it does not guarantee that the cache objects are physically complete, usable, or recoverable. Therefore, hit rate is not the correct correctness metric. What must be guaranteed is the recoverable hit rate.
Why “Prefix Hit” Does Not Equal “Cache Resumable”
Think of traditional cache reuse as looking up a book by its shelf number: once you find the number, you go and fetch the book. In the mixed-attention scenario, finding the block number is only the first step. You then need to verify several things: whether the compressed historical block is still in memory; whether the recent uncompressed tail is complete; whether the sliding window covers the required positions; whether the sparse routing index is in sync with the tokens; and whether the prefix swapped out to disk has been migrated back and validated. Only when all conditions are satisfied can the cache be safely used for continuation.
More concretely, once cache objects become heterogeneous, the allocator is no longer just “find an empty memory slot to write new tokens.” It must also record for each token segment its current shape, version, and recovery path. Meanwhile, the prefix-matching module may still report a hit based on hash, but if the actual read lacks the tail, the sliding window, or the routing index, the generation result will be incorrect. This situation—logical hit but physical unavailable—is the most underestimated risk in long-context serving with mixed attention.
Five Types of Cache Objects and Their Recovery Semantics
In practice, it is recommended to explicitly divide reusable caches into five object types and define their lifecycle, recovery conditions, and hit judgment separately.
Compressed KV: This represents the compressed key-value representation of the full history. Its recovery condition is not only that the block is in memory, but also that the compression version and granularity match the current layer configuration. If the compressed block has been evicted, it must be migrated back and its integrity verified.
Uncompressed Tail: This represents the recent fine-grained KV that has not yet gone through compression. It is usually tightly coupled with the current generation position. Recovery must confirm that the tail length, offset, and token boundaries are exactly aligned. Missing a tail cannot be simply skipped, as it would make the attention window incomplete.
Sliding Window: This represents the historical scope actually visible to sliding-window attention. Hit judgment cannot only check block existence; it must also check whether the window start, end, stride, and current query position match. Window boundary misalignment can cause the visible history to diverge from the training/inference semantics.
Routing Index State: This represents the index, metadata, or selection results that sparse routing depends on. It must be updated synchronously with token history. If the index lags behind the KV data, reuse will route to wrong fragments.
Disk Prefix: This represents prefix objects that have been pushed to external storage. Its recovery path includes migration back, format conversion, validation, and re-registration. Before migration completes, it can only be considered “potentially reusable,” not directly usable in computation.
The lifecycles of these five object types are not uniform: some stay resident in memory, some are evicted by sliding-window policies, some are merged by compression strategies, and some are swapped out by hot/cold tiering. Therefore, the prefix-matching result must carry a state signature and a restore plan, rather than returning just a hash hit marker.
hit_result = {prefix_hash,object_states: {compressed_kv: READY | EVICTED | STALE,uncompressed_tail: READY | MISSING | PARTIAL,swa_window: READY | OUT_OF_RANGE,indexer_state: READY | LAGGING,on_disk_prefix: READY | MIGRATING},restore_plan: [...]}
The Allocator: From “Finding an Empty Block” to a State Machine
Under mixed attention, the allocator must take on two responsibilities simultaneously: allocating physical blocks for new tokens, and maintaining the current cache shape of each token segment. Traditional allocation only picks an empty block and writes into it; the new allocator must also record whether a token segment is compressed, whether the tail is complete, whether the sliding window is valid, whether the index is in sync, and whether it is on disk.
More critically, the allocation action and the prefix-matching action must be coordinated. Prefix matching may report a hit based on hash, but the allocator may discover that the corresponding object lacks a tail or has an outdated index. In that case, the correct approach is not to force reuse, but to generate a restore plan: migrate back what can be migrated, fill in what can be filled, and fall back to full or partial recomputation when recovery is impossible. In practice, the reuse decision can be abstracted as a three-phase flow: validate → restore → compute, with timeout and degradation policies for each phase.
plan = build_restore_plan(prefix_hit)for obj in plan.required_objects:if not obj.is_recoverable():return fallback_recompute(plan.safe_prefix_len)execute_restore(plan)append_new_tokens()
Four-Level Validation: Deciding Whether a Hit Can Actually Continue
Even when all five object types exist, four levels of validation are still needed to confirm “resumable computation.”
The first level is block-granularity full-block validation. Different cache groups may have different block sizes; the reuse boundary must be aligned to the least common multiple or a unified granularity. Otherwise, a half-block reused and half-block recomputed state can occur.
The second level is kernel ABI validation. Different attention kernels expect different layouts, strides, and metadata formats. A hit block must satisfy the target kernel’s ABI constraints; otherwise, it needs reordering or conversion.
The third level is multi-cache-group state intersection. Mixed attention often involves multiple cache groups. A hit is valid only when all required groups are in a recoverable state; if any group is missing, the safe reuse length must be reduced.
The fourth level is byte-level hardware alignment validation. Recovery, DMA, and vectorized reads have alignment requirements on addresses and lengths. Misalignment can trigger extra copies, out-of-bounds reads, or severe performance degradation.
Four Consistency Checks: Rules the State Machine Must Not Violate
To avoid “state changed after validation,” the server also needs four consistency checks.
Admission limit: Limit the number of requests entering the recovery flow simultaneously, to prevent a large number of hit requests from triggering migration or gap-filling at the same time and congesting the recovery queue.
Five-state routing: Route requests by object state into direct reuse, wait-for-recovery, partial recomputation, or full recomputation paths, to avoid all hits going through the same queue.
TOCTOU and state-data pointers: There is a time window between validation and actual use. Version numbers, reference counts, or state snapshots must be used to ensure that an object is not evicted or modified during reading.
Static graph execution topology: Device-side static graph execution typically requires a stable topology. If the recovery path dynamically changes branches, the branch must be fixed in advance or the execution graph must be recaptured to avoid runtime topology mismatch.
Engineering Recommendations
When implementing, it is recommended to split the “prefix hit rate” into two metrics: hash hit rate and recoverable hit rate. The former only reflects token-prefix matching; the latter reflects the proportion that is truly safe to reuse. Additionally, monitor the miss rate, recovery latency, and failure reasons for each of the five cache object types, and run targeted stress tests at long-context boundaries, sliding-window boundaries, and compression switch points.
Mixed attention brings not a single optimization point, but a systemic challenge in cache state management. The real advantage lies not in reporting a higher hit number, but in incorporating heterogeneous cache objects into a verifiable, revertible, and observable reuse state machine. Only when a prefix hit simultaneously satisfies the recovery of all five object types, the four-level validation, and the four consistency checks can cache reuse move from “looks like a hit” to “confirmed resumable.”
