State Cache Reuse and Prefill Acceleration for Hybrid-Architecture LLMs

    This article explains how to reuse caches and accelerate prefill for hybrid-architecture LLMs that combine a few full-attention layers with mostly linear layers (linear attention/linear RNN and state-space modules). It contrasts the two kinds of reusable state, the token-sharded KV cache of attention layers and the fixed-size state matrix of linear layers, and shows why cache reuse is harder for hybrid models. You will find a unified design based on "prefix snapshots," block-level key design and prefix-tree matching, layered placement that keeps state in remote caches and KV close to compute, snapshot continuation with a replay-and-advance pipeline after a cache hit, handling for branching, editing, and distributed storage, plus prefill scheduling (affinity routing, single-flight, chunked prefill), consistency safeguards, and a gray-release checklist. It targets engineers building long-context coding agents on hybrid models who want lower TTFT without correctness regressions.

    Several open-source LLM families now use a hybrid structure of “a few full-attention layers plus a majority of linear layers.” Here, linear layers refer to linear attention/linear RNN and state-space (SSM) modules. This structure trades a small amount of global attention for a smaller cache footprint and faster token-by-token decoding, and it has become one of the important design directions for long-context models.

    In coding-agent scenarios, however, the picture is different. A coding agent often needs to run ten or even dozens of turns in a row: reading multiple files, calling retrieval and command-execution interfaces, analyzing errors, editing code, and running regression checks. Each turn resubmits the entire prior context to the model, so the input length grows quickly from a few thousand tokens to hundreds of thousands. This means the dominant cost per turn falls on the prefill stage rather than on token-by-token decoding, and time-to-first-token (TTFT) directly determines how the agent feels to interact with.

    Traditional prefix caching relies on “skip on hit” to reduce TTFT, but for hybrid models what can be reused is not only the KV cache but also the state matrices of linear layers. The two representations differ significantly in performance characteristics and reuse semantics. Ignoring this difference and simply applying the caching strategies from the pure-Transformer era not only makes the gains fall short of expectations but can also introduce correctness problems such as inconsistent outputs.

    This article is not tied to any specific system implementation. Instead, it abstracts cache reuse under a hybrid architecture into a general engineering problem, broken down as follows:

    • The differences between the two kinds of state;
    • Why hybrid-model caching is harder;
    • A unified cache model and key design;
    • Cache tiering and replay-then-resume after a hit;
    • Three scenarios: branching, editing, and distributed storage;
    • Scheduling, consistency guarantees, and gray-release validation.

    2. Two kinds of cache: token-sharded KV and model-specific state

    2.1 Full-attention layers: KV cache sharded by position

    Full-attention layers follow standard scaled dot-product attention. To avoid recomputation, inference caches the key (K) and value (V) for every historical token. Its characteristics are:

    • The size is proportional to the context length; per token the cache size = 2 (K and V) × KV heads per layer × head dimension × bytes per element;
    • It is stored item by item by token position, so any prefix can be sliced off and used directly;
    • Writes are append-only, and any historical position can serve as the starting point for computing a new token.

    This addressable, truncatable structure is exactly the modeling basis for mechanisms such as paged KV cache and prefix-tree caching.

    2.2 Linear layers: a state matrix that rolls forward as tokens arrive

    The core of linear attention and SSM-style modules is to summarize all history with a fixed-size state. Each time a token is read in, the state is updated in a rolling fashion:

    • Linear attention: update the state matrix incrementally with the outer product of the key vector and the value vector, then apply a decay factor to forget stale information;
    • State-space models: the state evolves according to a discretized transition matrix, then the output-mapping matrix is applied to produce this layer’s output.

    For both kinds of modules, the output comes from “query vector × current state.” Their shared characteristics are:

    • The size of a single-element state matrix depends only on the hidden dimension (state dimension, key dimension, value dimension), not on the context length;
    • There is strict temporal dependence between states: given an initial state and a fixed token sequence, the result is fully determined, but you cannot “arbitrarily slice” from the middle of the sequence;
    • During inference the state rolls forward token by token; during prefill, a long sequence is chunked and scanned in parallel, using intra-chunk parallelism and inter-chunk recurrence to balance throughput and correctness.

    The most intuitive difference between the two is size. Using a mainstream parameter scale as an example:

    • A model with 32 attention layers, 8 KV heads, and a head dimension of 128 uses about 2 GiB of KV when processing 32K tokens;
    • If the linear layers have a key dimension and value dimension of 128 each, then at the same scale the combined state of 32 linear layers is only about 32 MiB;
    • When the context grows to 128K, KV grows to about 8 GiB, while the state is still only about 32 MiB.
    Dimension Attention-layer KV cache Linear-layer state
    Cache size Proportional to token count Independent of token count; fixed per layer
    Indexing Addressed by position; any prefix can be truncated Only one “full-history” entry, with no positional index
    Incremental growth Append the K and V for new tokens Must resume from a complete prefix state; no partial replacement
    Copy and transfer Large; costly to migrate across nodes Small; suited to remote sharing and multiple replicas
    Reuse granularity Supports page-level and block-level partial hits Can only resume from a snapshot as a whole; no partial hits
    Consistency risk Numerically deterministic; largely consistent across implementations Affected by gating, scan algorithm, and chunking; subject to numerical drift

    3. Why hybrid-model caching is harder: three root causes

    3.1 A hit is not just “finding the KV”—the state is the real entry point for reuse

    In a pure Transformer model, a cache entry that covers the prefix’s KV lets you skip that computation directly. Hybrid models are different: a linear layer’s output depends on its state. If you reuse only the KV without the corresponding state, you must re-run that prefix across all linear layers before you can continue computing new tokens. The result is that the KV is saved, but the prefill work is only partly reduced.

    Conversely, once the state is missing, the prefix cannot be “skipped as-is” even if the KV is complete. Therefore, a hybrid-model cache hit must satisfy: the KV and the state correspond to the same prefix length and the same model version. When the operator implementation, chunk size, or precision differs, you should also check whether there is a numerical difference between the two.

    3.2 Partial hits are costly, so cache boundaries are more “rigid”

    In a pure Transformer model, hitting 90% of a prefix usually lets you skip about 90% of the prefill computation, and it naturally supports cache paging, forking, and merging. In a hybrid model, the state is a single block of data with no intermediate form such as “the 80%-complete state”:

    • Either you hold a snapshot covering the entire prefix and skip the whole segment;
    • Or you can only fall back to an earlier snapshot and then replay token by token;
    • During replay you cannot skip any token, and the compute cost is proportional to the rollback distance.

    Moreover, replay is performed layer by layer in series, so GPU utilization is often noticeably lower than in normal prefill. This makes the location of state snapshots critical: if a snapshot is kept at a “cheap” location, the cost of a partial hit is low; if it is kept at an “expensive” location, most of the benefit is eaten up.

    3.3 Hit and cache ownership are asymmetric, so scheduling must be redesigned

    KV is large, so it can usually only be stored close to the compute node, and “the node that holds it” decides whether a hit is possible. State is small, so it can be centralized in a distributed cache and shared across multiple nodes, and a hit no longer depends on which node the request is routed to.

    This brings two changes:

    • Cache hits can be decoupled from request routing, giving more scheduling freedom, but state transfer becomes a new bottleneck that needs a dedicated bandwidth budget;
    • The focus of storage-tier design shifts from “how much KV fits” to “how fast state can be distributed and how complete snapshot retention is.”

    In addition, coding agents often “retry by falling back when one path does not work”: one prefix spawns multiple branches, or a rerun happens after changing the prompt or model version. In a pure Transformer scenario, branch reuse relies on block-level KV sharing; in a hybrid scenario, a state snapshot is a ready-made fork point, but once a branch grows long, snapshot density directly determines rollback cost.

    4. Design principle: treat hybrid state as a “unified prefix snapshot”

    The core of addressing these problems is to treat attention-layer KV and linear-layer state as different parts of the same cache object, and to abstract them uniformly as a “prefix snapshot.” The following five principles are recommended.

    4.1 Unified snapshot semantics

    A prefix snapshot = the complete set of intermediate computation results of a fixed token sequence on a given model, including the KV of all attention layers and the state matrices of all linear layers. A snapshot’s metadata should at least include:

    • The model identifier (weight version or fingerprint) and quantization/precision configuration;
    • The tokenizer and prompt-template versions;
    • Key numerical parameters such as the chunk size used by the linear-layer scan;
    • The corresponding prefix length, the KV page-table pointers, and the state storage location.

    4.2 Key-value separation

    • Use a logical key for prefix matching, such as a token-sequence hash, session ID, and turn information;
    • Store the data payload in a layered store and reference it through handles.

    This avoids the key-comparison overhead of long prefixes and makes it easy to store and evict the KV and the state separately.

    4.3 Three-tier resource profile

    • KV cache: large, frequently accessed, value proportional to token count, naturally supports paging;
    • Linear state: small, accessed at a single point, supports whole-prefix reuse for any prefix, suited to cross-node sharing;
    • Snapshot directory: small metadata, but the basis for hit decisions and routing decisions.

    The eviction and placement strategies for the three should be designed separately, rather than handled uniformly by “bytes per layer.”

    4.4 Layered placement: keep the expensive close and the cheap far

    Arrange the tiers by “access frequency × reuse value ÷ transfer cost”:

    • GPU memory: holds only the current request and hot prefixes, mostly KV;
    • Node-local memory: holds the KV and state most recently generated on this node;
    • Distributed cache: centrally stores state snapshots, directory metadata, and hot KV pages;
    • Object storage: serves as persistent archival after a session ends, for cross-session reuse.

    The “must continue computing after a hit” property of linear state also means the earlier a snapshot is kept, the lower the cost of a partial hit, so high-frequency prefixes should keep several early snapshots.

    4.5 Bidirectional feedback between caching and generation

    A cache-hit decision depends on a deterministic prefix. Therefore:

    • Prompt assembly should keep stable content at the front and place tool results appended during the session at the end;
    • Operations that change history, such as truncation and summary compression, should be done at turn boundaries, and the affected snapshots should be invalidated promptly;
    • Agent systems should keep sampling reproducible (for example, by fixing the random seed) to avoid the same prefix producing different results on two runs, which would undermine the reuse value of snapshots.

    5. Key design: make hits decidable, alignable, and repairable

    5.1 Use a prefix tree plus block-level keys to locate the longest common prefix

    Organize the session history as a token sequence and split it into blocks of a fixed size. Each block’s key is composed of “model fingerprint + block type + block content hash,” and adjacent block keys are chained into a prefix chain:

    1. # Illustrative code: block-level keys and prefix matching
    2. MODEL_FP = "model-v1.3" # weight fingerprint + quantization/precision config + linear-layer scan chunk size
    3. def block_key(kind: str, token_ids: list[int]) -> bytes:
    4. # kind distinguishes block types such as system / history / user, to avoid
    5. # false sharing of identical content with different semantics
    6. return hashlib.sha256(
    7. MODEL_FP.encode() + kind.encode() + np.asarray(token_ids, dtype=np.int32).tobytes()
    8. ).digest()
    9. class PrefixIndex:
    10. """Map a request's token sequence to reusable KV pages and state snapshots"""
    11. def match(self, token_ids: list[int], block_size: int) -> tuple[int, int]:
    12. prefix_len, hit_pages = 0, []
    13. for start in range(0, len(token_ids), block_size):
    14. key = block_key("history", token_ids[start:start + block_size])
    15. page = self.pages.get(key) # KV page table: block key -> per-layer KV page handles
    16. if page is None:
    17. break
    18. hit_pages.append(page)
    19. prefix_len = start + block_size
    20. return prefix_len, sum(hit_pages, [])
    • Block-level keys turn prefix matching into O(n/block_size) table lookups, avoiding per-token hashing;
    • The block key encodes both the model fingerprint and the block type, naturally preventing false hits across different models and templates;
    • If the last block misses, you can fall back token by token within the block, turning a partial hit into the intermediate state of “KV usable, state needs replay.”

    5.2 Directory: register state snapshots and KV pages into the same prefix chain

    Maintain a directory for each prefix chain that records the state snapshots saved on that prefix (snapshot location, token offset, version number) along with the KV page handles. The directory is very small and can be replicated to all inference nodes, serving two purposes:

    • Determining the length of the hit prefix;
    • Deciding from which snapshot to resume computation.

    5.3 Invalidation: generational management by version

    Model upgrades, changes to quantization, and changes to the linear-layer scan implementation all invalidate old snapshots. It is recommended to manage this with a unified “generation number”:

    • After a new generation goes live, snapshots from the old generation may only be read on a hit, and no new writes are allowed;
    • Clean them up only after natural eviction or after gray release completes.

    At the same time, run sampling validation at key points to check whether the difference between “continue from a hit” and “full recomputation” stays within tolerance.

    6. Cache tiering and cross-node reuse: state goes remote, KV stays local

    6.1 A state-centric distributed cache

    Linear state is independent of context length, and a single prefix needs only tens of MiB. Even retaining snapshots for hundreds of sessions stays in the GiB range, which suits a shared in-memory cache service. For example:

    • Put the state of the first few turns of each session into a shared memory cache;
    • Use a message/task queue to asynchronously sync state updates to replicas, reducing the impact of a single point of failure;
    • Use object storage to save session-level archives for offline analysis or cross-session reuse.

    For coding agents, the most valuable prefixes are long-lived, cross-session-stable ones such as the system prompt, repository documentation, and code-structure overview. Their cache can be reused repeatedly across sessions and nodes.

    6.2 Keep KV local, evict by value

    KV is large and bandwidth-sensitive, so it should stay local to the compute node first, spilling to fast local storage only when necessary. Introduce a value function for eviction:

    • Reuse value = expected prefill compute saved over the remaining conversation turns × hit probability;
    • KV in the early turns of a session is accessed frequently and has a long reuse cycle, so give it a higher weight;
    • The closer a turn is to the end of the session, the fewer the remaining turns and the lower the weight.

    6.3 Sharded layout and cross-node migration

    Under tensor parallelism, attention heads are split across devices, so KV is distributed by layer and by head; linear state is also split along with the attention heads. Cross-node reuse must keep the sharded layout consistent, otherwise a repartition is needed first. It is recommended:

    • Scale inference services up and down at the granularity of a “shard group,” not a single instance;
    • Migrate or rebuild the KV and state as a group, to avoid a half-migration that causes hit failures.

    7. After a hit: snapshot continuation and a “replay-and-advance” pipeline

    7.1 Continue from a snapshot instead of replaying from zero

    Let the hit prefix length be H and the nearest state snapshot be at S (S ≤ H). You only need to:

    1. Mount the KV pages for the region before S directly onto the request’s page table;
    2. Load the linear-layer state at S as the initial value;
    3. Replay the interval [S, H) across all linear layers to obtain the state at H;
    4. Take H as the new starting point and run prefill normally.

    Compared with replaying the entire prefix from zero, the replay length shrinks from H to H − S. This is the benefit brought by “segmented snapshots.”

    7.2 The trade-off in snapshot granularity

    The longer the snapshot interval, the less storage and fewer writes, but the more replay on a partial hit; the shorter the interval, the lower the continuation cost after a hit. In practice:

    • Use session heat to guide snapshot density, keeping more snapshots for hot sessions;
    • Set the interval by balancing “the cost of replaying one interval” against “the storage cost of one snapshot”;
    • For a fixed, unchanging system prompt, a single snapshot is enough.

    7.3 Overlap replay with advance

    Continuation and normal prefill have different operator shapes and can be pipelined:

    • A state-replay thread advances the snapshot to the hit position;
    • The prefill thread begins processing new tokens as soon as its dependencies are ready.

    For long sessions, this can usually compress the wait time for a partial hit to a fraction of the original.

    1. # Illustrative code: KV mounting, state replay, and snapshot write-back after a hit
    2. def build_from_snapshot(token_ids, hit_len, seg_id, snap_stride, model):
    3. snap_len = seg_id * snap_stride # position of the most recent state snapshot
    4. kv_handle = load_kv_pages(token_ids[:snap_len]) # mount directly via the page table
    5. state = load_state_snapshot(seg_id) # read only the linear-layer state (small)
    6. # 1) Advance the snapshot to the hit position: must replay layer by layer in series,
    7. # to keep the numerical implementation consistent with the snapshot
    8. for layer in model.linear_layers:
    9. state[layer] = layer.scan(token_ids[snap_len:hit_len], state[layer])
    10. # 2) Start normal prefill from the hit position; the KV only handles new tokens
    11. outputs = model.prefill(token_ids[hit_len:], kv_handle, state)
    12. # 3) Write back: new KV pages + save a state snapshot every snap_stride crossed
    13. store_kv_pages(token_ids[snap_len:hit_len], kv_handle)
    14. for pos in range(snap_len + snap_stride, hit_len + 1, snap_stride):
    15. store_state_snapshot(pos // snap_stride, snapshot_at(model, token_ids[:pos], state))
    16. return outputs

    8. Three typical scenarios: branching, editing, and distributed storage

    8.1 Branch reuse

    Coding agents often switch between plan A and plan B, or try different model and prompt versions on the same context. Hybrid models can copy a snapshot onto multiple paths and advance them in parallel:

    • Each branch only needs to append KV for its new tokens;
    • The linear state is copied from the snapshot, so branches do not interfere with each other.

    Pure Transformer scenarios usually rely on block-level KV sharing to fork; in a hybrid scenario, a state snapshot is a ready-made fork point. Once a branch grows long, you should periodically save separate snapshots for each path to avoid rolling back to the common ancestor.

    8.2 Local rerun after an edit

    An agent’s retry often only changes one prompt, one tool output, or clears part of the history. As long as the prefix content changes, the KV and state after it change too:

    • If the edit point is near the tail of the prefix, you can continue from the most recent snapshot;
    • If the edit point is earlier, you must fall back to an earlier snapshot and replay.

    Therefore, the caching strategy should “tolerate change”: prioritize keeping the stable system prompt and early turns, and avoid long-term caching of volatile tool outputs to prevent frequent invalidation.

    8.3 Distributed storage and cross-node sharing

    Write state snapshots into a distributed cache and keep KV pages local to nodes, forming a “shareable state, local KV” combination:

    • When different inference nodes handle different turns of the same session, they fetch the same state from the shared cache, avoiding hit failures caused by routing jitter;
    • A newly scaled-out node can first pull the state of hot prefixes from the shared cache, then pull KV pages on demand, shortening cold start;
    • For very long sessions, KV pages can be archived to object storage while state stays in the hot tier, balancing cost and reuse.

    9. Prefill scheduling: affinity routing, single-flight, and observability

    9.1 Affinity routing and single-flight

    • Affinity routing: apply consistent hashing on the session ID so that adjacent turns of the same session land on the same node, maximizing reuse of local KV and state;
    • Single-flight: when the same prefix is hit by multiple requests simultaneously, let only one request perform replay and prefill while the others wait for its result;
    • Routing-cost control: because state sharing lowers the sensitivity of hits to routing, you can appropriately sacrifice locality in exchange for load balancing.

    9.2 Chunked prefill in continuous batching

    Linear-layer replay depends on state recurrence and is hard to batch well with normal decoding. On the scheduling side, you can:

    • Separate the queues for “prefill containing replay” and “decoding” to avoid mutual blocking;
    • Adopt chunked prefill, splitting long requests into small chunks that alternate with decoding requests to smooth the TTFT tail;
    • Reserve a bandwidth budget for loading and writing back state snapshots, so they do not contend with KV transfer.

    9.3 Key observability metrics

    • Hit rate: track the KV hit rate and the state-snapshot hit rate separately, both counted by token;
    • Replay ratio: replayed tokens ÷ request tokens;
    • Cache health: snapshot age, generation distribution, and invalidation-reason distribution;
    • Transfer and overhead: state cross-node transfer volume and bandwidth, and snapshot write-back latency;
    • Latency metrics: prefill throughput, TTFT at p50/p90/p99, and the actual turn count and success rate of coding tasks.

    On the business side, when deploying on cloud inference platforms such as Baidu AI Cloud, you can host the inference and caching services on a general-purpose container platform, use logging and monitoring/alerting to observe the metrics above, and use object storage for session-level archives. The specific dashboard layout and alert thresholds should be determined based on load-testing results for your workload.

    10. Common pitfalls and a go-live checklist

    10.1 Common pitfalls

    • Caching only KV and ignoring state: the hit rate looks high, but actual prefill savings are limited;
    • Using a model-agnostic content hash as the key: after a model upgrade, old cache is falsely hit and outputs go wrong;
    • Treating state as data that can be “arbitrarily truncated”: a state truncated in the middle cannot be recovered and must fall back to a snapshot;
    • Ignoring numerical-implementation consistency: replaying with a scan implementation different from the snapshot’s can cause the results to drift;
    • Counting only request-level hit rate: long requests carry far more token weight than short ones, so token-weighted metrics should be the primary measure.

    10.2 Go-live checklist

    1. Map out the session structure, identifying stable prefixes (system prompt, repository documentation) and volatile parts (tool outputs);
    2. Build a block-level prefix index and a snapshot directory, with clear model-fingerprint and version management;
    3. Split storage tiers for KV and state, setting eviction and migration strategies for each;
    4. Implement the “snapshot continuation + replay advance” hit path, and validate consistency against full recomputation;
    5. Add affinity routing and single-flight at the scheduling layer, and reserve bandwidth for state transfer;
    6. Roll out in gray-release phases: first observe hit rate and TTFT, then gradually expand the cache scope;
    7. Build observability and regression mechanisms, and fold model upgrades into the cache-invalidation and validation process.

    11. Summary

    Hybrid-architecture LLMs put “addressable KV” and “non-addressable state” into the same model. This brings a smaller cache footprint and faster decoding, but it also makes cache management more complex in multi-turn coding-agent scenarios. The solution is not simply “cache a bit more,” but rather:

    • Treat KV and state as a unified prefix snapshot;
    • Manage hits and invalidation with block-level keys, a snapshot directory, and version generations;
    • Keep state remote and KV local;
    • After a hit, replace full replay with a “continue-and-advance” pipeline.

    Implement these mechanisms solidly, and prefill latency for hybrid-architecture models on long-context coding agents can be significantly improved, shifting the user experience from “waiting for the model to respond” to “continuously collaborating with the agent.”