Prefix Caching for Hybrid LLM Architectures: A New Design Paradigm
Hybrid LLM architectures that combine full attention with linear or state-space layers can reduce cache size and speed up decoding, but they move complexity into prefill and prefix caching. This article explains why token-block KV caching alone is not enough when recurrent states are present, and proposes a prefix-tree design where cache recovery points are selected based on reuse potential rather than block boundaries. It also covers runtime restore strategies, coarse checkpoints, tiered caching, session-aware state sharing, serialization, scheduling, and evaluation.
Introduction: After Hybrid Architectures Become Popular, Caching Becomes the New Bottleneck
A class of hybrid architectures has become increasingly common in large language model design. The basic idea is to combine two types of layers inside one model:
- A small number of full attention layers, which preserve exact access to historical information as context length grows.
- A larger number of linear layers, including linear attention and state space models (SSMs). Some designs also introduce gating or delta-rule mechanisms.
This “full attention + linear layers” combination is already visible in mainstream open-source models. A common configuration keeps full attention at a few critical positions and uses gated linear attention or state-space layers elsewhere. Some designs periodically alternate the two layer types at a fixed ratio. The benefit is a smaller cache footprint and faster decoding.
Linear layers maintain a compressed state that is independent of sequence length. They can pack historical information into a fixed-size recurrent state. Full attention layers then decide when exact historical access is required.
This structural change is friendly to the decoding stage, but it shifts complexity to prefilling and prefix caching. Coding agents, meaning agents driven by AI coding tools, typically produce workloads with several distinctive characteristics:
- Many conversation turns, where context keeps growing as tool-call results, file contents, and command outputs are appended.
- Many requests share long common prefixes, such as system prompts, project structure descriptions, and code retrieval results.
- Sessions often branch: retries, parallel subtasks, or re-execution after a plan is modified.
Under these workloads, whether prefix caching works efficiently directly determines time to the first token and the overall throughput of the serving cluster.
Prefix caching in existing inference frameworks is mainly designed for pure Transformer models. Its core mechanism is KV-cache reuse at token-block granularity. When a model contains recurrent-state layers, this mechanism exposes a series of new problems. This article starts from the fundamental mismatch between cache granularity and recovery location, and systematically reviews design principles and engineering considerations for prefix caching in hybrid architectures.
Background: The Fundamental Difference Between Two Types of Cache in Hybrid Architectures
Prefix caching in standard Transformers
In a pure Transformer decoder, every attention layer must compute over the Key/Value tensors of all historical tokens. Inference frameworks usually organize these KV tensors into fixed-size token blocks and compute a hash for each block based on its contents.
The hash serves two purposes. First, it acts as the cache key. Second, it links blocks into a chain, ensuring that only a completely matching contiguous prefix can be reused.
This design relies on two assumptions:
- KV is addressable: for any token position, the KV values at that position and before it can be retrieved precisely.
- Prefixes can be stitched together: if two requests share the same leading tokens, the later request can skip prefill computation for that shared prefix.
Because the key is a content hash, the way blocks are divided has no essential coupling with reuse capability. Block size mainly affects the tradeoff between granularity and overhead.
This mechanism is mature and efficient, but it assumes that everything that needs to be cached can be aligned to token blocks.
Recurrent state layers: fixed-size, order-dependent global summaries
The conflict between linear layers and prefix caching comes from how linear layers compute. A simplified update of a common linear-attention recurrence can be abstracted as:
S_t = a_t * S_{t-1} + b_t * (k_t v_t^T) # recurrent state updateo_t = S_t^T * q_t # read state using current query
S_tis a fixed-size matrix, usuallyd_k × d_vper head.a_tis a decay or gating factor, determined by the input, the previous state, or a fixed decay.k_t,v_t, andq_tare the current key, value, and query projections.
Different model families differ substantially in details. Some layers use diagonal decay. Others use structured state transitions. Local mixing layers with convolution kernels may also need to maintain a convolution window.
Even so, all linear layers share two key properties:
- Length-independent space: whether processing 1K tokens or 1M tokens, the size of a single state remains constant.
- Irreversible sequential dependency: the state at time
tis accumulated from all history. There is no intermediate position that can be addressed as an independent segment.
It is useful to think of the state as a global summary that is continually overwritten. To obtain the state for prefix [1, N], there are only two options: compute sequentially from the beginning, or continue from an earlier snapshot.
The core conflict: KV is organized by blocks, while states are organized by prefixes
Putting the two together reveals a structural mismatch:
- KV is a set of addressable segments. It can be cached, evicted, and shared across requests by block.
- A state is a non-addressable global summary. Each state implicitly contains the complete computation history from the beginning of the sequence to that point.
This means that if two requests diverge at token N, their states at that position are exactly the same, regardless of which blocks came before. In other words, the natural reuse granularity for a state is the divergence point itself, not an artificially defined block boundary.
In traditional implementations, however, states are usually cached at block boundaries, or not cached at all and are recomputed on every request.
Another important property is that recurrent states are lossy compressed summaries. As decay factors continue to act, the contribution of older tokens gradually weakens. States produced from different histories become numerically closer over time.
This creates an important design space: a state does not necessarily have to be reused at an exact boundary; an approximate state is often acceptable. The impact of approximation on task quality must be confirmed through evaluation. This also complements the latency sensitivity of agent workloads.
Core View: Recovery Position Should Be Determined by Where Reuse Is Possible
Traditional block-based caching implicitly assumes that all cached tensors can be indexed by the same block hash. This assumption holds for pure Transformers, but not for hybrid architectures.
A more accurate design principle is:
The recovery position should be determined by where reuse is possible, not by how KV is divided into blocks.
This principle has three implications.
Reuse is determined by prefix structure, not block boundaries
In coding-agent sessions, cache hits are determined by the common prefix shared across requests: the same system prompt, the same project background, or the same file content already read.
This information may be organized semantically at the message level and may have no relation to token-block boundaries. If state recovery is forced to align with blocks, two kinds of waste can occur:
- Downward alignment: if the common-prefix length
Lis not an integer multiple of block sizeB, recovery can only fall back to⌊L/B⌋ × B, recomputing up toB-1extra tokens. Larger blocks make this waste more severe. - No state caching: if states are not cached, every request must recompute from scratch, or only the most recent request’s state is kept, which prevents cross-request reuse.
State cost and benefit should guide the recovery strategy
Choosing a recovery position is essentially a tradeoff between cost and benefit:
- Cost = the computation required to advance from a snapshot to the divergence point.
- Benefit = the prefix computation that is reused.
A naive approach is “as far back as possible”, always selecting the best matching position. However, because recurrent states are compressed, when decay is strong, cache capacity is limited, or hit rate is insufficient, a newer and closer approximate state may be more cost-effective than a very old and distant exact state. The latter requires preserving longer history and has higher memory and transfer overhead.
Therefore, the recovery position should be treated as an optimizable decision variable, rather than being fixed in advance by blocking rules.
KV and state should be managed separately
KV caches and recurrent states differ greatly in size, lifecycle, shareability, and approximation tolerance. They should use different indexing, eviction, and storage strategies.
| Dimension | Full-attention KV | Linear-layer state |
|---|---|---|
| Size | Grows linearly with context length | Independent of length, constant |
| Granularity | Can be split and addressed by block | Can only be represented as a whole up to some prefix |
| Sharing | Shared only when the prefix matches exactly | Shared at the divergence point |
| Approximation tolerance | Low; usually exact caching | Higher; lossy compression and decay allow approximation |
| Eviction granularity | Block-level LRU or reference counting | Session-tree node level |
Decoupling does not mean isolation. Ideally, both should share the same prefix/session tree: KV blocks are attached along tree paths, while states are attached to tree nodes. Each node naturally represents a historical endpoint. Traversing the tree simultaneously determines the recovery position for both KV and states.
Key Design: From Block Hashing to Prefix Trees
Based on the principles above, cache management for hybrid architectures can be reorganized at four levels. This describes a general design approach and does not represent any specific implementation.
A prefix tree centered on reuse points
Traditional prefix caching computes a block hash from the parent block hash plus the current block’s tokens, forming an implicit prefix chain. Hybrid architectures are better served by an explicit prefix tree or radix tree:
- Each node represents a continuous shared token interval. Leaf nodes correspond to the end of a specific request.
- Each node stores the list of KV blocks for that interval and a state snapshot representing the state after reaching that node.
When a request arrives, traversing the tree to find the longest matching path simultaneously provides reusable KV and a recoverable state. The recovery position naturally falls at the divergence point.
Multiple recovery strategies can coexist
Not every request needs exact recovery from the divergence point. At runtime, the system can dynamically choose based on the estimated cost of candidate recovery points:
- Exact recovery: recover from the snapshot at the divergence point, with no extra computation. This is suitable when the shared prefix is long and the state is already resident.
- Approximate neighbor recovery: use a snapshot from before the divergence point, such as the most recent coarse checkpoint, and recompute a few tokens. When the shared prefix is short, recomputing a small number of tokens may be far cheaper than loading a large, distant state.
- Zero recovery: when the system prompt is extremely short and almost nothing is shared, compute directly from the beginning.
- Cache-friendly recovery: choose the insertion order among multiple states to avoid frequent swapping in and out.
Illustrative pseudocode:
def plan_restore(session_tree, req_tokens, cfg):node = session_tree.longest_prefix(req_tokens) # find the divergence pointif node is None or node.shared_len < cfg.min_share:return RestorePlan(mode="zero", start=0)# Enumerate candidate snapshots: the divergence point plus coarser prior checkpointscandidates = [node.snapshot] + session_tree.coarse_snapshots(node)best = Nonebest_cost = float("inf")for snap in candidates:if snap.resident_level > cfg.max_level:continue # e.g., snapshots on remote storage are excluded from exact restore by defaultcost = (snap.restore_cost # load/deserialization cost+ (node.shared_len - snap.position) * cfg.unit_prefill_cost+ snap.layers * cfg.layer_switch_cost)if cost < best_cost:best_cost = costmode = "exact" if snap.position == node.shared_len else "approx"best = RestorePlan(mode=mode, start=snap.position, snapshot=snap)return best or RestorePlan(mode="coarse", start=node.coarse_pos)
The key point is that the recovery plan is computed at runtime based on cache residency, rather than being hard-coded by block size or hashing rules.
Coarse checkpoints and tiered caching
A useful property of recurrent states is that any prefix position can be made “fresh enough” by recomputing additional tokens.
Therefore, it is practical to place coarse checkpoints periodically in the session tree, for example every few semantic segments or every N conversation turns. These checkpoints can be tiered across GPU memory, host memory, and remote storage:
- GPU memory: stores exact states and hot KV blocks for recently active sessions, serving online requests directly.
- Host memory: stores states and KV for more historical sessions, enabling cross-request reuse.
- Remote storage: stores full snapshots for cold sessions and prefetches them asynchronously when needed.
Because states are small, keeping them as lightweight recovery anchors in GPU memory or host memory, while loading larger KV blocks on demand, is often a cost-effective combination.
Session-topology-aware state sharing
In coding-agent workloads, one session often branches into multiple parallel subtasks. These subtasks share the same prefix but take different subsequent actions.
Useful mechanisms include:
- Copy-on-write: when a session branches, the parent state is shared initially. Each branch copies the state only when it modifies it, avoiding massive redundant computation.
- Structure-aware eviction: prioritize evicting states at branch ends that are unlikely to be accessed soon, while preserving anchors on the shared backbone.
- Distributed reuse: in cross-node scenarios, a session state computed on one node can be reused by another node for recovery, provided there is a globally consistent prefix/session index. For example, if two nodes handle different tasks on the same repository, the shared repository context only needs to be computed once. This requires the state tensor layout and computation implementation to remain consistent.
Engineering Practice: Applying the Design to the Inference Stack
Cache keys and state serialization
- Prefix keys: use cascading hashes based on parent node ID plus token-range content, avoiding repeated hashing of the entire prefix.
- State fingerprints: record model version, tokenizer version, and compute backend for each state. If any of these changes, old states must be invalidated to prevent incorrect cross-version use.
- Storage format: state tensors are usually small three-dimensional floating-point tensors, organized as layer × head × matrix. A contiguous memory layout and fixed data type are preferable. Memory mapping or shared memory can enable zero-copy loading and avoid frequent serialization and deserialization overhead.
Precision safeguards for approximate recovery
Approximate recovery introduces error, so measurable safeguards are needed:
- Set state-error limits for each task type, such as cosine similarity to the exact state.
- Use offline evaluation to observe the impact of approximate prefixes on task metrics, such as code-generation pass rate and tool-call accuracy.
- Continuously monitor recovery quality online, and fall back to exact recovery when necessary.
Scheduling coordination
Prefix caching is not an isolated component. It must be coordinated with request scheduling:
- Cache-aware routing: route requests that hit a known prefix preferentially to nodes holding the corresponding state. Otherwise, cross-node state loading may cancel out the reuse benefit.
- Preemption protection: when a long prefill task is interrupted, preserve state snapshots that are close to completion to avoid recomputing the entire segment.
- Prefetching combined with chunked prefill: for predictable follow-up requests, such as the next tool call in the same session, preload states into GPU memory. For extremely long prefixes, chunked prefill can process tokens in batches and advance the state gradually, reducing peak computation cost per iteration.
- Observability: monitor prefix hit rate, state-restore hit rate, the share of state-loading time in TTFT, and cache memory usage.
Session organization for agents
The upper bound of cache efficiency depends heavily on the prefix structure of the requests themselves. Application-side practices matter:
- Prefix stabilization: place unchanged content such as system prompts, project descriptions, and tool definitions at the front. Append dynamic content as late as possible, avoiding situations where a small early change invalidates the rest of the prefix.
- Append rather than rewrite: update context by appending whenever possible, reducing rewrites of existing prefixes.
- Explicitly mark session structure: record branch relationships and parent-child sessions in session metadata so that the inference framework can construct a prefix tree.
- Structure tool outputs: structure and trim large tool outputs as needed to prevent ineffective tokens from entering the prefix.
Results: Joint Improvement in Latency and Throughput
Public experimental results show that rebuilding prefix caching for hybrid architectures can deliver significant gains on real workloads. The reported tests were conducted on a single next-generation high-end GPU, using session-trace replays from a real coding assistant. The original text notes that these are public results from a research team on an NVIDIA B200, not Baidu AI Cloud test data.
| Metric | Best reported improvement |
|---|---|
| Prefill throughput | Up to about 3.14x versus a mainstream inference framework |
| p90 TTFT, 90th-percentile time to first token | Reduced by up to 88.4% |
| Expanding to 512 GiB of host memory as a second-level cache | Still delivered significant gains |
There is a clear mechanism behind these numbers:
- TTFT is highly sensitive to prefix hits. In coding-agent traces, many requests share prefixes ranging from a few thousand tokens to tens of thousands of tokens. A hit means that portion of computation is skipped entirely.
- Tail-latency improvements are especially pronounced. The fact that p90, rather than the mean, improves significantly suggests that the optimization mainly helps the slowest requests. In coding-agent scenarios, these are often the sessions with the longest context and deepest branching, which are exactly the cases that traditional block-based caching struggles to serve.
- The benefits extend to larger tiered caches. Because states are small and can be restored quickly, host memory and even larger storage tiers can participate efficiently, without requiring all data to occupy expensive GPU memory.
It should be emphasized that the specific multipliers depend closely on the model, block size, session-length distribution, and hardware. They should not be directly extrapolated. What is truly reusable is the underlying design logic.
Conclusion and Outlook
Hybrid architectures provide a path toward long-context inference with less cache and faster decoding. However, they move the challenge from decoding to prefill and cache management.
The core conclusions can be summarized in four points:
- KV is block-addressable, while states are prefix-dependent. Their caching semantics are fundamentally different.
- Recovery position is an optimizable decision. It should be determined by where reuse is possible, not by blocking rules.
- Prefix trees are a natural way to unify the two. With multiple recovery strategies and tiered caching, a prefix tree can serve GPU memory, host memory, and remote storage simultaneously.
- Engineering details determine the final benefit. Key design, serialization formats, session organization, and scheduling coordination are all necessary.
For teams that are deploying or planning to deploy hybrid-architecture models, especially for coding-agent workloads with multi-turn long contexts, a practical starting point is:
- Analyze the prefix structure and reuse potential of incoming requests.
- Perform offline precision evaluation and online A/B testing for candidate recovery strategies.
- Use p90 TTFT and prefill throughput as core benefit metrics, and build a complete monitoring and alerting system.
When deploying or calling model services on the Baidu AI Cloud Qianfan Model Platform, users can access the ERNIE family and mainstream third-party models through OpenAI-compatible interfaces. Prefix-cache management in the inference framework is an internal platform mechanism, maintained and continuously optimized by the platform, and developers do not need to manage implementation details. Regardless of architecture, stable and structured session organization remains the prerequisite for efficient reuse.
Looking ahead, an important direction is a unified abstraction for prefix caching: allowing KV and recurrent states to share the same session/prefix-tree index, while the scheduler uniformly decides where to recover from and how precisely to recover. Only when cache management moves from “block-based transport” to “reuse-driven decision-making” can hybrid-architecture models fully realize their potential on multi-turn, long-context workloads.
