Rethinking Prefix Caching for Hybrid Architecture Agents: KV Cache and Recurrent State Co-Management
As open-weight LLMs increasingly interleave full-attention layers with linear-attention or state-space layers, prefix caching has become the weak point of hybrid models. KV cache shrinks and lives only in attention layers, but the linear layers carry a fixed-size recurrent state that grows nowhere, cannot be sliced at arbitrary positions, and cannot be reconstructed from the middle. This article explains why coding agent workloads turn prefill into the bottleneck, compares the two "memory" types side by side, breaks down a hybrid state cache design (aligned state snapshots, joint indexing, state blocks under paged management, prefix-tree sharing, tiered caching, and cross-node reuse), interprets reported results of up to 3.14x prefill throughput and up to 88.4% lower p90 TTFT on a single B200-class GPU, and gives a staged rollout path with the metrics to evaluate it.
1. Introduction: Hybrid Architectures Are Popular Now — Cache Management Is the New Bottleneck
The architecture of open-weight large models has shifted noticeably over the past year or two. Rather than making every layer a standard softmax attention block, many models now interleave full-attention layers with linear attention or state-space layers in a fixed ratio. Common building blocks include:
- Gated delta-rule linear attention layers, which maintain a state matrix using a gating factor and a delta-rule-like update.
- Gated state-space layers (SSMs), which keep a selective-scan style state recurrence.
A typical industry ratio inserts one linear layer every 2–8 full-attention layers, and several open-weight models released in the last two years use this “full attention + linear layers” hybrid structure.
The design buys two main benefits:
- A smaller KV cache: the KV cache exists only in full-attention layers, so memory usage drops substantially at the same context length.
- Faster decoding: generating one token requires reading only the KV of the full-attention layers, while the linear layers maintain a fixed-size state — far less memory traffic.
In multi-turn, long-context workloads such as coding agents, KV cache reuse depends on prefix caching. System prompts, repository summaries, tool descriptions, and earlier conversation turns tend to recur across consecutive calls. If an inference service recognizes a shared prefix, it can load the already-computed result and skip prefilling that part. For pure Transformer models, this mechanism is mature.
For hybrid models, however, an inference service must reuse not only the KV cache but also the recurrent state of the linear layers — and the two kinds of data have very different lifecycles and access patterns. Prefix cache management for hybrid models is therefore considerably more complex than for pure Transformers, and the prefill stage feels it first.
A research team has addressed exactly this problem by redesigning KV and state cache management for hybrid architecture models, calling the result a hybrid state cache scheme. Replaying real coding-tool usage traces on a single B200-class GPU, its prefill throughput reached up to 3.14x that of mainstream inference engines, and p90 time-to-first-token (TTFT) dropped by up to 88.4%. The gains remain significant with a 512 GiB CPU cache attached, and the scheme supports distributed storage plus cross-node hybrid state reuse.
This article analyzes the root of the bottleneck, unpacks the design and the reusable lessons, and outlines an adoption path with evaluation methods.
2. Two Kinds of “Memory”: How KV Cache and Recurrent State Differ
2.1 KV Cache: A Discrete Cache That Grows with the Sequence
In full-attention layers, every token produces a set of key/value vectors. Because inference computes attention between all old and new tokens pairwise, each token’s KV must be retained. Its size can be estimated as:
KV Cache bytes ≈ 2 × num_full_attention_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
The KV cache has three defining characteristics:
- It grows linearly with sequence length. Longer context means a larger footprint — the main memory pressure in long-context inference.
- It is naturally splittable by position. The KV of tokens 1..N depends only on those N tokens and nothing after them. Given the same prefix, this part is identical and can be reused exactly.
- It grows by appending one token at a time. Each additional token writes one more KV chunk, so it can be managed in fixed-size paged blocks, indexed by block hash, with the hit length determined by prefix matching.
Precisely because of these properties, inference engines have built block-based paged caching, prefix-tree indexes, block-level hashing, and LRU eviction — letting multi-turn conversations and multiple requests sharing a system prompt reuse computed results effectively.
2.2 Linear Layers: A Fixed-Size Accumulated State
Linear layers do not look at individual tokens; they progressively update one fixed-size state along the sequence. Taking a simplified single-layer F.linear as an example, the forward pass can be abstracted as:
# S: state matrix of shape [d_k, d_v]; k_t, v_t: key/value vectors of the current token# decay_t: gating decay factor; beta_t: write strength of the delta rule; r_t: optional normalization term# out_t = q_t @ S (illustrative, grouped heads and output projection omitted)S = S * decay_t + (beta_t * (v_t - k_t @ S)).unsqueeze(-1) * k_t.unsqueeze(-2)out_t = q_t @ S
The first line is the recurrence at its core: the old state is decayed as a whole by the gating factor, then the current token’s key/value is written in — a soft “forget plus write” update over history. State-space layers follow a similar form:
h_t = A · h_{t-1} + B · x_t, y_t = C · h_t
Here A controls how much history is retained, B writes in the current input, and C reads the output. All such layers use a fixed-size state that summarizes the entire history from the start of the sequence to the current position. This yields three key properties:
- Fixed length: the state size depends only on model dimensions (e.g., key dim × value dim). Whether the context is 1K or 128K tokens, each layer’s state footprint is the same.
- Not splittable at arbitrary positions: we can only obtain the state after reading token
t. Reusing the state at tokenkrequires that a snapshot at stepkwas saved; unlike KV, it cannot be picked up on the fly at any block boundary. - Not reconstructible from the middle: the state is accumulated step by step. Once a snapshot is lost, recurrence must restart from the last available snapshot — or from the beginning of the sequence. There is no “fill in the missing block” equivalent.
These properties mean recurrent state can only be cached and reused as snapshots. In principle you could store one at every chunk boundary, but the number of snapshots grows with sequence length, and the storage and transfer cost is not negligible — so a sensible sampling granularity must be chosen.
2.3 Side-by-Side Comparison
| Dimension | KV Cache | Linear-layer recurrent state |
|---|---|---|
| Size | Grows linearly with sequence length | Determined by model dimensions, independent of sequence length |
| Structure | Splittable into fixed blocks, pageable | A single state object as a whole |
| Snapshotting | Every block is inherently a snapshot | Must be saved deliberately at chunk boundaries |
| Reuse granularity | Any block boundary | Any saved snapshot point |
| Indexing | Block hash | Hash over prefix content |
| Recovery when missing | Local recomputation from the missing block | Full recomputation from the last snapshot or the sequence start |
| Multi-branch sharing | KV blocks shared at prefix tree nodes | One snapshot usable as the start point for several branches |
In short, the KV cache solves “can we store it and find it,” while recurrent state raises the harder questions of “where to store it, when to store it, and whether a stored snapshot can actually be aligned and reused.” Prefix caching for hybrid models is essentially about unifying two mechanisms with different rules under a single index and scheduling scheme.
3. Why Coding Agent Workloads Make Prefill the Bottleneck
3.1 Conversation Structure Determines Reuse Patterns
In a coding agent, each new user request typically sends the agent through a “think → call tool → observe result → think again” loop. Tools include searching code, reading files, and running tests. Context only grows, and multiple sessions may share the same system prompt or project background. As a tree, the shared portion of the requests is a path from the root to some node:
[System prompt + tool descriptions + project background] <- shared across sessions├── Session A: turn 1 → turn 2 → turn 3 → turn 4 <- turns appended within one session│ └── retry after editing code <- branch└── Session B: turn 1 → turn 2
This structure is very friendly to prefix caching: high duplication, long shared prefixes. But for hybrid models, if the linear-layer state is not reused, every request must still feed the entire KV-hit prefix back through the linear layers and run the token-by-token recurrence again. The result: the KV is reused, the state is not, and the most compute-heavy part is not actually saved.
3.2 Typical Workload Characteristics
Coding agent workloads usually have three characteristics:
- Long system prompts and tool definitions: often thousands to tens of thousands of tokens, repeated in every request.
- Context grows monotonically with the conversation: long sessions can reach hundreds of thousands of tokens, while each turn’s new content is a small fraction of the whole.
- Requests branch and retry: one session may spawn multiple branches that continue from different turns.
In the second and third cases, if cache granularity does not match the request structure, the actually reusable prefix shrinks noticeably. As sessions grow, the portion that has to be prefilled again gets larger each time.
3.3 Where the Cache Complexity Comes From
Combining with Section 2, the bottleneck reduces to four points:
- Mismatched matching granularity: KV block hashing can locate a fairly precise prefix position, but linear-layer state exists only as sparse snapshots, so recurrence must continue from the nearest one.
- Mismatched cache lifetimes: KV blocks are evicted per block with clear reference counts; state snapshots have a fixed size but are read repeatedly under high request rates, so eviction and retention policies need to be redesigned.
- Mismatched capabilities across tiers: at the CPU and distributed storage tiers, KV blocks suit batched token transfer, whereas state snapshots require “complete prefix, aligned boundary.”
- Mismatched scheduling and offload policies: chunked prefill, request preemption, and similar mechanisms were designed around KV; saving and restoring state snapshots needs an additional path.
So prefill performance on hybrid models is usually limited not by the compute capacity of the linear layers, but by a “match → hit → load” pipeline for prefix caching that fails to coordinate the two cache types.
4. Design Breakdown: How a Hybrid State Cache Reorganizes Prefill
Schemes of this kind can be summarized in four parts: snapshot alignment, joint indexing, tiered caching, and cross-node reuse.
4.1 State Snapshots: Where to Store, How Densely
Snapshot design has to answer two questions:
- Where to store: only at chunk boundaries, which naturally aligns with the linear layers’ blocked computation and requires no extra synchronization inside a layer.
- How densely: at one extreme, saving at every step maximizes the hit rate but makes storage and transfer cost grow linearly with the sequence; at the other, keeping a single snapshot at the end of the sequence minimizes cost but makes intermediate prefixes unreusable. The practical approach is usually a hybrid: save snapshots at a fixed interval along the sequence, so that any position within a shared prefix has a nearby snapshot to reuse.
4.2 Joint Indexing: Reuse Only Where Both Caches Can Be Reused
Given a batch of input tokens, the system does two things:
- Hash the KV blocks to determine “how far the KV can be reused.”
- Use a hash over prefix content to look up the nearest state snapshot, determining “how far the state can be reused.”
The final reused prefix length is the smaller of the two, and the remainder is prefilled normally. Illustrative code:
# Illustrative pseudocode; does not represent any specific implementationdef resolve_prefix(tokens, block_size, kv_index, state_index):# 1) Match KV blocks progressively to get the number of fully reusable blockskv_blocks, hashes = 0, []for start in range(0, len(tokens), block_size):h = hash_block(tokens[start:start + block_size], prefix=hashes[-1] if hashes else None)if h not in kv_index:breakhashes.append(h)kv_blocks += 1kv_hit = kv_blocks * block_size# 2) Look up the nearest state snapshot within that prefix rangestate_hit = state_index.lookup_nearest(tokens, upper_bound=kv_hit) # illustrative API# 3) Take the commonly reusable position; prefill the resthit = min(kv_hit, state_hit)return hit, tokens[hit:]
The value of a joint index is that it collapses “how far the KV can be reused” and “how far the state can be reused” into a single decision. The inference service does not have to run a full match twice, and it never loads the KV while silently falling back to an earlier state point.
4.2.1 State Blocks: Paged Management with Snapshots as Nodes
To let state snapshots be managed like KV blocks, the snapshot covering every N tokens can be treated as a state block:
- State blocks are addressed by a content hash of the prefix they belong to, coexisting in the same prefix index as KV block hashes.
- Each block holds the state tensors of all linear layers at that position, and uses the same reference counting and LRU eviction rules as KV blocks.
- On eviction or preemption, references to both cache types change together, avoiding dangling references where “the KV block is still there but the state block has already been reclaimed.”
With this, hybrid prefix caching goes from “a KV cache plus a temporary state” to a uniformly managed two-level paged structure — and the storage layer only has to deal with one addressing scheme.
4.2.2 Merging and Deduplicating the State Tree
In the tree-shaped sessions of a coding agent, multiple branches share the same prefix. Storing a separate copy of the state snapshot for each branch both wastes storage and, because eviction order differs, produces situations where “the snapshot exists but the KV does not.” The safer approach is to hang KV blocks and state blocks on the same prefix tree nodes, keeping a single copy along shared paths; a new branch continues the recurrence from the nearest snapshot and then creates new snapshot nodes for its added content.
4.3 Tiered Caching: Keeping Hot Snapshots Closer to the Compute
Session-level prefixes are often sizable. Take a 4K-token system prompt: with a KV block size of 128, a session has about 32 KV blocks; saving a state snapshot every 512 tokens gives about 8 state blocks. When GPU memory is tight, swapping out all of these hot prefixes to CPU means every later hit still pays a cross-medium transfer, and TTFT becomes hard to keep stable.
The sensible approach is to place KV blocks and state blocks together and manage them in tiers under one policy:
- GPU memory: the active prefix of hot sessions, plus the most recent snapshot of the sequence currently being prefilled.
- CPU memory: session-level shared prefixes and secondary hot spots.
- Distributed cache: prefixes shared across nodes, serving the whole cache pool.
Once tiered, a hit is decided jointly by the tiers, and total cache capacity is no longer bounded by one machine’s GPU memory — it extends to tens or hundreds of GiB of CPU memory and cluster-level storage.
4.4 Cross-Node Reuse: Small State Blocks Are Actually Easier to Share
In distributed deployments, later requests in the same session may be routed to different instances. If each instance recomputes the prefix from scratch, a large share of cluster compute goes to duplicated prefilling.
Hybrid models have a natural advantage here: with fewer full-attention layers, KV blocks are smaller, and with fixed-length state blocks, the total data volume of a long-prefix snapshot is far smaller than the KV cache of a same-length pure Transformer model. Replicating it to several nodes, or moving it between nodes on demand, is correspondingly more tractable. Three engineering pieces are needed:
- Shared cache pool: multiple inference instances attach to the same distributed cache, addressing by content hash and deduplicating by reference count.
- Request affinity: the routing layer directs requests from the same session to instances that already hold the corresponding prefix, reducing cross-node transfer.
- Asynchronous prefetch: when hitting a remote cache, overlap computation and I/O in a pipeline so loading stays off the critical path.
5. Reading the Measured Results
5.1 Evaluation Setup
- Hardware: a single B200-class GPU.
- Workload: replay of real coding-tool usage traces, which is closer to production turn distributions and context growth patterns than synthetic workloads.
- Baseline: the prefix caching implementation of mainstream inference engines under equivalent conditions.
- Core metrics: prefill throughput (effective prefilled tokens processed per unit time) and p90 TTFT (the upper bound of first-token latency for 90% of requests).
5.2 Results and Where They Come From
In testing, the scheme reached up to 3.14x the baseline’s prefill throughput and reduced p90 TTFT by up to 88.4%; adding a 512 GiB CPU cache kept the gains significant. Given the mechanisms above, the improvement comes mainly from four sources:
- Eliminating repeated recurrence: the KV-hit portion no longer has to be recomputed by the linear layers, saving a great deal of duplicate work each turn in long sessions.
- Stabilizing the latency tail: under high concurrency, requests no longer queue behind a “recompute the whole prefix” stall, so p90 TTFT drops noticeably.
- Widening the reusable range: tiered caching lets session-level prefixes stay resident, raising the hit rate and keeping load time controllable after a hit.
- Lowering cross-node duplication cost: prefixes are reusable across nodes, so cluster-wide throughput rises.
5.3 Applicability Boundaries
These numbers deserve a measured reading:
- They come from a comparison on specific hardware, with a specific workload and a specific baseline; they do not mean every scenario sees equivalent gains.
- Workloads with low session sharing, where requests barely reuse prefixes (for example, many unrelated one-off short requests), have limited room for optimization to begin with.
- Snapshot granularity, cache capacity, and routing policy all affect the final result and need tuning against real traffic.
6. Adoption Path and Evaluation Methods
6.1 Incremental Rollout
Budget the memory: using the context-length distribution and session count, estimate KV and state snapshot footprints separately. Illustrative formulas (per state block, stored as bfloat16, measured as key dim × value dim):
KV total ≈ KV bytes per token × average active context length × concurrent sessionsState total ≈ key_dim × value_dim × 2 bytes × snapshots per session × concurrent sessions# Example: key_dim = value_dim = 128, snapshot interval 512 tokens# one state block = 128 × 128 × 2 ≈ 32 KiB# a session with 128K context ≈ 131072 / 512 × 32 KiB = 8 MiB
State snapshots are largely independent of context length, so they add up linearly with session count and belong in capacity planning.
Enable the joint prefix cache: start with state snapshots and joint indexing so that KV and state are reused at the same boundary.
- Introduce tiered caching: based on the memory budget, place session-level prefixes in CPU memory or the distributed cache. Cloud deployments can build on general cloud infrastructure — object storage for persisting low-frequency prefixes, and logging, monitoring, and alerting services to track hit rates, latency, and cache capacity. For example, Baidu AI Cloud’s elastic compute, object storage, log service, and monitoring/alerting capabilities can be combined to build an elastic inference and cache cluster, with a unified monitoring system to observe runtime behavior continuously.
- Optimize routing: session affinity and asynchronous prefetch are advanced optimizations to enable gradually once the hit rate is stable.
6.2 Metrics to Track
| Metric | Meaning | What to watch |
|---|---|---|
| KV block hit rate | Share of KV blocks served from cache | Reflects how much sessions share |
| State block hit rate | Share of state blocks served from cache | Reflects whether snapshot granularity is sensible |
| Average reused prefix length | Mean tokens reused per request | Measures the prefill actually saved |
| Prefill throughput (including cache hits) | Effective prefilled tokens completed per unit time | Overall benefit indicator |
| p50 / p90 TTFT | Median and 90th-percentile first-token latency | Tail experience |
| Cache footprint and eviction rate | Occupancy and swap-out frequency at the GPU / CPU / distributed tiers | Basis for capacity and policy tuning |
6.3 Common Pitfalls
- Watching hit rate but not reused length: a few long prefix hits can deliver far more benefit than many short ones.
- Evicting state snapshots as if they were KV blocks: a missing state block cannot be patched locally, so it must be evicted and retained in lockstep with its corresponding KV blocks.
- Ignoring branch structure: when sessions branch and retry, snapshots should be shared through the prefix tree rather than saved independently per request.
- Treating cross-node transfer as free: hitting a remote cache still costs network time; without session affinity and prefetching, tail latency can get worse rather than better.
7. Conclusion
Hybrid architectures are becoming a serious option for long-context and agent workloads, but the architectural “savings” only translate into real throughput and latency gains when cache management keeps up. The fixed length, non-splittability, and inability to reconstruct from the middle make the linear layers’ recurrent state fundamentally unsuitable for the KV-style “reuse by block, exactly” logic.
The core idea behind this scheme is to discretize continuous recurrence into addressable cache objects via state snapshots, to reuse KV and state at the same boundary via a joint index, and to extend reuse from a single request to a whole session and even a whole cluster through tiered caching and cross-node sharing. That is the root cause of the roughly 3x prefill throughput improvement and the sharp p90 TTFT reduction observed on real coding agent workloads.
For teams evaluating hybrid architecture models, the recommendation is to include “is a hybrid state cache reusable, how fine is the snapshot granularity, and can it be shared across tiers and nodes” in model selection alongside accuracy and context length — and to keep validating the gains against real business traces.
