KV and State Cache Optimization for Hybrid-Architecture LLMs
Hybrid large language models interleave full-attention layers with linear attention or state-space layers. This article explains why agent workloads make prefix reuse harder, and how a unified paged cache for KV blocks and recurrent state can cut redundant prefill, overlap transfer with compute, and share state across storage tiers and nodes.
1. Background: Hybrid Architectures Are Becoming Mainstream
In recent years, attention designs in large language models have diverged. Alongside conventional full-attention layers, more open-source models now use linear attention and state-space layers, including variants such as gated delta attention, Gated DeltaNet, and Mamba-2. A typical hybrid stack interleaves full-attention layers with linear layers:
- Full-attention layers provide precise long-range retrieval, but their KV cache grows linearly with context length.
- Linear attention and state-space layers compress history into a fixed-size state matrix. At inference time, that state does not grow with sequence length, so decoding throughput is higher and memory use is lower.
Combining full attention with linear layers trades a small loss in modeling quality for faster decoding and a smaller cache footprint. A common pattern places one full-attention layer every four to six layers and uses linear layers for the rest. The benefit is especially clear in long-context settings.
2. The Problem: Agent Workloads Complicate Prefix Cache Management
In conventional chat, multi-turn prefix reuse is straightforward. If the new request shares a prefix with what is already cached, the full-attention KV cache can be reused and the linear-layer state can be continued.
Coding-agent workloads break that assumption:
- Context is a mix of appends and edits. Each turn appends new material such as tool results and code-execution output, while system prompts and earlier conversation often stay the same. The prefix is not strictly append-only.
- Prefix checks are finer-grained. A hybrid model must validate two kinds of cache: paged KV blocks for full-attention layers, and state blocks for linear layers. Their alignment boundaries and invalidation rules differ. A state block can be reused only as a whole segment for a given layer; it cannot be sliced at token granularity the way a KV block can.
- State cannot be resumed from an arbitrary position. Linear-layer state is the result of compressing the history through recurrence. If tokens change in the middle of a prefix, every state after that segment must be recomputed. That cost is much higher than partial recomputation in a pure Transformer.
- Concurrent turns increase cache contention. An agent session often has several parallel tool-call branches. Eviction of state blocks and KV blocks must be scheduled together. Frequent swap-in and swap-out can erase the throughput advantage of the hybrid design.
Without a redesigned cache layer, much of the hybrid-architecture gain disappears under agent load, and prefill often falls back to large amounts of repeated computation.
3. Approach: A Unified State Cache Manager
A recent direction in the field is to place hybrid KV caches and state caches under one paged management and scheduling framework. The core design has four parts.
3.1 Paged State Blocks
Recurrence in linear layers is split into fixed-length segments. The state produced by each segment is stored as its own state block. State blocks and KV blocks share one block-table index. On arrival, a request first matches its prefix. Hits are reused; the miss is computed by continuing recurrence from the last matched segment boundary. The prefill flow looks like this:
# Pseudocode: prefill for a hybrid architecturematched = prefix_cache.match(request.tokens) # matched KV blocks and state blocksif matched.state_block is not None:# Resume linear recurrence from the state breakpoint; compute only new segmentsstate = load_state(matched.state_block)state = linear_scan(state, request.tokens[matched.offset:])else:state = linear_scan(zero_state(), request.tokens)for layer in attention_layers:kv = prefix_cache.load_kv(layer, matched.blocks) # load KV blocks on demandattn_out = attention(kv, request.tokens[matched.offset:])
3.2 Overlapping Compute and Transfer
A state block is much smaller than a KV cache of the same length, but it still has to be loaded from GPU memory or host memory. The key optimization is to pipeline state loads, KV prefetch, and GPU compute. Before prefill starts, the scheduler asynchronously prefetches prefix blocks so transfer is hidden behind computation and the GPU does not stall waiting for data.
3.3 Multi-Tier Storage and Distributed Reuse
A single machine has limited GPU memory, so state caches fit a tiered layout: hot data stays in GPU memory, warm data moves to a large CPU-side cache, and cold data goes to distributed object storage or a shared memory pool. Because a state block is self-contained—a given segment input maps to a determined state output—it can be reused across nodes. When several inference instances serve the same set of agent sessions, a state block computed on one node can be referenced directly by the others. That is a basis for deduplicating prefill across a large cluster.
3.4 Invalidation and Eviction
A unified block table simplifies invalidation: if the token-content hash of a segment matches, the corresponding state block is valid. Eviction is weighted by last-access time plus recomputation cost, so deeper state blocks that are expensive to recompute are kept first.
4. Results and Practical Guidance
Public evaluations of real multi-turn agent traces report that a hybrid inference system with unified state-cache management can reach more than 3× the prefill throughput of conventional open-source inference frameworks, and can cut 90th-percentile time to first token (TTFT) by up to nearly 90%. Gains remain significant even with a CPU-side cache pool on the order of hundreds of GiB. The same idea applies to other long-context agent settings, such as retrieval-augmented generation and batch analysis of long documents.
When putting this into practice, pay attention to the following:
- Segment length. Segment size sets the tradeoff between reuse precision and recurrence cost. Tune it against the typical length of context added in one agent turn.
- Hit-rate monitoring. Track KV hit rate and state hit rate separately. The first reflects prefix management; the second reflects state scheduling. Analyze them together.
- Cloud-native integration. Object storage can hold cold state blocks. Monitoring and alerting can track cache waterline and eviction rate. On a container platform, scale inference instances with load.
- Numerical consistency. When state blocks are reused across nodes, every node must use the same precision settings, including numeric format and accumulation order. Otherwise state drift can make outputs non-reproducible.
5. Conclusion
The value of a hybrid architecture is not only in the model. It also depends on the inference system around it. Treating linear-layer state as a first-class cache citizen—unified paging with KV, multi-tier storage, and cross-node reuse—is what turns the theoretical advantage under agent load into real throughput and latency gains. As agent applications demand longer context and more multi-turn interaction, inference-infrastructure work aimed at this “state era” will be a key path to lower cost and higher efficiency for large-model serving.
