Hybrid LLM Agent Inference: Redesigning State Cache Management for Higher Prefill Throughput
Hybrid architectures combine full attention with linear attention or state space model layers to reduce inference cost, but long-context Agent workloads expose limitations in KV-only caching. This article explains a dual-channel cache design that treats recurrent states as first-class cache objects, enabling safe prefix reuse, distributed state sharing, and significant improvements in prefill throughput and time to first token (TTFT).
1. Background: Why Hybrid Architectures Are Becoming Standard in Open-Source LLMs
Large language model architectures have increasingly diverged in recent years. Pure Transformer architectures with full attention provide strong representational capacity, but as context windows grow, KV Cache memory consumption and the quadratic complexity of attention become major bottlenecks for decoding speed and deployment cost.
To address these limitations, researchers and engineers have developed two types of sequence modeling layers with lower state costs:
- Linear attention variants: By using kernel approximations or gated recurrent formulations, these layers reduce attention complexity from quadratic to linear and compress historical information into a fixed-size state matrix.
- State space model (SSM) layers: These layers model long-range dependencies through selective state recurrences. During inference, each token only requires a fixed-dimensional hidden state.
In practice, replacing every attention layer with a linear layer can reduce fine-grained retrieval and long-range exact copying capabilities. As a result, mainstream systems increasingly use hybrid architectures: most layers are linear attention or SSM layers, with a smaller number of full-attention layers interleaved throughout the network.
This approach preserves the precise global retrieval capabilities of full attention while reducing the cache cost of most layers from growing linearly with sequence length to remaining fixed-size. The result is higher decoding throughput and a substantially smaller KV Cache. Several leading open-source models have adopted this design, making hybrid architectures an increasingly common industry pattern.
2. The Challenge: Agent Workloads Complicate Cache Management
Hybrid architectures perform well for single-turn generation, but programming Agents introduce a different workload profile. Typical characteristics include:
- Multi-turn interaction with shared long prefixes: An Agent repeatedly interacts with its environment. Each request usually begins with a prefix containing the system prompt, conversation history, and code context. This prefix can easily reach tens or even hundreds of thousands of tokens.
- Very high prefix reuse: Between adjacent turns, most of the prefix remains unchanged. Usually, only new observations are appended at the end.
- Prefill as the primary cost: The number of newly added tokens per turn is often small. Without an efficient prefix cache reuse mechanism, however, the server must recompute the entire prefix repeatedly, making prefill the dominant source of time-to-first-token latency and compute cost.
Pure Transformer models already have mature prefix caching approaches. KV Cache is divided into token-level pages, and a prefix tree performs block-level matching. When a block matches, the cached KV values can be reused directly.
Hybrid architectures break this model.
For full-attention layers, the cache remains token-level KV data, so conventional paged caching still applies. The situation is different for linear attention and SSM layers:
- Their states are recursive summaries of an entire sequence. A prefix corresponds to one overall state, which cannot be arbitrarily split into independently reusable token pages like KV data.
- State updates are sequential and recurrent. To obtain the state at position N, the system must scan subsequent tokens sequentially from an existing state. If no state has been cached for a prefix, there is no state from which to resume.
- The alignment between states and KV data is nontrivial. Different layers in a hybrid model use different state representations. Prefix matching must simultaneously preserve token-level KV alignment for full-attention layers and valid segment boundaries for linear or SSM layers. The lifecycles and eviction policies of both cache types must therefore be coordinated.
A generic inference framework may take a simple but inefficient approach: once a prefix is only partially matched, it may abandon linear-layer state reuse and recompute the entire sequence. Under Agent workloads, this can prevent hybrid models from achieving their expected prefill performance.
Redesigning hybrid state caching addresses this core problem.
3. Core Design: Treating Linear-Layer States as First-Class Cache Objects
The design principles of representative work such as FlashStates—which focuses on caching and scheduling for hybrid-architecture inference engines—can be summarized as follows.
3.1 Dual-Channel Caching: KV Pages and State Segments
The prefix cache is divided into two coordinated subsystems:
- KV data for full-attention layers continues to be managed as token pages, supporting fine-grained block matching.
- States for linear and SSM layers are managed as segmented states. During prefill, the engine periodically checkpoints the recurrent state. Each checkpoint stores the state tensor at the end of a segment together with the corresponding full-attention KV position. Together, they form an atomic reusable unit.
3.2 Segmented Prefix Matching
Traditional prefix trees match token blocks. Hybrid architectures instead need to return the longest prefix length L that can be safely reused, subject to two conditions:
- At the KV level, all cache pages from position 0 through
Lmust be available. - At the state level, a checkpoint must exist whose ending position is exactly
L.
Only when both conditions are satisfied can prefill start directly from position L. Linear layers resume from the checkpoint state and scan only the new tokens after L, while full-attention layers load the cached KV data directly.
Illustrative pseudocode:
def match_hybrid_prefix(prompt_tokens):# 1. Use the KV page tree to obtain candidate match lengthskv_hits = kv_page_tree.longest_match(prompt_tokens)# 2. Find a candidate position with an exact state checkpointfor L in sorted(kv_hits, reverse=True):if state_store.has_checkpoint(layer="linear", pos=L):return L, state_store.get(layer="linear", pos=L)return 0, None # No reusable prefix; recompute the entire sequence
3.3 Computing and Persisting State Checkpoints
During prefill, linear layers already update their recurrent states token by token. Creating a checkpoint is therefore nearly free from a computation perspective: the system only needs to save an additional copy of the state tensor at selected positions.
The main trade-off is checkpoint density. Excessively frequent checkpoints increase storage and loading overhead, while sparse checkpoints reduce the probability of a future match. In practice, checkpoint placement should reflect the append patterns of the Agent workload. Boundaries with a high probability of becoming branching points—such as the point after a tool response—are often more valuable than uniformly spaced checkpoints.
3.4 Distributed and Cross-Node Reuse
In Agent workloads, the same session may be scheduled on different nodes. State caching must therefore support distributed reuse:
- State tensors can be quantized or compressed before being written to a remote cache layer, such as a tiered cache backed by a CPU memory pool. Object-storage-style capabilities may also be used for colder backup copies.
- When routing a request, the system first checks the target node’s local state cache. On a miss, it retrieves the state from the remote layer. This transfer is typically much cheaper than recomputing prefill for tens of thousands of tokens.
- Cache keys are composed of a hash of the session prefix and the checkpoint position, ensuring consistency across nodes.
4. Results and Engineering Validation
In an evaluation designed to approximate real-world workloads—using B-class data-center GPUs, replaying multi-turn programming Agent traces, and comparing the redesigned cache with a general-purpose open-source inference framework—the hybrid state cache produced substantial gains:
- Up to approximately 3× higher prefill throughput: Linear layers can resume directly from checkpoints instead of recomputing the entire recurrent prefix.
- Up to nearly 90% lower p90 time to first token: When long prefixes are reused, prefill computation is reduced from the full-prefix cost to the cost of processing only the incremental portion.
- Benefits remain after integrating a CPU tiered cache at the scale of hundreds of GiB: The design also supports distributed storage and cross-node state reuse, making it suitable for large Agent clusters.
5. Practical Recommendations
Teams building their own inference services can apply the following engineering principles:
- For hybrid models serving Agent workloads, verify that the inference framework supports persistence and reuse of linear-layer states. KV page caching alone is not sufficient.
- Place checkpoints at application-level branching points, such as tool responses and user-message boundaries, rather than relying only on uniform intervals. This can significantly improve the cache hit rate.
- Bind the prefix hash to the checkpoint position when designing cache keys. This prevents state and KV data from becoming misaligned and producing inconsistent results.
- Quantize and compress state tensors before placing them in a tiered cache, such as GPU memory → CPU memory pool → object storage. A controlled loss in precision can be exchanged for greater cache capacity and more reuse opportunities.
- Monitor cache hit rate and TTFT, then continuously tune checkpoint density and eviction policies.
6. Conclusion
Hybrid architectures are an important way to balance the capability and efficiency of large language models. To realize their full performance potential in multi-turn, long-context Agent workloads, however, cache management must evolve from a KV-centric design into a dual-channel system that gives equal importance to KV data and recurrent states.
This approach is not tied to a specific framework. Teams developing or extending inference services can adopt the same principle: treat linear-layer states as reusable, distributable cache objects and manage them systematically. Doing so can deliver substantial performance improvements during prefill.
