Prefix Caching Dual-State Reuse for Hybrid Model Inference Acceleration

    Prefix caching can eliminate most of the redundant prefill work in append-only LLM sessions, but hybrid models that combine full attention with linear attention or SSM layers make it harder: a valid cache hit must reuse both the attention KV cache and the recurrent state at the same token position. This article explains why append-only coding sessions are a natural fit for prefix caching, why hybrid architectures require dual-state reuse, how to break a cache miss into separately diagnosable causes (prefix consistency, state completeness, and position alignment), how to read acceleration benchmark numbers critically, why saved compute does not automatically translate into a lower API bill, and practical steps for keeping prefixes stable and observing hit rates.

    1. Why Inference Acceleration Should Focus on Repeated Computation

    In the LLM inference pipeline, a single request typically has two phases: prefill processes all input tokens at once to build the initial context representation, while decode generates new tokens one by one. For long-context requests, prefill is often the main source of time-to-first-token (TTFT) latency — the longer the input, the more matrix operations happen before the first token, and the longer users perceive the wait.

    Append-only coding tool sessions have a distinctive trait: nearly every turn re-sends the system prompt, tool definitions, and the entire prior conversation history. The system prompt and tool definitions are usually large and nearly unchanged, while the conversation history grows turn by turn. This creates a large amount of “tokens that have already been computed but must be computed again.”

    2. Why Append-Only Sessions Naturally Suit Prefix Caching

    The basic idea of prefix caching is: if the opening sequence of tokens in the current request is identical to that of a previous request, save the inference intermediate results for that prefix and reuse them directly in the next turn instead of re-running prefill.

    In append-only scenarios where “history is neither compressed nor edited,” this premise is almost always met:

    • Prefix stability: the system prompt and tool definitions stay unchanged throughout the session;
    • Incremental append: new content is only appended at the tail of the history and never modifies earlier token sequences;
    • The longer, the more you save: as turns increase, the reusable prefix grows longer and the cost of repeated prefill grows too.

    So an inference engine that can seamlessly continue from “where the previous turn ended” can theoretically avoid most of the repeated computation.

    3. Hybrid Models: Why Both Things Need Caching

    Early pure-attention (transformer) models only needed to cache the attention key-value pairs (the KV cache). Today, many models use hybrid architectures: some layers use full attention, while others use linear attention or state space models (SSMs). The latter behave more like recursive systems — instead of retaining a full KV, they maintain a fixed-size “recurrent state” that evolves over time.

    This introduces a key constraint: to let the next turn resume from the same position, the prefix cache must provide both kinds of state at the same time:

    1. The attention key-values of the full attention layers — which determine the history visibility of these layers;
    2. The recurrent state of the linear attention / SSM layers — which determines what these layers “remember.”

    If either is missing, or the two are not stopped at the same token position, continuation is not truly possible; the engine can only fall back to re-running prefill for that prefix. In other words, the cache hit condition for hybrid models is stricter than for ordinary models.

    Conceptually, a hit check can be understood like this:

    1. # Illustrative code: a prefix cache hit requires both states and position alignment
    2. def try_reuse_prefix(cache, token_ids):
    3. prefix_len = longest_common_prefix(cache.token_ids, token_ids)
    4. kv = cache.fetch_kv(prefix_len) # KV cache for full attention layers
    5. state = cache.fetch_recurrent(prefix_len) # recurrent state for linear attention/SSM layers
    6. # Both states must exist and be at the same position; otherwise fall back to re-prefill
    7. if kv and state and kv.position == state.position:
    8. return prefix_len
    9. return 0

    4. Breaking “Cache Miss” into Separately Addressable Causes

    The methodological value of this kind of work is that it breaks a vague “cache miss” into causes that can be investigated and optimized separately. A genuine hit must satisfy at least three conditions:

    • Prefix consistency: the token sequence of the request prefix matches what is recorded in the cache (affected by sampling, templates, tool definition concatenation order, and so on);
    • State completeness: both the attention KV and the recurrent state have been cached and have not expired or been evicted;
    • Position alignment: the two kinds of state sit at the same token offset and can serve as the continuation point.

    Once broken apart, you can locate problems individually: whether a low hit rate is caused by the prefix being rewritten, states being evicted, or the two states being misaligned. Compared with looking at a single overall hit-rate number, this decomposition is more directly actionable for engineering optimization.

    5. How to Rationally Interpret Numbers in Acceleration Demos

    Any nice number from an acceleration scheme needs to first distinguish whether it comes from a demo, the main experiment, or an extended experiment, and what the measurement conditions are. Take a public demo as an example: the same batch of coding-agent sessions, the same execution trace, and three inference systems running on the same batch of GPUs, over a 30-minute measurement window:

    • Completed sessions were about 20 / 7 / 8 respectively (not all total tasks were completed);
    • Cache hit rates were about 87% / 30% / 6% respectively;
    • Output tokens per minute (output TPM) were about 61.7K / 22.0K / 15.4K;
    • Per 100 input tokens, the number that had to be recomputed was about 13 / 70 / 94.

    61.7K versus 15.4K is roughly a 4x difference, but reading these numbers requires three cautions:

    1. A demo is not the upper limit. A demo does not necessarily fully disclose how tuned the baseline is, so you cannot conclude that a baseline system’s hit-rate ceiling is 6%;
    2. “100x speedup” usually refers to the playback rate of a recorded replay, not the actual experiment duration. The main experiment often has a clear runtime and statistical measure (for example, statistics over the last 30 minutes after warm-up); you cannot divide the trace duration by the playback rate to back out the experiment time;
    3. Conditions cannot be mixed. Demos, main experiments, and extended experiments have different input distributions and measurement methods, so cross-condition number comparison is meaningless.

    6. Saved Compute Is Not Necessarily Saved Money

    For teams that self-host inference services, computing fewer repeated tokens means directly saving GPU time, which means a lower per-token cost and higher throughput — a tangible gain.

    But for applications that buy managed APIs, how much compute the server saves does not automatically translate into a discount on your bill — the billing basis (by input tokens, output tokens, or another dimension) is decided by the provider, and the benefit of the optimization may fall on the provider’s cost side rather than your bill side. So for users, a more observable benefit signal is often at the experience level: whether the wait time before generation begins in a long session (time-to-first-token latency) drops noticeably.

    7. Practical Recommendations

    1. Keep the prefix as stable as possible: fix the concatenation order of the system prompt and tool definitions, avoid rewriting history mid-session, and let the prefix be reliably reusable;
    2. Focus on the joint caching of both kinds of state: in hybrid architectures, the optimization target is not a single-point KV cache, but the consistent reuse of the KV and the recurrent state at the same position;
    3. Build decomposable hit-rate observation: record separately “whether the prefix is consistent, whether the states are complete, whether the positions are aligned” — this is better for locating problems than watching only the overall hit rate;
    4. Treat benchmark numbers rationally: distinguish demo from experiment conditions, focus on real experience metrics such as time-to-first-token latency, rather than looking only at a single amplified speedup factor.

    In summary, the core difficulty of hybrid model inference acceleration is not whether “a cache is used,” but whether the attention key-values and the recurrent state can be completely and precisely aligned for reuse at the same position. Only by clearly breaking down the hit conditions and viewing demo data back within its real measurement conditions can you both obtain the performance benefit you deserve and avoid being misled by nice-looking numbers.