Cache Coherence for High-Concurrency Long-Context LLM Inference

    Large-scale coding agents, multi-turn tool use, and very long repository contexts have pushed LLM inference beyond passing sample cases: outputs must remain correct continuously under production traffic. This article explains why short, low-concurrency tests miss cache-coherence failures, how speculative decoding acceptance metrics can serve as low-cost canaries, and how two common root causes—premature KV-cache reclamation in disaggregated prefill/decode and asynchronous KV loads that become visible too late—produce garbled or repetitive output. It also covers lifecycle state machines, kernel-level synchronization, layer-partitioned prefill throughput trade-offs, and practical observability and release gates for catching these defects before deployment.

    Large-scale coding agents, multi-turn tool calling, and very long repository contexts have moved LLM inference from “can answer sample cases correctly” to “must remain continuously correct under production traffic.” These workloads have clear characteristics: high concurrency, long contexts, uneven request lifetimes, and frequent cancellations and timeouts. Many failures that appear only under this combination are nearly invisible in offline evaluation and low-concurrency smoke tests. The typical symptom is not a service crash but quietly degraded output: occasional garbled text, insertion of characters that are extremely rare in the training distribution, or mechanical repetition of the same span. From the user side, it looks like fluctuation in model capability, but the root cause is often cache coherence in the serving runtime.

    Why Short Tests Miss It

    Prefill writes the existing context into the key-value (KV) cache, and decode reads it step by step while appending new tokens. To improve cluster utilization, many systems split these two stages across different processes or even different devices: prefill is compute-bound, while decode is bound by cache bandwidth and scheduling. Cache blocks are borrowed and returned rapidly in a global pool. At low concurrency, the interval from allocation and write to read and reclamation is long, so race windows are hidden. At high concurrency with very long contexts, the pool turns over extremely quickly, and timeouts, cancellations, retries, and preemption interleave. A microsecond-level “reclaim before write” can become a user-visible dirty read.

    Therefore, checking only whether the final answer looks like natural language is not enough. Runtime instrumentation needs signals that distinguish “the model would naturally say this” from “the cache has been corrupted.” Speculative decoding happens to provide such side products.

    Using Speculative Decoding Metrics as Canaries

    Speculative decoding lets a smaller model or draft head propose several candidate tokens, which the target model then verifies in one pass. In production, two aggregate metrics should be monitored continuously and bucketed by model version and context length, rather than looking only at global averages.

    • Low acceptance length: Draft-proposed spans are almost never accepted by the target model, often accompanied by garbled text, invisible characters, or rare symbols at the tail of the vocabulary. This looks more like “the cache read is inconsistent with the current request” than simply high sampling temperature.
    • Abnormally high acceptance rate with repetitive output: The draft and target models agree strongly on the same local pattern, which often occurs when the cache retains state from a previous turn or from an earlier position in the same request, locking decode into a self-loop.

    These two metrics cannot replace correctness tests, but they are suitable canaries: when a metric deviates from baseline, automatically retain the request ID, KV block IDs, prefill completion time, and decode start time for post-hoc alignment. The logic can be sketched as follows.

    1. on_step(req):
    2. if accept_len < p10_baseline(bucket(req)) for N steps:
    3. mark(req, "possible_corruption")
    4. dump(req.kv_block_ids, prefill_done_ts, decode_start_ts)
    5. if accept_rate > p95_baseline and repeat_ngram(req.output):
    6. mark(req, "possible_stale_cache")

    Thresholds must be bucketed by context length and concurrency. Long contexts naturally reduce acceptance length, so short-request baselines must not be used and cause false alarms.

    Root Cause 1: Cache Reclamation Race in Disaggregated Architectures

    After a request times out or is cancelled on the decode side, the scheduler returns its KV blocks to the memory pool as quickly as possible for the next batch of prefill. If prefill is still writing old data from the previous owner into the same block, two-way operations occur:

    1. The decode side considers the block free and assigns it to a new request.
    2. The asynchronous write on the prefill side still points to the old address.
    3. The new request reads a mixture of new and old KV data, contaminating the logits distribution and producing garbled text or rare tokens.

    The fix is not simply “add another lock.” The lifecycle of a cache block must be collapsed into a single state machine, and reclamation must wait for an explicit completion or safe abandonment from the prefill side.

    • Allocation: The block enters Writing and is bound to a request generation ID. All writes from older generations are discarded.
    • Prefill completion: Only if the generation ID still matches does the block enter Readable, with its write range recorded (layer IDs and token interval).
    • Cancellation or timeout: Prefill first sends Abort. Only after in-flight writes have ended or been invalidated by generation ID is the decode side allowed to Free.
    • Reuse: When a new request receives the block, it validates the generation ID and write range. On mismatch, the entire block must be recomputed; “read as much as possible” is forbidden.

    Across processes, completion signals should use persistent request identifiers rather than raw pointers or reusable slot indices. Slot indices in a high-churn pool are immediately occupied by the next request, which is exactly where the race becomes most insidious.

    Root Cause 2: Starting Forward Before Asynchronous Load Completes

    To hide storage or cross-device transfer latency, systems often asynchronously pull historical KV data back into device memory. If the forward stream does not wait for the load to complete before the indexer or attention kernel starts, the kernel reads uninitialized or partially updated blocks. The symptoms are similar to the race above, but the timing differs: it is not “written and then reclaimed,” but “the read happens before the write is visible.”

    The correct approach is to place the synchronization point immediately before the kernel that actually consumes the cache, not at some host-side position that is merely “logically issued.” Successful host queue submission does not mean device memory is visible.

    1. load_kv_async(block_range)
    2. wait_device(load_event) # must precede indexer / attention
    3. run_indexer(block_range)
    4. run_attention(...)

    Synchronization consumes part of the overlap benefit, so wait only for the layers and token ranges that this step actually reads, and use events rather than device-wide global synchronization. Also write “load complete” into the same trace as the speculative decoding canary, so it is easier to distinguish a reclamation race from a load-visibility problem.

    Prefill Throughput Only After Correctness

    Once the cache can be dirty-read, no throughput optimization matters. After the state machine and synchronization points are stable, the prefill bottleneck often becomes this: each device must hold KV data for all layers, memory cannot fit long contexts, and blocks are swapped in and out frequently. A common industry approach is layer-partitioned storage: each device persistently stores only a subset of layers, fetches or exchanges with neighboring devices when computing the corresponding layer, and overlaps communication with subsequent layer computation.

    Keep three points in mind during design to avoid reintroducing coherence problems through a throughput scheme.

    • Clear ownership: Each layer has only one writer at any time; partition boundaries are checked together with generation IDs.
    • Bounded overlap: Communication can run in parallel with computation that does not depend on that layer’s data, but it must not cross the wait points above.
    • Benefits depend on shape: Layer count, context length, batch size, and interconnect bandwidth jointly determine whether overlap pays off. Do not treat peak numbers from an experimental environment as a capacity commitment; before launch, replay your own traffic to measure tail latency and acceptance length, not just average throughput.

    Observability, Reproduction, and Release Gates

    The reproduction condition for these defects is “the pool turns fast enough,” not a particular prompt. A minimal set of gates should include the following:

    • Stress test with long contexts, high cancellation rates, and mixed long and short requests; continuously observe acceptance length, acceptance rate, KV block generation mismatch count, and device synchronization wait time.
    • Fault injection: artificially delay prefill completion notifications, reclaim early, and start forward before load completion. Confirm that the service rejects dirty blocks instead of emitting garbled output.
    • Version alignment: Scheduler, cache pool, and kernels must ship as one release unit. Upgrading only model weights while keeping an old cache protocol is the easiest way to reopen a previously fixed window.
    • Preserve evidence: When metrics are abnormal, persist block IDs, generation IDs, timestamps, and hashes of a few tokens before and after, rather than saving full user code content. This balances debugging and privacy.

    Conclusion

    After scaling laws raise model capability, production stability depends on whether the cache lifecycle is provable. High concurrency and long contexts amplify two windows: premature reclamation on the decode side, and asynchronous loads that are not yet visible before forward execution. Speculative decoding acceptance length and acceptance rate are low-cost canaries; generation IDs, write confirmation, and pre-kernel synchronization are the correctness baseline; layer partitioning should be built only on top of that baseline. Putting these checks into release gates is much closer to real failures than running the same sample ten thousand times in a test environment.