LLM Inference Acceleration: Cache Residency and Prefetching in Practice

    This article explains how to determine whether LLM inference is limited by computation or by data arrival, then applies two complementary strategies: cache residency to shorten reuse distance and prefetching to overlap data movement with computation. It covers working-set analysis, operator fusion, KV-state caveats, double buffering, prefetch timing, optimization order, and controlled testing to distinguish real gains from metric changes.

    1. Identify the Bottleneck: Slow Compute or Late Data?

    LLM inference typically has two phases: prompt processing and step-by-step decoding. Prompt processing can handle multiple tokens in the input in parallel, and its computation is often more concentrated. Step-by-step decoding repeatedly executes the model forward pass, generating the next token for each active request at every step. With small batches, weight reuse per round is limited, and weight reads may become a major cost. As context grows, the cost of reading historical key-value (KV) state during attention computation may also increase. The actual bottleneck depends on model architecture, numerical precision, request batch size, context length, and kernel implementation. It is not enough to conclude that the system is memory-bandwidth-bound simply because it is decoding.

    Keeping data on-chip also does not mean moving all model weights and historical state out of GPU memory. A more practical goal is to keep data that has a chance of reuse from being evicted before its next access, and to consume short-lived intermediate results before they are written to memory. For another class of data that must be read from GPU memory, the goal is to move it earlier so that the read overlaps with current computation. The first two approaches focus on reducing data movement; the third focuses on shortening waiting time.

    2. Strategy 1: Shorten Reuse Distance and Keep Hot Data On-Chip

    What determines whether a cache is useful is not the total model size, but the effective working set during a given execution interval: which data has already been loaded, which will be used again, and how much other data competes for cache between those two accesses. For example, if a weight block will be read again in adjacent compute tiles, the tiling schedule can exploit its temporal locality. If the same weight block will only be accessed again after all subsequent layers have executed, then even if it is small, it may already have been evicted by other data.

    Therefore, “working set smaller than cache capacity” is only an initial check, not a guarantee of cache hits. The cache may be shared by multiple execution units, different addresses may map to the same cache set, and concurrent tasks also consume space. Evaluation should account for inputs, weights, temporary results, and extra overhead such as double buffering. In practice, start with smaller tiles, gradually increase the tile size, and compare execution time, off-chip traffic, and resource usage instead of filling the nominal hardware capacity exactly.

    Another way to reduce memory access is to keep intermediate results out of off-chip storage. If adjacent operations can be fused, data produced by the previous operation can be consumed directly by the next operation in registers or explicitly managed on-chip storage, eliminating one intermediate write and one subsequent readback. This differs from “data happens to still be in the hardware cache after being written”: the former changes the dataflow, while the latter is still subject to cache and write policies. For read-only weights, the benefit of a cache hit is avoiding repeated reads from GPU memory, and there is no writeback problem to begin with. For writable data, whether it is written to GPU memory depends on data placement, write policy, and whether it must be read later across operators or decoding steps. Historical KV state usually must be retained across steps and cannot be handled like temporary results.

    Residency also has costs. If tiles are too small, loop control and boundary handling overhead may increase. Excessive fusion may increase register or on-chip storage usage, reduce the number of tasks that can run concurrently, and even introduce extra spill reads and writes. Therefore, a higher cache hit rate does not necessarily mean faster decoding; ultimately, kernel time and end-to-end latency still need to be measured.

    3. Strategy 2: Prefetch the Next Tile and Overlap Transfer with Compute

    If future access addresses can be determined in advance, there is no need to wait until computation actually needs the data before initiating a read. A typical arrangement uses two buffers alternately: first load the first tile; while processing the current tile, copy the next tile into the other buffer; after the copy completes, swap the buffers and proceed to the next round. This arrangement requires the hardware and execution method to support effective transfer-compute overlap and correct synchronization for buffer reuse.

    Prefetching does not mean the data no longer needs to be read from GPU memory; it moves that read off the consumer’s direct waiting path. If transfer time is shorter than the computation time available for overlap, most of the waiting in steady state may be hidden. If computation is very short, or multiple requests have already saturated GPU memory bandwidth, initiating the request earlier cannot raise the bandwidth ceiling. This also explains a common measurement result: the cache hit rate for on-demand reads rises, but the number of bytes read from GPU memory does not drop significantly. The reads may simply be happening earlier, rather than the overall process moving less data.

    Prefetch timing needs to be determined experimentally. If initiated too late, the data may not be ready when consumed. If initiated too early, and hardware cache is used to hold prefetched data, it may be evicted before use. If explicitly managed buffers are used, moving data too early extends buffer occupancy time. For tiles with stable address patterns and access order, it is usually reasonable to first test prefetching one tile ahead. For requests with highly variable lengths and frequent scheduling changes, pay particular attention to ineffective prefetches: data that has not yet been consumed may become useless when a request ends or the batch changes, while still consuming bandwidth.

    4. Optimize in Order: Reduce Movement First, Then Prefetch

    The first step is to inspect the execution path for unnecessary intermediate result writes and readbacks. If data between two adjacent operations is only used locally, evaluate operator fusion or rearranging the data layout. If the same data really must be read multiple times, check whether adjusting tiling and traversal order can bring those reads closer together. This prioritizes reducing GPU memory traffic rather than relying on a more complex pipeline to hide movement that could have been avoided.

    The second step is to introduce prefetching for the remaining necessary reads. Prioritize paths that account for a high share of time and have stable access order, and verify whether transfer-compute overlap can be achieved without changing the computation logic. When combining the two strategies, recalculate on-chip space: buffers added for prefetching may take space originally used to retain hot data. If each strategy works alone but the combination regresses, check cache eviction, on-chip resource usage, and reduced concurrency instead of continuing to increase prefetch distance.

    5. Use Controlled Tests to Separate Real Gains from Metric Changes

    Tests can be divided into four groups: baseline, data residency only, prefetching only, and both combined. Keep the model, precision, request batch size, and input/output length distribution as fixed as possible, and measure cold start and warmed steady state separately. In addition to total service throughput, record per-request inter-token latency, time to first token, and high-percentile latency. A throughput gain may sometimes come from a larger batch size, while per-request generation speed may not improve. Reporting only a single tokens-per-second number cannot distinguish these two outcomes.

    If hardware counters can be collected, further compare GPU memory read/write bytes, cache hit rate, and memory access wait time. For the same effective work, lower off-chip traffic and shorter time suggest that reduced movement may be the main benefit. If traffic remains roughly unchanged but waiting decreases, that is more consistent with latency being hidden. If the hit rate rises but time becomes longer, check synchronization overhead, on-chip resource usage, and extra reads, rather than treating hit rate as the optimization goal. Finally, retest with short context, long context, and different concurrency levels to confirm that the gain is not limited to a single workload.

    To decide whether this kind of optimization is worth keeping, the key is not whether you can claim that “data never touches GPU memory,” but whether you can answer two questions: which data movement was reduced, and which necessary movement no longer blocks computation. Separating these two accounts is what explains the source of the performance improvement and whether it still holds after the workload changes.