Cost Optimization for Long-Context Agent Inference: Prefill and KV Cache
The rise of agentic applications has pushed model context windows from a few thousand tokens of chat into hundreds of thousands of tokens of live task state, while multimodal screenshots and documents add thousands more visual tokens. This shifts where inference cost actually goes: not into single-step decoding, but into prefilling long prompts and holding KV caches that grow linearly with context. This article breaks down the two cost bottlenecks of long-context agent inference and organizes the optimization space into four independent dimensions — per-record KV precision, the number of historical positions that actually participate in attention, cross-layer KV duplication, and how many layers a prompt token must traverse — showing how asymmetric compute, sparse attention, cross-layer reuse, quantization, and tiered storage combine multiplicatively. It also covers Engram-style decoupled memory, and closes with a metrics framework and a practical tuning checklist for engineering teams.
Cutting Long-Context Agent Inference Costs: Optimizing Prefill and KV Cache
The spread of agentic applications has changed what models are asked to handle. Context length has gone from “a few thousand tokens of conversation” to “hundreds of thousands of tokens of live task state”: a single complex software engineering task may involve hundreds of tool calls, large terminal outputs, and substantial chunks of a code repository. At the same time, multimodal visual understanding is growing fast — screenshots, charts, and scanned documents all enter the context as large volumes of visual tokens.
The cost structure changes with it. The dominant source of inference overhead is no longer single-step decoding, but the prefill computation of long prompts and the key-value cache (KV Cache) that grows linearly with context length. This article is not about one specific model. It takes a general architectural and engineering view, systematically walks through the two main cost bottlenecks of long-context agent inference, and lays out optimization paths you can actually implement.
1. Where the Cost Comes From: Two Bottlenecks
1.1 Three typical sources of long contexts
- Agent loops: every tool call requires the model to continue reasoning on top of the previous round’s results, so history keeps accumulating.
- Raw output fed back in: build, test, and log commands routinely emit tens of thousands of tokens. Concatenating them straight back into the context means most of that content is irrelevant to the current decision.
- Long-horizon task memory: deep research, GUI operation agents, and enterprise workflow automation all need to stay aligned with the goal and avoid repeating work across many rounds of interaction.
Multimodal input amplifies the problem further: a single high-resolution screenshot can translate into thousands of visual tokens after visual encoding, and a handful of images can fill the entire context window.
1.2 Prefill computation: the main driver of first-token latency
A conventional decoder-only Transformer processing a prompt of length N must traverse every token in each of its L layers, so prefill computation is roughly O(L·N). The attention part alone requires computing the interaction between all query positions and all historical keys — approximately O(N²·d) under a causal mask, where d is the head dimension — which grows quadratically with context length.
In long-context settings, prefill is often the largest component of Time To First Token (TTFT). After a user submits a task, they may wait seconds or even tens of seconds, and most of that time goes into prefill.
1.3 KV Cache: the bulk of long-context memory
To make autoregressive decoding incremental, models cache keys (K) and values (V) separately for every layer and every historical position. Cache size can be estimated as:
CacheBytes = 2 × Layers × KV_Heads × Head_Dim × PrecisionBytes × Tokens
With 32 layers, 8 KV heads, and a head dimension of 128:
def kv_per_token(layers, kv_heads, head_dim, bytes_per_elem):# Each token stores both K and V in every layer, hence the factor of 2return 2 * layers * kv_heads * head_dim * bytes_per_elemcfg = dict(layers=32, kv_heads=8, head_dim=128)fp16 = kv_per_token(**cfg, bytes_per_elem=2) # 131072 bytes/tokenfp4 = kv_per_token(**cfg, bytes_per_elem=0.5) # 32768 bytes/tokenfor ctx in (128_000, 1_000_000):print(f"{ctx:>9,} tokens: FP16 {fp16 * ctx / 2**30:.1f} GiB | "f"FP4 {fp4 * ctx / 2**30:.1f} GiB")# 128,000 tokens: FP16 15.6 GiB | FP4 3.9 GiB# 1,000,000 tokens: FP16 122.1 GiB | FP4 30.5 GiB
Three things stand out from these numbers:
- KV Cache grows linearly with context length, and easily reaches the hundreds of GiB range at million-token scale.
- Dropping cache precision from 16 bits to 4 bits yields roughly a 4x compression.
- This memory sits permanently in GPU HBM and directly squeezes out room for weights and concurrent batching.
When a single GPU cannot hold it all, the cache also has to be offloaded per session to host memory or SSD, and transferred between nodes in multi-turn conversations or distributed inference — adding further latency and bandwidth cost.
2. Four Compression Dimensions: The Overall Framework
To reduce long-context inference cost, it helps to go back to fundamentals and ask one question: does every token, every historical position, and every layer really need to be computed and stored the way it currently is? Around this question sit four independent compression dimensions:
| Dimension | Optimization target | Typical techniques | Main benefit |
|---|---|---|---|
| 1: bytes per KV record | KV Cache precision | Low-bit quantization, grouped scaling, error compensation | Lower memory footprint and transfer bandwidth |
| 2: historical positions that actually attend | Attention scope | Query-aware sparsity, sliding window, hierarchical summarization | Lower prefill compute and decode latency |
| 3: KV duplicated across layers | Cross-layer redundancy | Cross-layer KV sharing, shared indices and selections | Lower memory and indexing cost |
| 4: layers a prompt must traverse in full | Prefill coverage | Asymmetric compute, layer skipping | Prefill compute cut by nearly half |
These dimensions stack. Quantization shrinks “how big each record is,” sparsity controls “how many records actually participate in computation,” cross-layer reuse removes “duplication between layers,” and asymmetric compute reduces “how many tokens have to go through every layer.” When a publicly documented architecture combines several of these, the global KV Cache per token that stays resident in memory drops to the order of a hundred bytes — which confirms that the dimensions multiply rather than add.
3. Dimension 4: Cutting Prefill Cost with Asymmetric Compute
3.1 The core observation: tokens are not equal
In a typical agent prompt, only a small set of content actually determines subsequent behavior — system instructions, the return value of the most recent tool call, the user’s latest correction. The rest, including historical logs, full file contents, and repeated outputs, mostly serves as contextual backdrop. Running every token through the full deep stack layer by layer is a one-size-fits-all waste.
3.2 A representative approach
One representative design is the Causal Encoder-Decoder (CED), an asymmetric causal encoder-decoder structure. The basic idea:
- Most prompt tokens pass only through the first half of the network during prefill, where the encoder side performs relatively general representation extraction.
- Only a small subset of tokens (for example, the decoding starting point and its causal dependencies) continues into the second half, where the decoder side performs the deeper, generation-oriented reasoning.
This works because attention during generation can inherently only see content before the decoding position. Deep representations of historical tokens are usually not read directly by later generation — the key is to write their keys and values into the cache in as informative a form as possible.
3.3 Estimated gains
- Lower activated parameters per token: in the original design, decoding activates about 16B parameters per token. CED brings most tokens in the prefill stage down to roughly 8B activated parameters, cutting it in half.
- Prefill compute close to halved: assuming computational semantics stay essentially unchanged, the first half of the network still processes the same number of tokens, while the second half processes far fewer. Since deeper layers typically hold more parameters and do heavier computation, total prefill compute can approach a 50% reduction.
- Impact on TTFT: prefill is a compute-bound stage, so lower compute translates directly into lower TTFT. In multi-turn agent interactions, this benefit compounds round after round.
3.4 Engineering considerations
- The boundary between the encoder side and the decoder side must be defined explicitly. Typically the first half handles general features and the second half handles deeper features that matter more for generation.
- Watch the representation handoff at the boundary, and avoid semantic drift in the keys and values produced by the encoder side as they enter the decoder side.
- Evaluation cannot look at single-turn capability alone; long-horizon consistency across multi-round tool calls must be checked for regressions.
Illustrative structure:
Prompt Token (N)│▼┌──────────────────────┐│ Encoder side: L1…Lk │ ← every token passes through└──────────┬───────────┘│ cache K/V▼┌──────────────────────┐│ Decoder side: Lk+1…L│ ← only generation-relevant tokens└──────────┬───────────┘▼Next Token
4. Dimensions 2 and 3: Sparse Attention and Cross-Layer Reuse
4.1 Sparse attention: attend only to the most relevant positions
In standard attention, every query computes similarity against all historical keys. Sparse attention first scores each query with a lightweight indexer, selects the top-K highest-scoring historical positions, and then runs exact attention only over those positions. Attention computation drops from O(N²) to O(N·K). The key engineering details:
- The indexer should stay lightweight so it does not become a new bottleneck itself.
- K needs to be tuned per task type, with recall validated on retrieval-heavy tasks.
- Structures such as sliding windows and hierarchical summarization can be combined with Top-K selection.
4.2 Cross-layer reuse: stop letting layers work in isolation
In multi-head attention, different layers often attend to similar historical spans. Designs such as CSA2 reuse the main KV, the indexer’s K, and the Top-K selection results across layers, removing three kinds of duplicated overhead: duplicated cache writes, duplicated index computation, and duplicated Top-K selection. Conceptually:
# Pseudocode: combining cross-layer reuse with Top-K sparse attention# For illustration only; this is not the implementation of any specific frameworkdef shared_sparse_attention(query, kv_pool, reuse_from_layer=None):if reuse_from_layer is not None:# Cross-layer reuse: inherit the selection and cache from a lower layerselected = reuse_from_layer.topk_indiceselse:# Sparsification: score historical positions with a lightweight indexerscores = indexer.score(query, kv_pool.keys)selected = topk(scores, k=BLOCK_TOPK)kv = kv_pool.gather(selected)# Run exact attention over only the K selected positionsreturn softmax(query @ kv.keys.T / scale) @ kv.values
4.3 Benefits and costs
- Compute: both KV Cache reads/writes and attention computation touch only the selected positions, so memory traffic per decode step drops substantially.
- Memory: cross-layer reuse means layers no longer each maintain a full cache, so the global KV Cache falls roughly linearly with the number of layers.
- Cost: too small a Top-K loses information, so regression testing on task-specific retrieval evaluation sets is required; the quality of the indexer implementation directly determines selection accuracy.
5. Dimension 1: Quantization and Tiered Storage Push KV Cache to the Limit
5.1 Low-bit quantization
Compressing the KV Cache from 16 bits to 4 bits (FP4) gives roughly 4x compression in theory, and it is the most direct lever in Dimension 1. Low-bit quantization needs supporting measures:
- Grouped scaling: maintain scale factors per channel or per token group so individual outliers do not widen the dynamic range.
- Error compensation: model quantization error, or keep higher precision in critical layers.
- Precision evaluation: measure the accuracy drop separately on long-context retrieval, code completion, and mathematical reasoning.
The earlier example already shows the memory difference: for the same million-token context, FP16 needs about 122 GiB while FP4 needs about 30 GiB.
5.2 Tiered storage and swapping
When a single GPU’s memory cannot hold the entire cache, a tiered storage strategy helps:
- Hot tier: KV for recent tokens stays in GPU HBM.
- Warm tier: older historical cache lives in host memory.
- Cold tier: session-level historical cache is persisted to object storage or a shared file system, enabling cross-session reuse and failure recovery.
Combined with prefix cache reuse and prefetching, repeated prefixes within the same session no longer need to be prefilled again. For agents that make “many round trips within the same task,” the payoff is especially clear.
5.3 Putting it together
Combining Dimensions 1 through 4 can shrink the global KV Cache per token from thousands of bytes to the order of a hundred bytes. This also explains why, in some architectural designs, million-token contexts still fit within limited memory: compression is not a single-point trick but a multiplicative effect across four dimensions.
6. Expanding Knowledge Capacity: Engram-Style Memory Decoupled from the Backbone
Beyond compressing the context, another approach is to give the model a larger knowledge capacity without adding to the backbone’s compute burden. Memory mechanisms such as Engram decouple knowledge storage from backbone computation:
- The backbone network handles general reasoning and instruction following.
- The memory module is activated on demand in an approximately retrieval-like fashion, expanding knowledge capacity at a low activation cost.
This is not in conflict with retrieval-augmented generation (RAG). RAG keeps knowledge external, and every retrieval adds network and latency overhead. A memory module internalizes frequently used knowledge and invokes it on demand through sparse activation, which suits high-frequency, relatively stable domain knowledge. The two can be combined: memory covers “what is already familiar,” retrieval covers “what is newly appearing.”
Illustrative implementation:
# Pseudocode: memory module decoupled from the backbonedef engram_forward(hidden, memory_slots, topk_slots=8):# Query memory slots with the current hidden state, activating only a fewscores = hidden @ memory_slots.keys.Tidx = topk(scores, k=topk_slots)mem = memory_slots.values[idx] # sparse activationgate = sigmoid(hidden @ memory_slots.gates[idx])return hidden + gate * (hidden @ mem.T) # add to the backbone, structure unchanged
Three things need attention in practice: how memory slots are initialized and trained, capping the number of activated slots, and the joint training strategy between memory modules and backbone parameters.
7. Engineering Practice: Metrics and a Tuning Checklist
7.1 Establish metrics before optimizing
Evaluating long-context agent workloads cannot rely on single-turn answer accuracy alone. It is worth building a metrics framework:
- Time To First Token (TTFT): reflects prefill efficiency and is the core indicator of interaction quality.
- Time Per Output Token (TPOT): reflects memory access and compute efficiency during decoding.
- Effective context length: the maximum context length that can be served stably under a given memory budget.
- Cache hit rate: reuse ratio of prefix caches and cross-session caches.
- Long-horizon task completion rate: success rate and consistency across multi-round tool calls and long workflows.
- Inference cost per token: the final composite metric spanning memory footprint, throughput, and hardware utilization.
7.2 Tuning checklist
- Inspect prompt composition: summarize or trim large logs and full file contents before feeding them back. This usually reduces both prefill and cache pressure at once.
- Set Top-K and quantization precision deliberately: establish the lower bound on retrieval evaluation sets first, then tighten gradually.
- Enable prefix caching first: in agent systems, fixed prefixes such as system prompts and tool descriptions should go at the very front so they can be reused across rounds.
- Keep tiered storage as a fallback: persist very long session histories so single-GPU memory does not become the bottleneck.
- Build a regression process: before any compression technique ships, run comparison evaluations on long-context retrieval, code, and math tasks.
8. Conclusion
Cutting the cost of long-context agent inference comes down to subtracting on four dimensions at once: smaller KV records, fewer positions that actually participate in attention, less duplication across layers, and fewer tokens that need full computation. Asymmetric compute addresses prefill; sparse attention and cross-layer reuse address attention scope and inter-layer redundancy; quantization and tiered storage address per-record size; memory modules address the balance between knowledge capacity and activation cost. Only by stacking all of them can you satisfy latency, memory, and cost constraints simultaneously at million-token context lengths.
For engineering teams, the point is not to copy one architecture. It is to first build your own metrics framework, then validate the benefits and risks dimension by dimension, and find the combination that fits your business.
