Optimizing Inference Cost for Long-Context and Agent Workloads
This article explains why long-context and Agent workloads have become the primary driver of large-model inference cost. It breaks down Prefill and Decode as two distinct types of "bills," then walks through architecture-level optimizations — asymmetric compute structures to reduce Prefill work, cross-layer KV reuse, selective sparse attention, and FP4 KV quantization to shrink global KV cache, plus decoupled knowledge memory (Engram) to expand knowledge capacity at low activation cost. It also covers the serving-layer engineering — tiered KV storage, prefix reuse, chunked prefill, and disaggregated scheduling — needed to turn architectural gains into real reductions in cost and latency, along with the precision evaluation needed before rollout. Readers will get a structured framework for finding their own cost bottlenecks and building a continuous cost-latency-quality optimization loop.
1. Why Long-Context and Agent Workloads Are the Main Battleground for Inference Cost
1.1 The Workload Is Shifting: From “One Question, One Answer” to “Long-Horizon, Multi-Step”
Historically, most large-model calls followed a one-question-one-answer pattern: a single request carried a few hundred to a few thousand tokens, and the response was typically only a few hundred tokens. In these short-input, short-output workloads, inference cost is spent mainly in the generation phase — that is, token-by-token decoding.
Agents have completely changed this pattern. To complete a task, an agent typically loops through observe environment → think and decide → call a tool → observe the result, again and again. At every step it must resubmit the entire interaction history to the model. Take software engineering tasks as an example — a single full delivery may involve:
- Hundreds of tool calls, such as reading files, searching code, and running tests;
- The full contents of a code repository, plus the diffs produced by multiple rounds of edits;
- Terminal output that includes stack traces, build logs, and command echoes, where a single output can generate tens of thousands of tokens;
- The context accumulated across multiple rounds of edits, along with task constraints carried over between rounds.
Similar situations arise in other kinds of agents: deep-research agents must continuously accumulate retrieved results and cross-check them; GUI agents must retain both interface snapshots and action sequences at the same time; and enterprise workflow agents must span long time periods while continuously maintaining task state and business knowledge.
The conclusion is that long context is no longer an “optional optimization” — it is an infrastructural property of Agent systems. It directly determines per-task cost, system concurrency, and ultimately service quality.
1.2 The Structural Bottleneck of the Traditional Decoder-Only Transformer
In the scenarios above, a traditional decoder-only Transformer exposes two major costs.
The first is Prefill computation. With L layers and a prompt length of N, every layer must run a full computation over all prompt tokens. The linear portion of the sequence has a computational complexity of roughly O(NL), which also includes the quadratic term of self-attention. This cost directly affects a request’s time to first token (TTFT) and throughput.
The second is the KV Cache. Standard multi-head attention stores separate keys and values for every historical token in every layer. As the context grows, the KV Cache keeps occupying GPU HBM (high-bandwidth memory), and across multiple conversation turns it triggers a cascade of overheads spanning host memory, SSD, and even network transfers.
1.3 Four Dimensions of Compression
Once the problem is abstracted, the viable space for optimizing long context can be grouped into four directions:
- Shrink the size of each individual KV entry;
- Reduce the number of historical positions that actually participate in attention;
- Eliminate KV that is redundantly stored across different Transformer layers;
- Reduce the number of network layers that must run in full during the Prefill phase.
The following sections discuss these four directions in the context of concrete architecture designs.
2. Cost Breakdown: Prefill and Decode Are Two Different “Bills”
Before discussing optimization techniques, we first need to see where the cost actually goes.
Prefill phase: The entire prompt is fed to the model in one pass. Matrix-multiplication density is high, so this is compute-bound, and the cost has two components:
- The computation of the linear parts (the FFN and various projections), roughly proportional to “number of tokens × number of activated parameters”;
- The attention part, which grows quadratically with sequence length.
Decode phase: For every token generated, all historical KV must be read from memory into the compute units. Arithmetic intensity here is low, so this phase is memory-bound. The number of KV bytes that must be moved at each step largely determines the generation speed of a single request.
KV Cache memory can be estimated approximately as follows:
KV_bytes ≈ 2 (K and V) × layers × KV heads × head dimension × bytes per element × context length
For example (an illustrative estimate for intuition only): with 32 layers, 8 KV heads, a head dimension of 128, and FP16 precision (2 bytes per element), each token occupies about 128 KiB of KV. At that rate, a 128K context consumes roughly 16 GiB of memory — and that does not even count the model weights themselves.
In short, the KV Cache determines how much concurrency the system can run and how long a context it can support, while Prefill cost determines the response speed and throughput ceiling of long requests. Therefore, long-context optimization must cover both dimensions: “compute less” and “store less.”
3. Architecture-Level Optimization I: Reducing Prefill Computation
3.1 Asymmetric Compute Structure: Making “Reading History” Cheaper
One typical idea borrows from the asymmetric thinking of the encoder-decoder design. Most prompt tokens mainly pass through the first several layers of the network during Prefill, while the small number of tokens generated during decode go through the full network. As a result, the number of activated parameters per token during Prefill can be significantly lower than during decode. For instance, dropping from the 16B scale to the 8B scale roughly halves the linear Prefill computation.
The intuition behind this design is that in Agent tasks, the historical context is closer to “reference material to be consulted repeatedly,” while the generation phase usually needs to read only a small number of tokens in detail. Since most tokens do not need full-depth refinement, there is no reason to make them pay the full computation cost every time.
There are two accompanying design points:
- Preventing a shallow-layer bottleneck: attention-pooling structures should be introduced at the front of the network so that the representations produced by shallow layers are sufficiently rich;
- Extending this idea: the notion of “running an entire prefix through only the front part of the network” can be generalized to more scenarios, such as system prompts and tool definitions that are resubmitted repeatedly across multi-turn conversations.
3.2 Coordinating Training and Inference
For an asymmetric structure to truly land, the training phase must be coordinated:
- Inference-path consistency: during training, mix-sample “full path” and “asymmetric path” examples so the model learns to use context correctly at different depths;
- Objective-function adaptation: for tokens that pass through shallow layers, a weaker prediction constraint can be applied — for example, predicting only coarse-grained information about what follows, rather than matching each token precisely;
- Capability regression checks: focus on long-range dependency abilities, such as coreference resolution across multiple conversation turns and multi-step reasoning, to confirm that shallow representations have not lost key evidence.
3.3 Combined Gains at the Quantization and Operator Level
Low-precision computation is another main line for reducing Prefill cost. For example, using an FP4 (4-bit floating-point) format:
- The storage width of a single value is only one quarter of FP16;
- On hardware that supports low precision, the theoretical throughput of matrix multiplication can increase by a multiple;
- The amount of data moved for both weights and activations drops accordingly, which is especially effective for memory-bound portions.
On the engineering side, IO-aware fused attention operators, chunked prefill, and sensible compute-to-memory ratios can all convert architecture-level gains into real throughput improvements.
4. Architecture-Level Optimization II: Compressing the Global KV Cache
4.1 Cross-Layer Reuse: Eliminating Redundant KV and Indices Between Layers
Standard attention stores separate KV for every layer, and there is substantial redundancy between layers. The cross-layer sharing approach is to have only some layers compute and cache KV while the rest reuse it directly. For example, when a group of KV is shared every k layers, the number of KV entries drops to 1/k of the original.
Going further, the key vectors produced by a lightweight indexer, along with each layer’s Top-K selection results, can also be reused across layers, avoiding repeated indexing and selection in every layer. Illustrative code:
# Illustrative code: sharing KV, indexer keys, and Top-K results across layers (not a real API)class SharedKVLayers(nn.Module):def __init__(self, blocks, share_every=4, top_k=512):self.blocks = blocks # sequence of Transformer layersself.share_every = share_every # share KV every k layersself.top_k = top_k # Top-K size for sparse attentiondef forward(self, x, cache=None):main_kv = indexer_k = topk_ids = Nonefor i, blk in enumerate(self.blocks):if i % self.share_every == 0:# Only shared layers compute KV and indices and write to the cachemain_kv = append(cache, blk.kv_proj(x)) if cache else blk.kv_proj(x)indexer_k = blk.indexer_proj(x)topk_ids = select_topk(indexer_k, self.top_k)# Remaining layers reuse the KV, indexer keys, and Top-K resultsout = blk.attn_with_cache(x, main_kv, topk_ids)x = blk.ffn(out)return x
4.2 Selective Sparse Attention: Keeping Only the Key Historical Positions
Cross-layer sharing alone still leaves the number of KV entries growing linearly with context length. The idea of selective sparse attention is to select the most relevant historical positions (Top-K) for each query token and run attention only over those positions. This limits the amount of history each query must attend to a constant K, decoupling it from the total context length.
The key to this mechanism is a low-overhead, scalable selection process, which typically relies on a lightweight indexer:
- It generates index keys for historical tokens and uses them to compute relevance scores between the query and the history;
- The selected entries enter the KV set that each attention head actually participates in.
4.3 FP4 Cache Quantization: Compressing Each Entry by Another 4×
Even after the number of entries has dropped, the size of each entry can still be compressed further. The mainstream approach is KV Cache quantization. One typical combination is to store the backbone KV in FP4 while keeping numerically sensitive components such as index keys at higher precision. Compared with FP16, each entry’s size can drop to about 1/4.
A few points to watch when implementing quantization:
- The quantization granularity should be fine enough — for example, assigning scale factors per token or per channel — to reduce error from outliers;
- Numerically sensitive positions must retain precision, such as the part that computes attention scores, to avoid accumulated error;
- During decode, the selected entries must be processed in high-precision compute units, so the quantization format should make fast restoration at decode time convenient.
4.4 Working Together: From “Storing Full Layers, Full History” to “Sparse and Shared”
The three techniques above correspond to different compression dimensions and can be stacked:
- Cross-layer reuse reduces the layer dimension of the entries;
- Selective attention caps the number of positions each query actually attends to at a constant K;
- FP4 quantization reduces the byte count of each entry.
As an illustrative configuration: 48 layers, sharing KV every 4 layers, backbone KV in FP4, index keys in FP8, and Top-K of 512. Plugging into the earlier formula gives 2 × 12 × 512 × 128 × 0.5 B, or about 768 bytes of global KV per token. At this scale, the global KV for a 1M context is roughly 768 MB — a striking drop compared with the tens of GiB consumed before optimization.
It is worth emphasizing that the memory savings from KV compression are only a gain in “nominal capacity.” Although selective attention needs to read only the selected entries during decode, the full historical KV must still be retained for selection and reuse in later turns. Therefore, the more important role of the saved memory is to increase the context length and number of concurrent sessions the system can carry.
5. Architecture-Level Optimization III: Decoupling Knowledge from Backbone Computation
Long-horizon tasks require the model not only to “remember more” but also to “know more.” If knowledge capacity is expanded purely by adding parameters, the number of activated parameters grows in tandem, and Prefill and Decode costs rise accordingly.
The idea of a decoupled knowledge-memory (Engram) module is to separate knowledge memory from backbone computation: the backbone network handles reasoning and decision-making, while knowledge memory exists as a standalone module that participates in computation only when needed, at a low activation cost. Its design points typically include:
- Sparse activation: wake up only the part of memory relevant to the current input, avoiding the activation of all knowledge parameters on every forward pass;
- Index-based retrieval: locate relevant memory through key-value matching, controlling the computation and memory-access cost of a single retrieval;
- Joint training with the backbone: train the memory module and the backbone together so that they align in representation space;
- Complementarity with retrieval augmentation: the memory module carries stable, internally internalized knowledge, while external retrieval handles information that is highly time-sensitive.
With this mechanism, the model can expand its knowledge capacity without significantly raising the per-token computation cost, achieving better performance on tasks that require a large amount of background knowledge.
6. Turning Architectural Advantages into Real Gains in Long-Context Scenarios
Architectural optimizations only translate into real cost reductions when they land in the inference service. The following engineering practices are especially critical.
6.1 Tiered KV Storage
The global KV resides permanently in GPU HBM. In multi-turn conversations, historical KV can be reused across turns and stored in tiers according to access frequency:
- HBM: currently active sessions;
- Host memory: recently used sessions that may be resumed;
- SSD: sessions not accessed for a long time;
- Remote object storage: task records that need long-term archiving.
The scheduler migrates data up and down tiers according to session activity, and prioritizes TTFT and decode speed for active requests.
6.2 Prefix Reuse and Cache Management
There is a large amount of shared prefix content among Agent requests — for example, system prompts, tool definitions, project background, and the content of the knowledge-memory module. For these common prefixes, the KV can be computed and cached on the first request and reused directly by subsequent requests. Paged management can further reduce memory fragmentation and improve cache utilization.
Illustrative code:
# Illustrative code: request handling with a shared prefix cache (not a real API)def serve_request(request, prefix_cache, kv_store):key = request.public_prefix_key # system prompt, tool definitions, project context, etc.hit = prefix_cache.lookup(key) # 1. Look up the shared prefix KVif hit is None:prefix_kv = model.prefill(request.public_prefix)prefix_cache.store(key, prefix_kv) # first request: compute and cacheelse:prefix_kv = hit# 2. Restore the session's existing history KV from the cachesession_kv = kv_store.restore(request.session_id)# 3. Prefill only the new content, avoiding recomputation of the full historynew_kv = model.prefill(request.new_turn, prefix=prefix_kv, history=session_kv)# 4. Decode incrementally and write back to the session cachetokens = model.decode(new_kv, request.gen_config)kv_store.update(request.session_id, new_kv)return tokens
6.3 Scheduling Strategy
- Chunked prefill: split an over-long prompt into several chunks and interleave them with decode requests, preventing a long request from monopolizing compute for an extended period;
- Prefill/Decode disaggregated deployment: the two workload types have different resource needs, so compute and bandwidth can be configured separately and scaled independently according to each bottleneck;
- Continuous batching: dynamically assemble batches to improve GPU utilization.
6.4 Precision Evaluation and Regression
Both quantization and sparsification introduce precision loss, so an evaluation system must be in place before going live:
- Short-context tasks: standard evaluation sets;
- Long-context tasks: retrieval tests such as Needle-in-a-Haystack, and long-document question answering;
- Long-horizon Agent tasks: multi-step task completion rate, tool-call accuracy, and cross-turn state-retention ability.
Once precision is acceptable, measure TTFT, per-token generation latency, throughput, and per-task cost at different context lengths, forming a three-dimensional “precision-performance-cost” comparison.
6.5 Cloud Resource Coordination
Stable operation of an inference service also depends on the support of cloud infrastructure:
- Container platforms carry elastic scaling of inference instances;
- Object storage holds session snapshots and archived data;
- Message queues smooth out request peaks;
- Logging services and monitoring/alerting systems track key metrics such as TTFT, cache hit rate, and memory watermark.
7. Conclusion: Coordinated Evolution of Efficiency and Capability
The challenge that long-context and Agent scenarios pose to inference systems is essentially a tension between “context scale” and “computation and storage cost.” Resolving this tension requires work at two levels.
At the model architecture level:
- Asymmetric compute structures (CED) reduce Prefill computation;
- KV caching with cross-layer sharing and FP4 compression (CSA2) reduces global KV size;
- Decoupled knowledge memory (Engram) expands knowledge capacity at low activation cost.
At the engineering level:
- Tiered KV storage, prefix reuse, chunked prefill, and disaggregated scheduling turn architectural advantages into real service cost and response speed.
For teams building long-context or Agent capabilities, the recommendation is to start from the following points:
- First quantify your own context distribution — including input length, output length, number of turns, and concurrency scale — to find the real cost bottleneck;
- On your own business datasets, evaluate model precision and performance specifically for long-horizon tasks;
- Combined with cloud elastic resources and observability, build a continuous “cost-latency-quality” optimization loop.
One caution: the gains from architectural optimization ultimately depend on engineering execution. A low cache hit rate, an inappropriate scheduling strategy, or a missing evaluation system can all prevent paper advantages from being realized.
