Cutting LLM Inference Costs for Long Contexts: Four Architectural Optimization Paths

I. Background: Agentic Workloads Reshaping Inference Cost Structures

Large model applications are transitioning from simple “Q&A” to “continuous operations.” A complex software engineering task might involve hundreds of tool calls, cumulative terminal outputs of hundreds of thousands of tokens, and the retrieval and reading of an entire codebase. Deep research tasks require maintaining a complete chain of evidence across multiple rounds of searching and reading. GUI automation and enterprise workflows demand that models continuously retain and utilize early interaction history over ultra-long sessions.

The common characteristic of these workloads is that context length expands from thousands of tokens to hundreds of thousands or even millions of tokens, and historical information must be retained across turns and repeatedly accessed.

Traditional Decoder-only Transformer architectures hit two walls in this scenario:

• Prefill Computation: Assuming a model has L layers and a prompt length of N, each layer must process the full prompt. The primary computation scales linearly with N×L, compounded by the attention term O(N²). This directly determines Time To First Token (TTFT) and is the most glaring source of latency in long-prompt scenarios.
• KV Cache: Each layer independently stores K/V values, continuously consuming GPU HBM capacity and bandwidth as context grows. In multi-turn conversations, fault recovery, and cross-instance scheduling, this triggers a chain reaction of overhead across host memory, SSDs, and network transfers.

Recent technical disclosures from next-generation models in the industry converge on four optimization dimensions. This framework is equally applicable for any team evaluating its own inference architecture:

1. The size of each KV entry (byte-level compression);
2. How many historical positions actually participate in attention computation (sparsification);
3. How many KV copies are redundantly saved across layers (cross-layer reuse);
4. How many layers require full execution during the Prefill phase (asymmetric computation).

II. Calculating the Costs: Quantifying the Two Major Bottlenecks

Before optimizing, it is essential to establish a cost model.

2.1 Prefill: A Compute-Bound Bottleneck

Ignoring the attention term, the primary matrix multiplication compute for Prefill can be estimated as:

FLOPs_prefill ≈ 2 × N × P_act + O(N² × d)

Where P_act is the activated parameter count per token (much smaller than total parameters in MoE models), and d is the hidden dimension. Taking a MoE model with ~16B activated parameters and a 100k token prompt as an example, the linear term alone is roughly 3.2×10¹⁵ FLOPs. If the activated parameters per token drop to 8B, this term is nearly halved. This is the direct economic significance of “reducing activated parameters in the Prefill phase.”

Note that when N reaches the 100k or 1M scale, the O(N²) attention term can no longer be ignored, which is the fundamental reason various sparse attention and position selection mechanisms exist.

2.2 KV Cache: Dual Pressure on Capacity and Bandwidth

Under traditional architectures, the byte size of the KV Cache per token can be expressed as:

def kv_bytes_per_token(n_layers, n_kv_heads, head_dim, dtype_bytes):
    # K and V are each stored once
    return 2 * n_layers * n_kv_heads * head_dim * dtype_bytes
# Example: 48 layers, 8 KV heads, head_dim=128, FP16 storage
print(kv_bytes_per_token(48, 8, 128, 2))   # ≈ 196,608 Bytes/token

At this scale, a 1 million token request requires nearly 190GB for its KV Cache alone, not accounting for concurrency. Under concurrent servicing, the KV Cache rapidly exhausts HBM, forcing the system to reduce batch sizes and sacrifice throughput. Meanwhile, to support recovery and scheduling for multi-turn sessions, KV data must tier between HBM, host memory, and SSDs, or even transfer across nodes, further amplifying costs.

III. Four Architectural Optimization Paths

3.1 Asymmetric Prefill: Prompt Tokens Don’t Need the Full Network

The key observation is that the bottleneck nature of Prefill and Decode differs: Decode computes one token per step and is memory-bound; Prefill processes the entire prompt and is compute-bound. Furthermore, the vast majority of tokens in a prompt primarily serve as “attention context” for subsequent generation and do not need to retain complete representations at every layer.

Based on this observation, a new class of “causal encoder-decoder” style architectures adopts asymmetric computation: most prompt tokens only need to pass through the first half of the network during Prefill, while only tokens on the generation path traverse the full network. Using magnitudes disclosed in public technical reports as an example, activated parameters per token can drop from ~16B in the Decode phase to ~8B in the Prefill phase. Prefill computation is nearly halved, and TTFT improves significantly.

On the engineering side, there are two natural companions: Chunked Prefill (slicing long prompts and mixing them with Decode in batch scheduling to improve utilization) and PD Separation (using different instance pools for Prefill and Decode, scaling them independently).

3.2 Cross-Layer Reuse and Sparse Attention: Store Less, Compute Less

Two complementary methods address “redundant storage” and “redundant computation” respectively:

• Cross-Layer Shared Master KV: Adjacent layers share the same master KV, compressing the layer factor from L to L/g. This is based on the fact that deep and shallow attention distributions overlap significantly, making independent per-layer storage redundant.
• Indexer + Top-K Sparse Attention: Maintaining a lightweight index K for long contexts, it first coarsely filters the Top-K historical positions most relevant to the current query, then executes fine-grained attention only on these positions. This converges the “number of positions participating in attention” from N to a fixed budget. A further optimization is reusing the index K and Top-K selection results across layers, sharing the same index and filtering results to avoid redundant indexing and filtering per layer. Both cache and indexing computation decrease simultaneously.

3.3 Low-Bit Quantization: Compressing Each KV Entry to the Extreme

KV Cache quantization evolves along FP16 → FP8 → FP4. FP4 compresses single-element storage from 2 bytes to 0.5 bytes. Combined with block-wise scaling and outlier protection, it achieves about a 4x compression in KV size with controllable precision loss. A representative magnitude from public technical reports shows global KV Cache compressed from ~3.5KB/token to ~0.9KB/token.

The benefits are multiplicative: smaller KV → lower HBM usage → larger concurrent batch sizes → higher per-GPU throughput. Simultaneously, the bandwidth overhead of reading KV during attention decreases, improving Time Per Output Token (TPOT) in the Decode phase.

3.4 Decoupling Knowledge Memory from the Main Backbone: Separating Capacity from Cost

MoE already achieved one decoupling: total parameters can be large while activated parameters per token remain small (“separation of capacity and computation”). Next-generation models go further by stripping some knowledge memory from the main forward computation into independent modules. These are mounted on-demand with low activation overhead, expanding knowledge capacity without linearly increasing inference costs.

This solves a different dimension of the problem compared to Retrieval-Augmented Generation (RAG): RAG puts knowledge into context, consuming window length and Prefill compute; decoupled knowledge memory puts knowledge into parameters, consuming minimal activation overhead. In long-context Agent scenarios, the two are complementary, not substitutive.

IV. Combined Effects: Reducing Every Term in the Cost Formula

Mapping these methods back to the ledger in Section II, we see that each hits an independent term:

• Asymmetric Prefill → Compresses the linear main term of Prefill compute (activated parameters nearly halved);
• Sparse Attention + Index Reuse → Compresses the attention term and the “number of positions participating in attention”;
• Cross-Layer Sharing + FP4 (aided by low-rank latent space compression) → Compresses KV byte size, from hundreds of KB per token to under 1KB;
• Decoupled Knowledge Memory → Expands knowledge capacity without increasing activation compute.

Stacked together, frontier models can now support 1M token contexts while keeping the resident HBM global KV Cache at ~1KB per token, maintaining capability improvements in complex reasoning, software engineering, and long-horizon Agent tasks. This proves architectural optimization does not necessarily sacrifice capability; the key is compressing “redundancy” rather than “information.”

Multimodality makes these optimizations even more valuable: native visual understanding means documents, screenshots, and UI interfaces become first-class inputs, significantly increasing the proportion of visual tokens in the prompt. Every saving on the Prefill side is multiplied.

V. Engineering Implementation Advice for Developers

1. Calculate costs before selection: Estimate Prefill FLOPs and KV volume using real workloads (average/max context, concurrency, turns) based on the formulas in Section II. Avoid choosing models and instance specs based on intuition.
2. Track three key metrics: TTFT (dominated by Prefill), TPOT (dominated by KV bandwidth), and comprehensive cost per million tokens (including tiered storage and transfer overhead).
3. Session management: Leverage prefix caching to reuse system prompts and stable context. Apply structural pruning to ultra-long sessions (summarizing tool outputs, folding old turns) rather than indiscriminate truncation.
4. Tiered storage: Design a flow strategy for KV across HBM → host memory → SSD, balancing recovery latency against capacity costs.
5. Deployment topology: Long-context, Prefill-heavy workloads suit PD separation + Chunked Prefill. MoE models require attention to expert parallelism and load balancing.
6. Leverage managed services for quick validation: If you are not building your own inference stack, use cloud-based managed LLM inference services (such as the high-performance model inference services provided by Baidu AI Cloud), leaving underlying optimizations to the infrastructure layer so you can focus on application logic.
7. Evaluate long-horizon capabilities: Beyond standard accuracy, focus on the success rate and consistency of long-document Q&A, repository-level code tasks, and multi-turn tool calls. Beware of the “large window, low effective utilization” trap.

VI. Conclusion

In the era of long-context Agents, inference cost is no longer just about “having enough GPUs,” but rather an architectural design issue. “How large is each KV entry, how many positions participate in attention, how many layers redundantly save KV, and how many layers need full execution”—these four questions form a practical analytical lens. It is expected that cross-layer sharing, lower-bit cache compression, and further decoupling of knowledge and computation will remain evolutionary directions for next-generation models. For developers, understanding the cost models behind these optimizations is a prerequisite for effective model selection, stress testing, and cost governance.