Long-Context LLMs: KV Cache Memory, Attention Complexity, and High-Concurrency Costs
As context windows grow from thousands to millions of tokens, LLMs can ingest larger codebases and document collections, but memory, latency, and cost scale with input length. This article examines KV Cache memory math, attention complexity, chunking with rolling summaries, production checks for long-running agents, and stability practices for high-concurrency serving, with formulas, a comparison table, and runnable Python code.
1. Start with the Core Tradeoff
Context windows have grown from a few thousand tokens to a million tokens, allowing a model to read an entire code repository or a batch of financial reports in one pass. This capability has real value, but its cost is direct: each additional token processed increases the memory and compute required during inference.
The engineering question is therefore usually not whether a model can read long text, but whether memory, latency, and cost remain acceptable after it does. The following sections walk through several key stages of inference.
Public reports on request volume often lack a consistent methodology, and different organizations count things differently. This article does not use such figures to make judgments; it focuses on technical issues that can be verified with formulas and experiments.
2. KV Cache Memory Math
During autoregressive generation, the model stores Key and Value vectors for every token it has already processed. This cache is the KV Cache. Its size can be estimated with the following formula:
Bytes per token = 2 × layers × KV heads × head dimension × bytes per value
Assume a model has 80 layers, 8 KV heads, a head dimension of 128, and uses FP16 storage. Each token then takes about 327,680 bytes, or roughly 320 KB. At a context length of 1 million tokens, the KV Cache alone is about 328 GB. This does not yet include model weights and activations.
This number shows that the bottleneck for long context appears first in memory. The commonly used paged memory management approach, such as PagedAttention proposed by vLLM, splits the KV Cache into fixed-size blocks and allocates them on demand, reducing memory fragmentation and over-reservation. It addresses memory utilization; it does not reduce the total size of the KV Cache.
3. Attention Compute Complexity
Standard self-attention compute scales with the square of sequence length, written as O(N²). When sequence length doubles, attention compute becomes roughly 4 times larger. Note that this is quadratic growth, not exponential growth. Actual latency is also affected by other parts of the model, hardware, and batch size, so complexity should not be equated directly with latency.
Common optimization directions are listed below:
| Technique | Main problem addressed | Applicable conditions | Limitations |
|---|---|---|---|
| FlashAttention | Reduces memory reads and writes and speeds up attention compute | Supported GPUs and kernel implementations | Does not change the theoretical O(N²) compute |
| Sparse attention | Computes attention only between selected tokens | Tasks with lower requirements for global dependencies | May lose long-distance associations |
| Ring Attention | Splits long sequences across multiple GPUs for parallel compute | Multi-GPU or multi-node environments | High communication overhead and complex deployment |
| PagedAttention | Improves KV Cache memory utilization | Inference engines that support paged management | Does not reduce the total KV Cache size |
Before choosing among these options, measure peak memory, time to first token, and throughput under real workloads, then decide the optimization order.
4. Chunking Example
The code below demonstrates a common approach: split a very long input into overlapping windows and update a summary memory chunk by chunk. The code depends only on the Python 3.8+ standard library and can run directly.
from typing import Callable, Listdef chunk_tokens(tokens: List[int], window: int, overlap: int) -> List[List[int]]:"""Split a token sequence into fixed windows, keeping overlap tokens between adjacent chunks."""if window <= overlap:raise ValueError("window must be greater than overlap")step = window - overlapchunks = []for start in range(0, len(tokens), step):chunks.append(tokens[start:start + window])if start + window >= len(tokens):breakreturn chunksdef rolling_summary(tokens: List[int],window: int,overlap: int,summarize: Callable[[str, List[int]], str],) -> str:"""Update memory chunk by chunk: pass existing memory and the new chunk to summarize."""memory = ""for chunk in chunk_tokens(tokens, window, overlap):memory = summarize(memory, chunk)return memoryif __name__ == "__main__":# Use a placeholder function to demonstrate the flow. In real scenarios, call a model to generate summaries.fake_tokens = list(range(10))def fake_summarize(memory: str, chunk: List[int]) -> str:return f"{memory}[{chunk[0]}-{chunk[-1]}]"print(rolling_summary(fake_tokens, window=4, overlap=1, summarize=fake_summarize))# Output: [0-3][3-6][6-9]
This method has clear boundaries:
- It is suitable for long-document summarization, coarse-grained retrieval, and initial screening. It is not suitable for tasks that require verbatim, precise citations.
- Information is gradually lost across multiple summary rounds. Important numbers, clause IDs, and code line numbers should retain their original positions for later verification.
- It is not equivalent to the model seeing all 1 million tokens at once. If a task depends on subtle cross-paragraph relationships, evaluate it with retrieval or a larger window.
There is also an empirical phenomenon: even with a large window, a model may use information located in the middle of a long input less effectively. This is referred to as “lost in the middle” in relevant research. Key information should therefore be placed near the beginning or end of the input, or filtered out through retrieval first.
5. Production Checklist for Long-Running Agents
Long-running agents complete tasks through multi-step reasoning, tool calls, and state persistence. Long context allows an agent to store longer task history, but longer history also makes errors and costs harder to control. The checklists below are organized by scenario.
Before launching code agents, confirm the following:
- Can the agent retrieve relevant files by dependency instead of inserting the entire repository into context at once?
- After cross-file changes, are unit tests and integration tests run?
- Does each change produce an auditable diff record, with a human confirmation step retained?
- Are keys, credentials, and configuration files filtered to prevent the model from reading them or writing them into logs?
For financial research agents, the checklist focuses on data and compliance:
- Does data access follow access control, so the agent can only read materials the user is authorized to view?
- Does every number in the report carry source, reporting period, and methodology notes, and can it be traced back to the original file?
- Is the generation process logged, including tools called, files read, and intermediate conclusions, for audit purposes?
- Is the output explicitly labeled as research support material rather than investment advice?
6. Stability Under High Concurrency
As request volume grows, system bottlenecks are usually not limited to the model itself. They also include scheduling, queuing, and resource management.
Dynamic batching can combine multiple requests arriving within a short period into one computation, improving GPU utilization. But if a very long request occupies a batch, short requests will be slowed down. Continuous batching, such as the method proposed in the Orca paper, allows requests to enter and leave a batch during generation, which mitigates this problem.
At the infrastructure level, pay attention to the following metrics and mechanisms:
- Autoscale based on GPU memory usage, queue length, and time to first token, rather than CPU utilization alone.
- Set a timeout for each request, use exponential backoff for retryable errors, and cap the number of retries to prevent traffic amplification.
- Set rate-limit quotas for tenants or API endpoints, and return clear error codes when quotas are exceeded.
- Degrade under high load, for example by switching to a shorter context or a smaller model, and inform users about changes in output quality.
- Log input length, output length, latency, and error type for each call as a basis for tuning.
7. FAQ
Does a million-token context mean unlimited context?
No. The window size is a hard limit, and memory and compute grow with length. In addition, a model does not use information at different positions in a long input evenly. Being able to fit content into the window does not mean the model can use that content reliably.
How should long-running agents be evaluated?
Use a fixed task set and record task success rate, average number of steps, tool-call error rate, cost per task, and human spot-check pass rate. After each model change or prompt adjustment, rerun the same task set so results can be compared.
How can costs be controlled under high concurrency?
Start with four areas: set input and output token limits for each request; reuse caches for identical prefixes to reduce repeated computation; route simple tasks to smaller models; and allocate quotas by business priority to prevent a single tenant from consuming all resources.
8. Conclusion
Long context allows a model to process more information, but pressure on memory, latency, and cost rises accordingly. When selecting an approach, first measure bottlenecks with real data, then decide whether to use paged management, sparse attention, chunking, or multi-GPU parallelism. For long-running agents, permissions, traceability, and evaluation systems are as important as model capability, and they should be designed together before launch.
