Why Long-Running AI Agents Burn Through Tokens and Compute: Cost Control Explained

    Long-running agent tasks consume tokens and compute far faster than most teams expect. This article explains why consumption ramps up so quickly, where rate limits come from, and how to reduce costs.

    This article explains why long-running agent tasks consume tokens and compute so quickly, where rate limits come from, and practical ways to reduce costs.

    Why Long-Running Agent Tasks Consume So Quickly

    Attention Compute Scales with Context Length

    Mainstream large language models are built on the Transformer architecture. With standard full attention, the compute required for a sequence of length N grows at O(N²). When context expands from tens of thousands of tokens to the million-token range, the increase in compute far outpaces the increase in length. For example, doubling the sequence length roughly quadruples the compute in the attention portion.

    Two important nuances apply here. First, O(N²) describes the trend of how compute grows with sequence length — it does not mean the actual cost of every call increases by the same factor. Second, memory usage grows differently. The KV cache stores the key and value vectors of all previously processed tokens, and its size grows roughly linearly with sequence length N, while also depending on the number of model layers, attention heads, head dimension, and numerical precision. Chunked computation methods such as FlashAttention avoid materializing the full N×N attention matrix, so memory pressure comes mainly from the KV cache rather than the attention matrix itself.

    With standard full attention and without optimizations such as sparse or linear attention, long contexts significantly raise the compute cost of each call. Whether these optimizations are enabled depends on the specific model and the serving implementation.

    Token Accumulation Across Multi-Turn Agent Loops