LLM Price Cuts Explained: How Technical Optimizations Reduce Inference Costs

    Lower LLM inference costs come from improvements in computation, memory use, and data movement. This article explains prefill and decoding, attention optimizations, KV cache management, and prefix caching—and how these factors help explain separate input, output, and cached-token pricing.

    1. Why LLM Prices Keep Falling

    The unit price of large language model services has been trending downward. The underlying reason is straightforward: each request incurs costs for computation, GPU memory capacity, memory bandwidth, and allocated operational overhead. A structural improvement in any of these areas can give providers room to adjust pricing.

    For developers, understanding these cost drivers is more useful than comparing price figures alone. It helps assess whether lower prices are sustainable and which billing approach best fits a particular workload.

    This article does not evaluate any specific service’s prices. Instead, it examines the technical foundations of inference costs, common optimization techniques, and the logic behind pricing differences.

    2. The Two Stages of LLM Inference

    An LLM request typically involves two stages:

    1. Prefill: The model processes all input tokens in parallel, computes attention across its layers, and builds the key-value (KV) cache. This stage requires substantial computation and is typically compute-bound.
    2. Decode: The model generates output tokens one at a time. Each step reads the relevant KV cache from previous tokens. Individual steps involve less computation, but repeated GPU memory access often makes this stage memory-bandwidth-bound.

    Longer inputs require more prefill work; longer outputs require more decoding steps. Input tokens generally cost less to process because they can be handled in parallel. Output tokens generally cost more because generation proceeds sequentially. This helps explain why many services price input and output tokens separately.

    3. Optimizing Attention

    Standard self-attention has computational complexity that grows quadratically with sequence length. For a sequence of length n, the attention computation in a single layer is approximately O(n²). As sequences grow longer, the cost rises rapidly.

    Common optimization approaches include:

    • Sparse attention: Computes attention only between selected positions, reducing unnecessary work.
    • Sliding Window Attention (SWA): Restricts each token’s attention to a fixed window of preceding tokens. With a window size of w, attention computation falls to approximately O(n × w).
    • Grouped-Query Attention (GQA): Allows multiple query heads to share a group of key-value heads, reducing the size of the KV cache.

    Sliding window attention introduces a trade-off: information from distant positions must propagate through multiple layers. Model design therefore requires balancing window size, layer count, and model quality.

    The following code demonstrates how to construct a causal sliding window mask:

    1. import numpy as np
    2. def sliding_window_mask(seq_len: int, window: int) -> np.ndarray:
    3. """Return a (seq_len, seq_len) boolean mask; True allows attention."""
    4. idx = np.arange(seq_len)
    5. rel = idx[:, None] - idx[None, :] # Query position minus key position
    6. return (rel >= 0) & (rel < window) # Causal and within the window
    7. mask = sliding_window_mask(seq_len=6, window=3)
    8. print(mask.astype(int))

    Each row identifies the positions that a query token can attend to. This example illustrates mask construction only; it does not represent the implementation of any particular model.

    4. Reducing Data Movement: An Often-Overlooked Cost

    During inference, moving data can be a greater bottleneck than performing arithmetic. Reading model weights and KV cache data from GPU memory places significant demands on memory bandwidth. Reducing the amount of data read and written per token can therefore be as important as reducing computation.

    Common approaches include:

    • KV cache compression: Reduces the number of key-value heads or stores cached values at lower precision.
    • Weight quantization: Reduces the number of bytes read for model weights.
    • Operator fusion: Reduces how often intermediate results must be written back to GPU memory.
    • Batching adjustments: Allow data read from memory to serve more requests.

    These techniques generally aim to preserve output quality while lowering per-token hardware costs, creating room for price adjustments. Their effects on output quality vary, however, so deployment should be preceded by regression testing on real application data.

    5. Prefix Caching and Cache-Hit Pricing

    Many applications send requests with shared prefixes. Multiple requests may use the same system prompt, a conversation may repeatedly reference the same document, or several queries may use the same codebase as context.

    If the server retains the KV cache for a shared prefix, subsequent requests can reuse it rather than repeat the corresponding prefill computation. This is the basic idea behind prefix caching.

    Its effectiveness depends on two factors:

    1. Cache hit rate: The more frequently an identical prefix appears, the more computation can be avoided.
    2. Cache retention time and capacity: Cached prefixes occupy memory, so providers must balance cache capacity against the resulting hit rate.

    Cache hits substantially reduce the computation required for the reused portion of the input. Providers can therefore charge a lower rate for those cached input tokens.

    For developers, placing stable content at the beginning of a prompt and variable content at the end can often reduce total costs:

    1. [System prompt: fixed] [Reference document: fixed] [User question: changes]
    2. |<----------- Reusable cached prefix ---------->|<-- Recomputed input -->|