LLM API Pricing and Cost Optimization: How Token Billing and Context Caching Work

    As large language models move from demo prototypes to production systems, the cost structure often decides whether an application survives more than its raw model capability does. Mainstream LLM APIs meter usage by tokens rather than by request count, which means pricing depends on input and output token volume, on whether a context cache is actually being hit, and on which performance tier a workload runs on. This article breaks down those three layers: why output tokens cost several times more than input tokens, how prompt and context caching converts repeated computation into cheap cache reads, how to choose cache lifetimes, and how to route latency-sensitive traffic to faster tiers. It also covers concrete optimization tactics—structuring prompts so prefixes stay stable, constraining output length, tuning cache TTLs against measured hit rates, splitting traffic by tier, and instrumenting per-request token accounting—plus a short cost-estimation example.

    1. Why You Have to Understand the LLM Pricing Model First

    As large language models move from demo toys to production systems, the cost structure often decides whether an application can survive long term more than raw model capability does. Unlike traditional per-request billing, mainstream LLM services generally use token-based metering. Costs are therefore no longer driven by call counts, but are strongly correlated with the amount of text actually processed. Understanding this is the prerequisite for any cost optimization work.

    Here is a fact that is easy to overlook: after two rounds of refactoring, the same business workload can end up costing several times more or less, even though the calling logic barely changed. The difference usually comes from three dimensions — the pricing gap between input and output, whether the context cache is being used effectively, and whether the performance tier matches the workload. This article unpacks each of the three mechanisms in turn.

    2. The Basics of Token Billing

    A token is the smallest unit a large language model processes text in. A single Chinese character, a fragment of an English word, or a punctuation mark may be split into one or several tokens. Billing normally distinguishes two categories:

    • Input tokens: the prompts sent by the user, system instructions, conversation history, and any reference material concatenated into the context.
    • Output tokens: the content generated by the model in its reply.

    The key point is that the per-token price of output is usually significantly higher than that of input. The reason is that generation is autoregressive: the model has to compute and sample one token at a time, which is far more compute-intensive and far more demanding on GPU memory than a one-shot encoding of the input. In most services, the output price ends up being several times the input price. This directly determines where cost optimization should focus: controlling generation length is often more effective than compressing the input.

    Take a typical set of relative prices as an example (illustrative of the structure only, not any specific vendor’s quote):

    Billing item What it covers Relative cost
    Input tokens Prompts and context sent with each request Baseline
    Output tokens Content generated by the model Several times the input price
    Cache hit read Reusing an already-cached context prefix Well below the input price
    Cache write Establishing the cache for the first time Comparable to input, or slightly higher

    3. Context Caching: The Main Lever for Lowering Cost

    When an application has a large amount of repeated prefix content — for example a fixed system prompt, a long document, knowledge base fragments, or multi-turn conversation history — context caching (prompt or context caching) becomes the most important cost optimization technique available.

    3.1 Cache Hits vs. Cache Writes

    The caching mechanism introduces two billing items that did not exist before:

    • Cache write: when a prefix appears for the first time, the service has to process it and store it in the cache. This action is normally billed at close to the input price, and writes with a longer retention period may even carry a premium.
    • Cache hit: when a later request reuses the same prefix, those tokens do not need to be recomputed and are billed at a price far below the standard input price. In many implementations, the hit price can be as low as one tenth of the standard input price.

    In essence, this turns “repeated computation” into “cheap storage reads”. For long-prompt and long-document scenarios, the higher the cache hit rate, the lower the overall cost.

    3.2 Choosing a Cache Lifetime (TTL)

    Caches are not kept forever; they are given a time to live. In practice, services commonly offer multiple tiers side by side, for example “short-lived caching (around 5 minutes)” together with “long-lived caching (around 1 hour)”:

    • Short-lived cache: low write cost, suited to interactive scenarios with tightly spaced, continuous requests, such as real-time customer support or back-to-back question answering.
    • Long-lived cache: higher write cost, but reusable over a much longer window, suited to scenarios where requests are sparse but the prefix is fixed, such as scheduled batch jobs or daily report generation.

    The decision can be reduced to a simple test: only if the interval between repeated requests exceeds the lifetime of the short-lived cache does the long-lived cache premium get amortized by subsequent hits; otherwise you should choose the short-lived cache.

    4. Performance Tiers: Weighing Standard Against High-Speed

    Most LLM services offer different throughput tiers, typically a standard tier and a high-speed tier. The high-speed tier lowers time to first token (TTFT) and overall response time by allocating more compute, and the price for that is usually a multiple of the per-token rate — a common structure is roughly twice the standard price.

    The core decision here is not “more expensive is better”, but how sensitive the workload is to latency:

    • Interactive, end-user-facing real-time scenarios (streaming chat, code completion, voice assistants): latency directly affects the experience, so the value of the high-speed tier is perceptible.
    • Offline, batch, and background tasks (data labeling, bulk summarization, offline analysis): nobody is waiting in real time, so the standard tier should be preferred in order to keep unit cost down.

    One practical principle: do not apply a single tier to all traffic. Within the same application you can route by scenario — send the small set of latency-sensitive requests to the fast tier and let everything else use the standard tier.

    5. Cost Optimization Strategies You Can Apply

    Pulling the mechanisms above together, here are the tactics you can put to work directly in engineering practice.

    1. Structure the prompt so the prefix stays stable

    A cache hit requires an identical prefix. Put the fixed, unchanging parts (system instructions, tool definitions, reference material) at the very front of the prompt, and put content that changes on every request (user input) at the end. Once dynamic content is inserted into the middle of the prefix, everything after it loses its cache.

    1. [fixed] System role definition <- cacheable
    2. [fixed] Tool / function schema <- cacheable
    3. [fixed] Knowledge base reference <- cacheable
    4. [varies] Current user question <- not cacheable

    2. Control output length to suppress the most expensive part

    Since output tokens carry the highest unit price, generation length should be constrained explicitly. Asking for “brief answers” in the prompt, setting a max_tokens limit, or truncating during post-processing all reduce cost directly.

    3. Set cache TTLs deliberately

    Choose a short-lived or long-lived cache according to the request rhythm. Use the hit rate as an observability metric: if the hit rate stays low over time, the TTL is mismatched with the request interval, and you need to adjust it or turn off cache writes altogether so you are not paying for caches that never get used.

    4. Split performance tiers by scenario

    Route latency-sensitive and latency-insensitive traffic separately, and avoid paying the high-speed premium for offline jobs.

    5. Build cost observability

    At the calling layer, record the input, output, cache-hit, and cache-write token counts for every request, and turn that into a cost dashboard per unit of business volume. Without measurement there is no optimization — many “hidden costs” only surface once you break them out down to the cache hit rate.

    6. The Logic Behind Cost Estimation

    The pseudocode below estimates the cost of a single request and turns the concepts above into something computable:

    1. def estimate_cost(input_tokens, output_tokens,
    2. cached_tokens=0, cache_written_tokens=0):
    3. # Prices are illustrative; actual rates depend on the service provider
    4. price_in = 1.0 # standard input price (relative value)
    5. price_out = 5.0 # output is usually several times input
    6. price_cache_hit = 0.1 # hit reads are well below input
    7. price_cache_write = 1.0 # writes are close to input
    8. new_input = input_tokens - cached_tokens - cache_written_tokens
    9. cost = (
    10. new_input * price_in
    11. + cache_written_tokens * price_cache_write
    12. + cached_tokens * price_cache_hit
    13. + output_tokens * price_out
    14. )
    15. return cost

    This code makes one intuition obvious: raising the share of cached_tokens and shrinking output_tokens are the two most effective directions for reducing total cost.

    7. Summary

    LLM cost is not an uncontrollable black box. It is an analyzable and optimizable system made up of token metering, the pricing gap between input and output, context caching, and performance tiers. For developers, three ideas are worth internalizing: output is more expensive than input, cache hits are the main lever for lowering cost, and performance tiers should be chosen by scenario rather than by habit. Build prompt structure, cache strategy, and cost observability around those three points, and you can keep inference spend in a range that matches the scale of the business while preserving the user experience.