Prompt Caching for LLMs: How High-Frequency Calls Can Cut Input Cost by About 90%

    Prompt caching reduces repeated charges for large, stable context by writing that context once and reading it on later calls at a much lower rate. This guide explains what caching solves, how write and read pricing typically works, why TTL matters, how to structure cacheable prompts, and which metrics and pitfalls determine whether the savings are real.

    In LLM applications, the cost driver is often not the user’s latest question. It is the large context sent again on every request: knowledge-base summaries, business rules, role instructions, tool definitions, conversation history, and code snippets that stay the same across dozens or hundreds of calls. If that text is billed as ordinary input tokens every time, the system pays repeatedly for the same input.

    Prompt caching is a general optimization for this pattern. A block of context that will be reused within a short window is written to cache first. Later requests send only what changed, and the model service reads the cached portion. Cached tokens are often priced well below standard input tokens—on some plans, about one-tenth of the standard rate. In high-frequency, short-window workloads with highly repeated context, careful use of caching can therefore bring input cost down by close to 90%.

    1. What prompt caching actually solves

    A typical LLM request can be split into three kinds of content:

    1. Stable context: system instructions, company policies, product materials, knowledge-base sections, and tool definitions.
    2. Semi-stable context: recent dialogue, the current task background, a user profile, or session state.
    3. Dynamic input: the current user question, live data, temporary parameters, and action results.

    If all three are sent in full every time, stable context is billed again and again. In enterprise applications, a fixed prompt may run to thousands or tens of thousands of tokens, while each user turn is only tens or hundreds of tokens. The part worth optimizing is the highly repeated context, not the user question itself.

    The core idea is:

    1. First request:
    2. stable context + dynamic input → create the cache and run inference
    3. Later requests:
    4. read the cached stable context + new dynamic input → run inference

    Caching does not change the model’s reasoning ability and does not automatically improve answer quality. It mainly changes how input context is transmitted and billed. Developers still have to keep cached content accurate and version-consistent, and they must control cache lifetime.

    2. What cache billing usually includes

    To tell whether caching actually saves money, split the charges. Common cache billing fields fall into two groups:

    • Cache write tokens: how many input tokens in this request were used to create the cache.
    • Cache read tokens: how many input tokens in this request were read from an existing cache.

    Cache writes are usually not free. Some services charge an extra write fee on the first creation of cached input, on top of the standard input price. A typical rule described in source material is that the first write may cost about 25% more than ordinary input, while a cache hit may cost about 10% of the standard price.

    Caching is therefore not “cheaper the moment you turn it on.” It is an economic model that depends on call frequency.

    Let:

    • S = token count of the fixed context
    • D = token count of dynamic input per call
    • N = number of calls
    • P = ordinary input price
    • cache-hit price ≈ 0.1P
    • cache-write price ≈ 1.25P

    Without caching, the cost of the fixed context is roughly:

    1. ordinary cost = N × S × P

    With caching, the fixed-context cost is roughly:

    1. cached cost = 1 × S × 1.25P + (N - 1) × S × 0.1P

    When N is small, the first write can offset part of the gain. When N is large enough, the low price of later hits dominates. For requests that are mostly repeated context, several consecutive calls inside the cache lifetime usually raise the overall savings rate quickly.

    3. Why cache lifetime matters

    Caches are not kept forever. Some model services retain them for only a few minutes—about five minutes, for example. After expiry, the next request may have to recreate the cache and pay the write cost again.

    Caching fits best when:

    • An online support agent has many turns in a short period.
    • A coding assistant rereads the same code context during one task.
    • A document analysis job asks several questions within a few minutes.
    • A real-time workflow shares the same rules and tool definitions across steps.
    • A batch job repeatedly calls the same prompt in a short window.

    It is a poor fit when:

    • Requests are far apart, well beyond the retention window.
    • The fixed context is small, so write cost and operational complexity exceed the savings.
    • Every request is entirely different.
    • Context versions change so often that hits are rare.
    • Volume is too low to amortize the first write.

    Whether caching is worth it depends less on the savings of a single request and more on how many times the same context is reused inside its lifetime.

    4. How to organize cacheable content

    Cache effectiveness depends on how stable the context is. A practical rule is: put stable content first and changing content last.

    A useful structure looks like this:

    1. [System role and behavior rules]
    2. [Fixed business rules]
    3. [Long-lived knowledge]
    4. [Tool and output-format instructions]
    5. [Current session history]
    6. [This turn's user question]

    The first four blocks are cache candidates. The last two should be assembled dynamically.

    This layout helps because:

    1. Stable content can be managed and updated in one place.
    2. Dynamic content is less likely to invalidate the whole cache.
    3. On a hit, the read range is easier to predict.
    4. The token split between fixed context and dynamic input is easier to measure.

    Do not mix timestamps, random request IDs, live status, or the current user question into the fixed prefix. If the prefix changes on every request, the service may not treat it as the same cached context, and the hit rate will fall.

    5. A cache hit does not mean the content is still correct

    A cache read returns previously stored context, so versioning matters. If company rules change but the old text is still cached, the model may keep answering under the expired rules.

    Common safeguards:

    1. Add a version marker to the context

    Include a business version in the fixed context:

    1. Rules version: 2025-01
    2. Knowledge base version: v3

    When core content changes, update the version marker so new requests naturally form a new cache entry.

    2. Isolate tenants

    If users, departments, or tenants have different business rules, they should not share one cache. A cache key should at least consider:

    1. tenant ID + scene ID + context version

    Otherwise permission boundaries can blur and business content can leak across tenants.

    3. Be careful with sensitive data

    Do not put unnecessary personal data, credentials, access keys, internal secrets, or short-lived authorization into context that will be reused. Caching is a cost optimization. It does not replace access control, data minimization, or security auditing.

    6. Designing the cache call flow in code

    At the engineering level, the request flow can be abstracted as:

    1. 1. Hash the fixed context
    2. 2. Build a cache key from tenant, scene, and version
    3. 3. Look up local cache state
    4. 4. If the cache does not exist, create it
    5. 5. If it exists and has not expired, reuse it
    6. 6. Append dynamic input to the request
    7. 7. Record write tokens, read tokens, and total tokens
    8. 8. Keep tuning based on hit rate and cost metrics

    Illustrative pseudocode:

    1. context_key = hash(tenant_id + scene + context_version)
    2. cache = get_cache(context_key)
    3. if cache is None or cache.is_expired():
    4. cache = create_cache(
    5. stable_context=stable_prompt,
    6. expire_seconds=300
    7. )
    8. response = invoke_model(
    9. cached_context=cache,
    10. dynamic_input=user_input
    11. )
    12. record_metrics(
    13. cache_write_tokens=response.cache_write_tokens,
    14. cache_read_tokens=response.cache_read_tokens,
    15. total_input_tokens=response.input_tokens
    16. )

    Do not rely only on local cache state. The server-side cache can expire early or change under capacity policy. The program must handle “expected hit, actual miss” and recreate the cache automatically after a miss.

    7. Metrics you must monitor

    To verify that caching works, track at least the following.

    Cache hit rate

    1. cache hit rate = cache read tokens ÷ total cacheable tokens

    A low hit rate usually means frequent expiry, an unstable context prefix, a poorly designed cache key, or request gaps longer than the TTL.

    Share of fixed context

    1. fixed-context share = fixed-context tokens ÷ total input tokens

    If fixed context is only a small share of input, even a high hit rate may save little money.

    Average cost per request

    Do not look only at the cache-read price. Include write cost, dynamic input cost, and the cost of failed retries.

    Cache rebuild count

    A high rebuild count means the cache lifetime does not match the call pattern, or context versions change too often.

    Error rate after cache expiry

    Expiry itself is not necessarily a failure, but the application must fall back to recreation. A missing cache must not fail the business request.

    8. Common mistakes and how to avoid them

    Mistake 1: Cache every prompt.
    Caching is a poor fit for content that changes often. Prefer blocks that are stable, large, and reused many times.

    Mistake 2: Look only at the cache unit price and ignore write fees.
    Creating a cache can carry an extra charge. Account for the number of calls inside the lifetime.

    Mistake 3: Ignore expiry.
    If calls are more than a few minutes apart, the cache may be rebuilt constantly. Reassess whether caching belongs in that flow.

    Mistake 4: Use an overly simple cache key.
    A scene name alone can mix versions and tenants. The key should include the isolation dimensions you need.

    Mistake 5: Skip the token stats returned by the service.
    Only write tokens and read tokens together show the real hit behavior. Guessing savings from intuition is not enough.

    9. A practical go / no-go check

    Use questions like these to judge a scenario quickly:

    1. Is the fixed context large enough?
    2. Will the same context be reused within a few minutes?
    3. Can the request prefix stay stable?
    4. Is there no clear security risk in the cached content?
    5. Can the application handle expiry, misses, and rebuilds?

    If most answers are yes, caching is usually worth implementing. For enterprise workloads, logs, alerts, and cost analysis—on a cloud platform such as Baidu AI Cloud, for example—can be used to watch token consumption and cache hits over time. How you connect those tools depends on the model service you actually use; follow that service’s documentation.

    Overall, prompt caching is not a simple switch. It is an engineering design around context splitting, lifetime management, cost accounting, and observability. For applications that reuse a large context many times in a short window, a cache hit may cost about one-tenth of ordinary input, and total cost can fall sharply. If call frequency is low, content changes quickly, or the cache keeps expiring, the savings shrink.

    The safest path is to pilot one high-repetition workflow, record write tokens, read tokens, hit rate, average request cost, and response behavior, then adjust cached content and call strategy from real data. That avoids blind adoption and turns LLM cost control into a measurable, ongoing engineering practice.