Prompt Caching: Engineering Practices for Reducing LLM Call Costs

Prompt caching is a request optimization technique for large language model applications. Its core idea is to reuse prompt content that appears repeatedly across requests and changes infrequently, so the model service does not need to process the exact same context on every call. For applications that include long system instructions, business rules, knowledge base fragments, tool definitions, role settings, or fixed formatting requirements, this mechanism can reduce token consumption caused by repeated input and can lower time to first token (TTFT) or overall response time.

Unlike browser caches or database caches in the traditional sense, prompt caching caches part of the context in a model request and its preprocessing results. It is not equivalent to storing the model’s final answer directly, and it does not automatically replace the inference process. Whether caching takes effect depends on whether the request prefix is stable, whether the cache boundary is reasonable, whether the cache lifecycle is still valid, and whether the current request meets minimum length and consistency requirements.

1. Core Problems Solved by Prompt Caching

LLM applications typically combine multiple types of content into a single request:

1. System-level instructions, such as role definitions, output specifications, and safety requirements;
2. Business rules, such as approval workflows, field explanations, and calculation logic;
3. Fixed reference materials, such as product manuals, policy texts, and internal knowledge;
4. Current session content, such as user questions, conversation history, and tool results;
5. Parameters for the current call, such as task goals, format requirements, and temporary conditions.

The first three categories often remain unchanged for a period of time, while the last two change frequently. If every call submits the full context and reprocesses all of it, the application incurs high repeated computation costs. The value of prompt caching is to establish a clear boundary between static and dynamic content: the static part enters the cache, while the dynamic part continues to be processed in real time for each request.

For example, a document question-answering system may include the same rule descriptions and tens of thousands of words of documents in every request, with only the user question changing. With proper configuration, the system can treat the rules and documents as a cached prefix and place only changing content such as the user question and conversation history after the cache boundary. This preserves context completeness while reducing repeated processing.

2. Basic Working Principles

Prompt caching generally operates through the following flow.

2.1 Marking Cacheable Content

The application explicitly indicates in the request structure which content can enter the cache. A common approach is to set cache control markers near messages, tool definitions, or context fragments. Such a marker does not mean that all content will be stored permanently; it declares to the server that the marked part has high reuse value.

An illustrative structure is shown below:

{
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Fixed system rules, business instructions, and reference materials",
          "cache_control": {
            "type": "ephemeral"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "The question to be processed in this request"
    }
  ]
}

The structure above is only intended to illustrate the design approach. Actual field names and request formats should follow the model service interface being used. Developers should focus on the cache boundary rather than mechanically copying example fields.

2.2 Generating Cache Keys

The server typically computes a digest or cryptographic hash over the content within the cache scope and uses it as part of the cache index. A cache key is generally related to the following factors:

• The cacheable text and its order;
• Message roles and structure;
• Tool definitions or structured output constraints;
• Model and version information;
• Organization, tenant, or access isolation scope;
• Relevant inference configuration or context version.

If any of these key factors changes, the existing cache may no longer be reusable. Using a cryptographic hash instead of the raw text as the key can reduce the risk of exposing sensitive content and makes it easier to determine quickly whether two requests share the same prefix.

It is important to note that hashing only addresses indexing and consistency checks. It does not automatically perform data anonymization. If the original prompt contains personal information, credentials, internal secrets, or other sensitive data, that data should still be cleaned, replaced, or governed according to sensitivity levels before entering the cache.

2.3 Cache Hits and Misses

When the cache prefix of a new request matches an existing record and the cache is still valid, the server can directly reuse the processed prefix. This is a cache hit. If the content appears for the first time, the prefix changes, the cache has expired, or tenant isolation conditions do not match, a cache miss occurs and the corresponding content must be processed again.

In practice, caching is often not a simple matter of “all hit” or “all invalid.” A request may contain multiple cache breakpoints, with earlier content hitting the cache and later content missing it. Therefore, cache design should place stable content as early as possible and frequently changing content later, increasing the chance of partial hits and continuous reuse.

3. Cache Boundaries and Breakpoint Design

The cache boundary is one of the most important engineering decisions in a prompt caching design. If the boundary is too small, it may not cover the main repeated content. If the boundary is too large, a small change in a dynamic field may invalidate the entire cache.

A layered approach is recommended:

• Layer 1: Fixed system instructions and safety constraints;
• Layer 2: Relatively stable business rules;
• Layer 3: Versioned reference documents or knowledge fragments;
• Layer 4: Conversation history and tool results;
• Layer 5: The user’s current question and temporary parameters.

This ordering follows the principle of “stable content first, dynamic content later.” For scenarios that require multiple cache breakpoints, content can be split by document section, tool collection, or business module. However, more breakpoints are not always better. Over-segmentation increases management complexity and may prevent each segment from reaching the minimum cache length requirement.

Some implementations use a mechanism similar to a sticky latch to record established cache breakpoints and avoid frequently switching cache locations when request patterns change slightly. Its essence is to keep the cache structure stable and reduce repeated write costs caused by breakpoint movement. Developers can also use version numbers, template IDs, or fixed ordering strategies at the application layer to achieve a similar effect.

4. Cache Lifecycle and Invalidation Policies

Caches are not permanent storage. Common lifecycle policies include short-lived caching and longer-lived caching. Short-lived caching is suitable for high-frequency continuous conversations and can reduce long-term occupancy and data retention risk. Longer lifecycles are suitable for fixed knowledge bases, batch tasks, or stable automation workflows, but they may bring higher cache maintenance costs and consistency management pressure.

Cache invalidation is usually triggered by the following factors:

1. The time-to-live has been exceeded;
2. The cache prefix content has changed;
3. The model or key inference configuration has changed;
4. The tenant, organization, or permission boundary has changed;
5. The server actively cleans up or reclaims capacity;
6. The application explicitly updates a version identifier.

If business knowledge changes frequently, relying only on natural expiration is not recommended. A more reliable approach is to include an explicit version number or publication date in the cached content. When a document is upgraded, generate a new version identifier so that old and new caches separate naturally while the old version remains traceable.

5. Cost Model and Benefit Evaluation

Prompt caching usually involves three types of cost:

• Initial write cost: incurred when the server first processes and establishes the cache;
• Cache read cost: incurred when subsequent hit requests reuse processed content, usually at a lower unit price than the initial write;
• Miss cost: incurred when the cache becomes invalid and the full prefix must be processed again, potentially costing close to ordinary input.

Therefore, caching does not necessarily save money once enabled. If a prompt segment is used only once, writing it to cache may add overhead. Only when the same prefix is reused multiple times and the initial write cost is amortized is a clear benefit more likely.

A simplified model can be used to estimate the benefit:

Normal call total cost = number of calls x full prompt processing cost
Cached call total cost =
initial write cost
+ number of hits x cache read cost
+ number of misses x miss processing cost

When evaluating results, do not look only at the token unit price. Also observe:

• Cache hit rate;
• Average input tokens;
• Time to first token and total response time;
• Number of cache writes;
• Number of cache reads;
• Number of reprocessing events caused by expiration;
• Average cost per request;
• Peak traffic caused by invalidation.

When the cache hit rate is low, prompt content changes frequently, or call frequency is insufficient, consider disabling caching or reducing the cache scope.

6. Minimum Length and Suitable Scenarios

To avoid the management overhead of caching short texts, many services impose a minimum token length for cacheable content. Different models and context types may use different thresholds, and developers should not assume that all models apply the same standard. Before configuration, consult the latest interface documentation for the relevant service and verify actual behavior through testing.

Scenarios suitable for prompt caching include:

• Enterprise knowledge base question answering;
• Long document summarization and multi-turn analysis;
• Intelligent coding under fixed coding standards;
• Automation agents with many tool definitions;
• Multi-turn customer service and workflow approvals;
• Batch processing of tasks that use the same template;
• Model calls that continuously evaluate fixed rules.

Scenarios unsuitable for prompt caching include:

• Request content is almost entirely different every time;
• The static prefix is very short;
• The interval between calls is much longer than the cache lifecycle;
• Data is highly sensitive and no isolation solution is available;
• The application cannot tolerate the brief presence of older content versions;
• Request volume is too small to amortize cache write costs.

7. Common Engineering Issues

7.1 Dynamic Fields Placed in the Cache Region by Mistake

If timestamps, random numbers, request IDs, user identity information, and real-time status are placed in the cache prefix, each request will generate a different cache key. These fields should be moved after the cache boundary or stabilized.

7.2 Unstable Document Ordering

If knowledge fragments come from a database or retrieval system and the return order has no fixed rule, the concatenated result may differ even when the content set is the same. Use deterministic ordering by document ID, update time, or predefined weights.

7.3 Lack of Version Management for Cached Content

If business rules change but old cache content continues to be reused, the model may answer based on outdated information. Rule versions, knowledge base versions, or template versions should be included in cached content and monitoring labels.

7.4 Ignoring Tenant Isolation

Multi-tenant systems must ensure that cache keys include organization or tenant isolation information. Different customers must not share caches that may contain business data. Permission changes, user identity changes, and data scope changes should also trigger reevaluation.

7.5 Focusing Only on Hit Rate, Not Answer Quality

A cache hit does not mean the content is necessarily correct. Monitoring should record both cache status and business quality metrics, such as factual consistency, format compliance rate, manual correction rate, and user feedback. If old cache content causes quality degradation, adjust versioning and invalidation policies first.

8. Recommended Implementation Steps

Step 1: Inventory static and dynamic content in requests and determine the reusable context scope.

Step 2: Desensitize, layer, and version static content, and clarify which information is allowed to enter the cache.

Step 3: Design the prompt structure according to the principle of stable content first and changing content later, and control the number of cache breakpoints.

Step 4: Validate cache hits, expiration, updates, and tenant isolation behavior in a low-traffic environment.

Step 5: Establish cost, latency, hit rate, and quality metrics, and continuously observe real benefits.

Step 6: Adjust the cache lifecycle based on call frequency and content update cycles. For high-frequency stable tasks, the cache scope can be expanded appropriately. For content that changes quickly or carries high risk, shorten the lifecycle or avoid caching entirely.

Overall, prompt caching is not a simple parameter switch. It is a systematic optimization involving prompt engineering, cost control, data isolation, version management, and observability. Only by accurately identifying stable context, planning cache boundaries reasonably, and continuously validating hit results and content consistency can teams reduce token consumption while achieving more stable response performance and more controllable operating costs.