LLM API Cost Optimization: Pricing, Prompt Caching, and Long Context

1. Understand the Billing Structure First

Most LLM APIs use token-based pricing. A token is a basic unit of text processed by a model. Chinese and English are tokenized differently, and the same content may produce different token counts across models. For accurate cost estimates, use the provider’s tokenizer or the usage fields returned by the API.

Common billing dimensions include:

1. Input tokens: User messages, system prompts, conversation history, and retrieved documents all count toward input usage.
2. Output tokens: The content generated by the model. Generation occurs one token at a time, so output processing is often more computationally expensive than input processing. As a result, many providers charge more for output tokens.
3. Cache writes and reads: Some providers offer caching for repeated prefixes. Writing content to a cache may cost slightly more than standard input processing, while subsequent cache reads are priced lower.

Pricing, discounts, and billing rules vary widely. This article does not provide fixed ratios or prices. Before choosing a provider, check its official pricing page and record the date of review, since prices can change frequently.

2. How to Evaluate Pricing Changes in New Model Versions

When providers release a new model version, they often change the pricing at the same time. To determine whether a price reduction is useful for your application, examine three factors:

1. Which model tier is affected: Prices can differ substantially between flagship, lightweight, and reasoning models.
2. Which usage type becomes cheaper: If your application generates long responses, a change in output pricing may matter more than a reduction in input pricing. The same applies to cache reads.
3. How many tokens the new model uses for the same task: A new model may have a higher per-token price but require fewer tokens to complete the task, resulting in a lower total cost.

Use representative production requests for a small-scale comparison. Record average input tokens, output tokens, success rate, and latency, then calculate the cost per task. This is more reliable than comparing list prices alone.

3. Prompt Cache Hit Conditions and Prompt Design

Caching is useful when system prompts remain stable, background documents do not change, or earlier parts of a multi-turn conversation are no longer modified. Examples include a customer service bot’s persona instructions, coding standards for code review, and fixed explanatory sections in a RAG workflow.

Caching usually depends on prefix matching. To improve the chance of a cache hit, check the following conditions. The exact rules depend on the target provider’s documentation:

• The prefix must be identical: A changed punctuation mark, space, or timestamp may invalidate the prefix.
• Place stable content first: Put fixed content at the beginning of the messages and changing user input at the end.
• Meet the minimum token requirement: Prefixes that are too short may not be cached.
• Check the retention period: Cached content may expire after a period of inactivity and need to be written again.
• Confirm the billing model: Check whether cache writes are billed separately, how cache reads are priced, and whether additional storage fees apply.
• Implement a fallback path: If a cache miss occurs, the application should process the request normally instead of failing.

The following example shows one way to organize a prompt. Update the parameter names, cache options, and response fields according to the target provider’s API documentation.

# Keep stable content at the beginning to improve prefix-cache hits
SYSTEM_PROMPT = """
You are a professional code review assistant.
Use the following team coding standards when reviewing code.
[Insert the fixed coding standards here]
"""
def build_messages(user_query: str) -> list[dict]:
    # Keep changing content only in the final user message
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_query},
    ]
def review_code(client, model_name: str, user_query: str):
    messages = build_messages(user_query)
    # Use cache-related parameters documented by the provider.
    # The request should still work when the cache is not hit.
    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
    )
    usage = response.usage  # Save usage data to measure cache-hit rates
    return response, usage

This structure separates stable and changing content while preserving usage information for later analysis.

4. Choosing Between Long Context and RAG

Some models support very large context windows, allowing an application to send a large codebase, document, or collection of materials in a single request. This lowers the development barrier, but it does not mean that content can be added without limits.

There are three main concerns:

1. Longer inputs generally increase both cost and latency.
2. Models may not use information equally well at every position in a long document. Important content placed in the middle may be overlooked.
3. Irrelevant content can interfere with the answer.

For many use cases, a better approach is to combine retrieval with a focused amount of context: first retrieve relevant passages, then provide the model with enough surrounding information to answer the question.

When evaluating the two approaches, consider:

• How many tokens of information does the task typically require?
• What is the cost limit per request?
• What is the maximum acceptable latency?
• Do the retrieved documents actually contain the answer? Evaluate this with a set of labeled questions.
• Can the model find key information in the middle of a long document? Validate this with a targeted test set.

If the answers point to large information requirements, cost sensitivity, and a need for precise citations, RAG is usually the stronger option. If the task requires analyzing a complete document in one pass and request volume is low, sending the document through a long-context model may save development time.

5. Model Selection Comparison Template

Create a comparison table for each candidate model, and record the data source and date of review. The values below are placeholders and should be filled with the actual information from the provider’s official documentation.

Model	Context window	Input price	Output price	Cache write price	Cache read price	Fast-mode limitations	Source and date
Model A	To be filled	To be filled	To be filled	To be filled	To be filled	To be filled	Documentation link, review date
Model B	To be filled	To be filled	To be filled	To be filled	To be filled	To be filled	Documentation link, review date

Keeping the source link and review date makes it possible to verify each value later.

6. Fast Modes: Test Before Using Them

Some providers offer fast or low-latency modes that may require additional payment or have usage restrictions. Whether to enable such a mode should depend on how sensitive the application is to latency.

The evaluation process is straightforward. Send the same request set through both standard and fast modes, then record time to first token, total completion time, and cost per request. Enable the fast mode only when the latency improvement justifies the additional cost or directly affects the user experience.

The performance difference depends on the model, region, workload, and request length. Data from another application should not be applied without testing.

7. Use Monitoring to Control Costs

Cost optimization is not a one-time design task. It requires continuous observation. At a minimum, track the following metrics:

1. Input, output, cache-write, and cache-read tokens for every request.
2. Cache-hit rate, calculated as cache-read tokens divided by input tokens.
3. Average cost per task, broken down by business function.
4. Requests that exceed defined thresholds, such as unusually long inputs or unexpectedly large outputs.

After alerts are configured, a sudden increase in daily cost or cost per task can trigger an investigation. Possible causes include a prompt change that invalidated the prefix cache or a feature that accidentally sent excessively long documents.

8. Cloud Deployment Considerations

LLM applications are often deployed in the cloud. When evaluating a cloud platform, consider the availability zones and latency of its model services, the rate-limiting and authentication capabilities of its gateway, integration with object storage and vector databases, and the granularity of billing and alerting.

The choice should reflect the team’s existing technology stack and compliance requirements. Compare actual provider pricing rather than relying only on general estimates.

API gateway rate limits should be aligned with the model provider’s concurrency quotas. If sudden traffic triggers provider throttling and the application responds with excessive retries, costs can increase rapidly. Use exponential backoff and set a maximum retry count.

Conclusion

LLM API cost optimization requires considering billing rules, cache-hit conditions, and context strategies together. Model prices and capabilities will continue to change, so teams should do two things consistently: compare models using real request samples and validate assumptions through monitoring.

Placing stable content first, retrieving only what is needed, and recording usage are practical techniques that apply across many providers and model versions.