Optimizing LLM Inference Cost with Tiered Pricing and Context Caching

Why Inference Cost Is Now a Core Architecture Concern

As large language models become embedded in production systems, inference cost has evolved from a finance-level consideration into a core architectural constraint. For applications processing millions of requests per day, even small differences in input/output token pricing can translate into significant monthly expenses. Understanding the pricing structure, caching mechanisms, and service tiers of inference APIs is now a fundamental skill for backend engineers and architects.

This article examines a representative LLM inference pricing model and addresses three key topics:

1. Why input and output tokens are priced asymmetrically in standard token billing.
2. What engineering scenarios correspond to the three billing dimensions of context caching (hit, short-term write, long-term write).
3. How the price multiplier between high-speed and standard tiers informs practical traffic routing decisions.

Token-Based Billing: Why Output Costs More Than Input

In typical inference services, billing is measured per million tokens (/MTok). In the reference pricing model, the standard tier charges ¥13.00/MTok for input and ¥68.00/MTok for output—an approximate 5.2x difference. This ratio reflects the underlying computational characteristics of each stage rather than arbitrary pricing.

Input-side computation: Input tokens primarily participate in attention key-value computation. Modern inference frameworks use batching and prefill phases with high parallelism, which amortizes the per-token cost. Therefore, input pricing is relatively low.

Output-side computation: Output tokens are generated autoregressively—each token depends on the previous one, making parallel generation inherently difficult. The generation process also requires repeated reads of model weights and KV cache, placing heavy pressure on memory bandwidth. As a result, per-token compute cost is significantly higher, justifying the higher price.

The engineering takeaway from this asymmetry is straightforward: reduce output token volume while maintaining business outcomes. Common methods include:

• Instructing the model in the prompt to return concise, structured results without redundant explanation.
• Using constrained decoding for fixed-format responses such as classification labels or JSON fields.
• Reframing output-heavy tasks as input-heavy ones, e.g., replacing generative rewriting with template-based filling plus light validation.

Context Caching: Three Billing Dimensions and Their Use Cases

Context caching is a key mechanism for reducing the cost of multi-turn conversations and long-document processing. The reference pricing model splits cache billing into three dimensions:

Billing Dimension	Standard Tier Price	Description
Cache hit	¥1.30 / MTok	Cost of reading previously cached context
Cache write (5 min)	¥13.00 / MTok	Cost of writing to short-term cache
Cache write (1 hour)	¥26.00 / MTok	Cost of writing to long-term cache

Cache hit pricing is roughly one-tenth of the input price, which is the core value proposition of caching. A hit means the system reuses existing KV states without re-encoding or re-prefilling the entire context, substantially reducing cost.

The difference between the two write durations reflects storage resource costs. The 5-minute cache suits high-frequency, continuous interactions such as a user asking follow-up questions within a few minutes. The 1-hour cache fits sessions with longer gaps, such as a user filling out a form and then returning to continue. The right choice depends on the actual session interval distribution of your business.

A common optimization is to actively use cache writes for static system prompts that are identical across requests, ensuring stable hits in subsequent calls. The following pseudocode illustrates this idea:

# Pseudocode: separate fixed prefix from dynamic content to maximize cache hits
system_prefix = build_system_prompt()   # static, suitable for caching
user_turn = current_user_message()      # changes each time
request = {
    "context": [
        {"role": "system", "content": system_prefix, "cache": "write_1h"},
        {"role": "user", "content": user_turn}
    ]
}

A common pitfall: cache effectiveness depends on prefix stability. If the system prompt contains elements that change per request—such as timestamps or random IDs—the cache will never hit, and you will only incur additional write overhead.

High-Speed Tier Pricing and Traffic Routing

In the reference pricing model, the high-speed tier costs exactly 2x the standard tier:

• Input: ¥26.00 / MTok (standard: ¥13.00)
• Output: ¥136.00 / MTok (standard: ¥68.00)
• Cache hit: ¥1.30 / MTok (same as standard)
• Cache write (5 min / 1h): ¥26.00 / ¥52.00 (standard: ¥13.00 / ¥26.00)

Notably, cache hit pricing is identical across tiers. The high-speed premium applies mainly to compute-intensive stages, not cache reads. For applications with very high cache hit rates, the additional cost of the high-speed tier may be far lower than expected.

A pragmatic traffic routing strategy:

1. Use the standard tier by default for asynchronous, batch, and non-real-time workloads.
2. Route latency-sensitive paths to the high-speed tier—for example, real-time conversational first responses or interactive completions.
3. Evaluate tier value using cache hit rates. If a workload exceeds a certain hit threshold, the high-speed tier may improve user experience at minimal extra cost.
4. Define a fallback strategy. Automatically revert to the standard tier if the high-speed tier experiences capacity issues, ensuring availability.

A Reusable Cost Estimation Framework

Combining the above elements yields a simplified cost formula:

Total cost ≈ Input tokens × input unit price
           + Output tokens × output unit price
           + Cache hit tokens × hit unit price
           + Cache write tokens × corresponding duration unit price

For practical implementation, instrument your call chain to collect the following metrics:

• Input and output token counts per request.
• Cache hit token count and ratio.
• Distribution of cache write durations.
• Call volume and cost share per service tier.

These metrics support cost attribution and can guide prompt optimization and cache policy adjustments. For instance, if a particular endpoint consistently shows high output token counts, review whether the prompt lacks output length constraints or formatting guidance.

Key Takeaways

LLM inference cost optimization is not simply about switching to a cheaper model. It requires a systematic approach that covers pricing structure awareness, cache strategy design, tier routing, and ongoing measurement. Three core principles stand out:

1. The input/output price gap is significant. Focus optimization efforts on limiting output length and improving input reuse.
2. Cache hits offer the most cost-effective path, but only when context prefixes remain stable.
3. High-speed tier pricing is explicit and predictable. Use it selectively based on latency sensitivity rather than switching globally.

Applying these principles in architecture design and day-to-day operations keeps inference costs under control while model capabilities continue to grow.