LLM API Pricing Fluctuations: The Cost Logic Behind the Numbers and Engineering Responses

1. Price Signals: Why Output Becomes the Cost Focus

Recently, an LLM service adjusted its API billing structure: the output-side unit price is several times that of the input side, while the cache-hit price is significantly lower than the regular input price. Using a typical set of numbers—output at ¥100 per million tokens, input at ¥20 per million tokens, and cache hits at ¥1 per million tokens—the three form a gradient of roughly 1001. This gradient is not arbitrary; it is determined by the computational characteristics of the inference process.

Understanding this requires returning to the basic mechanics of Transformer decoding. Input processing is parallel: the entire prompt can complete a forward pass in one shot, fully utilizing the GPU’s matrix parallelism, resulting in a lower per-token computational cost. Output, however, is autoregressive: each token generated requires a full forward pass, with the result appended to the context before the next round. This means output token generation is serial and sequential, with GPU utilization inherently lower than during the input phase, and the per-token compute time significantly higher. The output price being several times the input price is a direct reflection of this serial decoding overhead.

The low cache-hit price corresponds to another optimization path: prefix caching. When multiple requests share the same prefix (such as system prompts, long document backgrounds, or tool definitions), the server can reuse the already-computed KV Cache, skipping the repeated prefill stage and thereby substantially reducing computation. A cache-hit price of ¥1 per million tokens essentially encourages callers to place stable, unchanging content at the front and variable content at the back to improve cache reuse rates.

For developers, these three prices constitute a clear cost signal: output is the most expensive, regular input is next, and cache hits are the cheapest. Any cost optimization strategy should revolve around two levers: reducing output tokens and improving cache hit rates.

2. Decomposing Billing Dimensions: Three Prices, Three Resources

Mapping the three prices to resource consumption yields a clear cost attribution table:

Billing Item	Corresponding Compute Phase	Resource Characteristics	Optimization Lever
Input price	Prefill	Parallel computation, high throughput	Compress prompts, remove redundancy
Cache-hit price	Prefix reuse	Skips repeated prefill	Stable prefix first, improve reuse rate
Output price	Decode	Serial generation, high compute occupancy	Control generation length, constrain output format

This table reveals a frequently overlooked fact: many teams’ cost overruns come not from “asking too much” but from “answering too long.” When the gap between input and output prices reaches several multiples, a single verbose output can cost more than a carefully organized long prompt. Therefore, the first step in cost governance should be establishing usage observability broken down by billing item, rather than looking only at the total bill.

A practical approach is to instrument the call side and separately track the three types of token volume:

# Pseudocode: logging usage by billing dimension
def log_usage(resp):
    usage = resp.usage
    record({
        "prompt_tokens": usage.prompt_tokens,          # regular input
        "cached_tokens": usage.cached_tokens,          # cache hits
        "completion_tokens": usage.completion_tokens,  # output
    })

With this data, you can assess whether the cost structure is healthy: whether the cache-hit share is rising, whether average output tokens are out of control, and whether there is a large amount of trimmable redundant content in the input.

3. The Economic Boundary Between Subscription Quotas and Pay-as-You-Go

Beyond pay-as-you-go billing, another common model is the monthly subscription, which provides call capacity through a “fixed monthly fee plus quota multiplier.” For example, some tiers use a base quota of 1x and offer tiers at roughly 6x, 14x, and so on, with prices increasing non-linearly as quota rises. The essence of such plans is trading usage commitment for unit price discounts: users lock in capacity through prepayment, and the provider exchanges predictable load for a lower marginal price.

The key to judging whether a subscription is worthwhile is estimating your own “stable usage baseline.” The method is as follows:

1. Analyze historical usage distribution: Take the daily token consumption over the past 30 days and calculate the median and peak.
2. Distinguish baseline from elasticity: Usage near the median is suitable for subscription coverage; the portion exceeding the peak is covered by pay-as-you-go as a fallback.
3. Calculate the break-even point: Divide the monthly subscription fee by the equivalent pay-as-you-go price of its included quota to get the discount rate; if actual usage consistently exceeds the break-even volume, the subscription is better.

Two reverse risks to watch for are “quota waste” and “overage premiums.” If actual usage is far below the subscription quota, the unit cost may actually be higher than pay-as-you-go. If peaks frequently exceed the subscription cap, the overage unit price may not be advantageous. Therefore, subscription strategy should be paired with usage monitoring, with tier selection reviewed quarterly.

4. Four Engineering Paths to Cost Optimization

Path 1: Improve Cache Hit Rates

Placing stable content—system prompts, role definitions, tool schemas, knowledge base backgrounds—at the front of requests and user real-time input at the back is the basic technique for improving prefix cache hit rates. Further, you can “stabilize the prefix” of multi-turn conversations: fix the opening template and avoid inserting timestamp-like variable fields into system prompts.

# Bad example: variable content mixed into system prompt, breaking prefix reuse
system = f"Current time is {now}, you are an assistant..."
# Good example: stable prefix separated from variable content
system = "You are an assistant, follow these rules..."   # stable, cacheable
user_turn = f"[time]{now}\n[question]{question}"  # variable, placed later

Path 2: Constrain Output Length and Format

Output is the highest-priced portion, so “controlling generation length” should be a default design constraint. Specific measures include: specifying word limits in prompts, requiring structured output (JSON, lists) instead of prose-style answers, setting hard truncation for summarization tasks, and limiting function granularity for code generation tasks. For batch tasks, truncation and deduplication can be done in post-processing to avoid paying for useless content.

Path 3: Tiered Calls and Model Routing

Not all requests require models of equal capability. Layer tasks by complexity: route simple classification, extraction, and formatting to lightweight models; route complex reasoning and long-form generation to high-capability models. The routing layer dynamically selects based on task type, input length, and historical success rate, significantly lowering the average unit price while maintaining quality.

Path 4: Budget Guardrails and Alerts

Set budget thresholds and rate limits per project and per caller at the gateway layer, paired with usage dashboards for day-over-day alerts. When the average output token count for a certain type of request rises abnormally, promptly identify whether it is prompt degradation or task type drift. The value of budget guardrails is not “saving money” but making cost changes attributable and intervenable.

5. Architectural Resilience for Pricing Volatility

Price adjustments are the norm in the cloud services market, and a single rise or fall should not be the sole basis for architectural decisions. A more robust approach is to abstract the “billing structure” into configurable parameters rather than hard-coding them into business logic:

• Make model selection, maximum output length, and caching strategy into configuration items, so that when prices change, only configuration needs adjustment rather than code;
• Maintain a lightweight cost estimator at the application layer, estimating the cost range of each request before calling, and degrading or queuing when thresholds are exceeded;
• Retain multiple model alternatives for cost-sensitive businesses, isolating specific services through abstract interfaces to avoid binding to a single pricing system.

# Pseudocode: configurable call policy
policy = {
    "max_output_tokens": 512,
    "cache_prefix": True,
    "model_tier": "standard",   # switchable by task
    "budget_per_request": 0.05, # degrade if estimated to exceed
}

The significance of this design is that when the relative relationships among output price, input price, and cache-hit price change, teams can quickly adapt by adjusting policy parameters without refactoring the call chain. Cost optimization thus shifts from a “one-time project” to a “continuous capability.”

6. Conclusion

The numbers—output at ¥100, input at ¥20, cache hits at ¥1—are superficially prices, but in essence they are a pricing map of inference computational characteristics: serial decoding is the most expensive, parallel prefill is next, and prefix reuse is the cheapest. Understanding this map allows developers to translate cost governance into concrete actions: improve cache hit rates, compress output length, implement tiered routing, and set budget guardrails. The choice between subscriptions and pay-as-you-go should be based on actual usage distribution, making decisions with data rather than intuition. Prices will change, but the methodology of “optimizing usage by computational characteristics” will not become obsolete—that is the true foundation for weathering volatility.