LLM API Cost Evaluation: Billing Metrics Breakdown and Cost Optimization Practices

Whenever a new flagship large language model is released, community discussion tends to focus on price first. Not long ago, after a vendor announced pricing for the official version of its flagship model, many people compared it against similarly priced models on the market quoted in USD, using the same billing basis, and found that the output-side unit price alone could differ by several times—leading to the conclusion that it was “very cheap.” The direction is right, but for engineering teams it is not enough. The list price of an API is only a “unit price.” What you actually pay is the product of unit price × usage structure × calling behavior. Two teams using the same model—one doing retrieval-based Q&A, the other doing long-form writing—may see monthly bills that differ by an order of magnitude. Therefore, “how to evaluate the cost of a large model API” is fundamentally an engineering accounting problem, not a matter of reading a price sheet.

1. Understanding Billing Metrics: What the Three Pricing Tiers Actually Mean

Mainstream large model APIs generally charge by token. A token is the smallest unit of text a model processes: in English, roughly 1 token ≈ 0.75 words, while in Chinese, one character typically corresponds to 1–2 tokens, depending on the tokenizer. Price sheets usually use “per million tokens” as the unified billing unit and divide pricing into three tiers:

1. Input price: tokens consumed by the prompt sent to the model (system instructions, reference context, user question).
2. Output price: the billing tier for model-generated content. Output being significantly more expensive than input is an industry-wide phenomenon, because the generation phase requires token-by-token autoregressive decoding, with compute and VRAM overhead far exceeding the one-time reading of input.
3. Cache-hit price: when a request prefix repeats a recent request, the server can reuse already-computed intermediate results, and this portion is billed at the cache tier. Taking a typical billing structure as an example—input at 6 CNY per million tokens, output at 18 CNY per million tokens, and cache hits at 0.3 CNY per million tokens—under the same billing basis, the cache-hit price is only 5% of regular input, which shows the vendor is explicitly encouraging developers to stabilize frequently repeated prefixes.

Once you understand these three tiers, many “pricing mysteries” resolve themselves: some models are cheap on input but expensive on output, some apply tiered surcharges for long context windows, and some have very short cache validity periods. Comparing only total price without looking at the billing basis is bound to be misleading.

2. The Right Way to Compare Across Models: Define Your Workload Profile First

The first step in comparison is not copying a price sheet, but answering “what do my requests look like?” Define two key ratios:

• Input-to-output ratio: the ratio of input tokens to output tokens. Retrieval-based Q&A scenarios typically carry large reference passages and produce short outputs, so the ratio can exceed 10:1; open-ended writing and code generation may approach 1:2.
• Cache hit rate: the proportion of the request prefix that can be stably reused. Customer service bots and applications with fixed system prompts and document libraries can achieve relatively high hit rates.

With a profile in hand, you can calculate a “blended unit price.” Using the typical billing structure above, assuming a business input-to-output ratio of 4:1 and no caching:

Blended unit price = (4 × 6 + 1 × 18) / (4 + 1) = 8.4 CNY / million tokens

Applying this same basis to all candidate models is what makes the conclusion meaningful. Consider a common real-world situation: a flagship model on the market quoted in USD has an output price of 10 USD per million tokens, which at roughly 7.2 CNY per USD is about 72 CNY—4 times the 18 CNY output tier. If the two models perform comparably on your evaluation set, the cost difference on the output side alone is enough to affect budget decisions.

When comparing, also verify a few easily overlooked points: whether the billing unit is consistent (converting between per-million and per-thousand tokens); whether pricing is tiered by context length; the actual discounted price of free quotas, tiered discounts, and prepaid resource packages; and whether failed requests and rate-limit retries are billed.

3. Doing the Math: Building a Reusable Cost Model

It is advisable to codify the price sheet into an internal tool with a short piece of code, so you don’t have to do mental arithmetic every time:

# Unit prices: CNY / million tokens (example values; replace with actual price sheet)
PRICE_INPUT = 6.0
PRICE_OUTPUT = 18.0
PRICE_CACHE = 0.3
def estimate_cost(input_tokens, output_tokens, cached_tokens=0):
    """cached_tokens: prefix tokens that hit the cache, split out from total input"""
    normal_input = input_tokens - cached_tokens
    return round(
        normal_input / 1e6 * PRICE_INPUT
        + cached_tokens / 1e6 * PRICE_CACHE
        + output_tokens / 1e6 * PRICE_OUTPUT,
        4,
    )
# A typical RAG Q&A call: 8000 input tokens (6500 hit cache), 600 output tokens
print(estimate_cost(8000, 600, cached_tokens=6500))  # ≈ 0.0218 CNY
print(estimate_cost(8000, 600))                       # 0.0588 CNY

For the same call, the billing gap with and without caching is immediately clear: 0.0218 CNY versus 0.0588 CNY, a savings of about 63%. Going further, you should tally real token consumption from call logs, establish a monthly cost baseline, and break costs down by business dimension: cost per session, cost per active user, cost per task (each summarization, each completion). General-purpose cloud logging and monitoring/alerting capabilities can handle this kind of statistics and alerting without reinventing the wheel.

4. Cache Hits: The Most Underestimated Cost Reduction Lever

The cache unit price is an order of magnitude lower than regular input, making it the biggest dividend in the current billing system—yet many teams have not seriously exploited it. The key to using caching well lies in prefix stability:

1. Put stable content first: system prompts, few-shot examples, reference documents, and other unchanging content go at the front of the prompt; user questions, timestamps, and other volatile content go at the end.
2. Eliminate prefix randomness: mixing the current time, random IDs, or personalized fields into the prefix will misalign it and cause the hit rate to plummet.
3. Watch the cache lifecycle: caches usually have a validity period, and misses beyond that window require rewriting; some services also have billing policies for the first write, which should be factored into capacity planning.
4. Validate with data: extract cache-hit data from the usage fields of responses, incorporate “hit rate” into routine monitoring, and iterate continuously.

5. Systematic Cost Reduction: From Calling Behavior to Architecture Design

With unit price covered, the remaining optimization targets “usage structure × calling behavior”:

• Tiered model routing: flagship tiers handle complex reasoning and long-form tasks, while simple classification, extraction, and formatting are relegated to lightweight tiers. The unit prices of the two tiers often differ by orders of magnitude, making the routing layer the highest-value optimization point.
• Context slimming: for retrieval augmentation, feed only top-k relevant passages rather than the entire corpus; compress multi-turn conversation history into summaries; set a maximum context threshold and truncate or segment beyond it.
• Output control: limit maximum output length via parameters, and require bullet-point or structured answers in prompts. Output is the most expensive tier, so saving output is the most cost-effective.
• Batch processing and async: offline tasks such as corpus cleaning and batch summarization should use asynchronous batch channels where possible; such channels typically offer discounts or higher throughput ceilings in the industry.
• Application-layer result caching: directly reuse historical results for identical or highly similar requests, paired with object storage for persistence—ideal for scenarios like “querying the same batch of documents and asking similar questions.”
• Retries and circuit breaking: limit retry counts with exponential backoff, and set budget circuit breakers per user or per task to prevent runaway costs during failures.

6. Beyond Price: What Else to Evaluate When Choosing a Model

Cost is only meaningful after effectiveness has been validated, and the decision requires several more items:

1. Meeting quality standards is a prerequisite: validate quality with your own evaluation set rather than marketing numbers, then compare prices among those that qualify.
2. Stability and quotas: rate-limit thresholds, concurrency caps, and peak-hour latency directly affect the online experience and indirect costs.
3. Capability fit: whether function calling, structured output, and long-context support meet the needs of your pipeline.
4. Compliance and data security: whether data is used for training and whether private deployment is supported are often hard requirements for enterprise users.
5. Migration cost: it is advisable to add a unified model invocation gateway at the architecture layer to abstract away interface differences, turning “switching providers to compare costs” into a configuration change rather than an engineering overhaul.

7. Conclusion

Every time a flagship model gets a new generation, the price sheet is refreshed—but the methodology stays the same: define the workload profile first, then convert to a blended unit price, account for costs using real logs, and finally optimize continuously through caching, routing, and output control. Treating API cost as an engineering problem to measure, analyze, and review is far more reliable than gathering around the price board every time a new model is released.