LLM API Selection: Look Beyond Unit Price to Calculate True Task Costs

1. Background: From “Capability Race” to “Efficiency Race”

Over the past year, the iteration pace of large language models (LLMs) has accelerated significantly. New-generation models often show substantial leaps in comprehensive benchmark scores, covering code generation, complex reasoning, and long-horizon Agent tasks. At the same time, API pricing has not risen with these capabilities; many model services directly retain the pricing levels of their previous generations.

On the surface, this is a boon for developers: stronger capabilities at the same price. However, if you shift your perspective from “unit price per token” to “true cost per task,” the conclusion is not so simple. A noteworthy phenomenon has emerged in industry benchmark data: some upgraded models exhibit double-digit growth in output tokens when completing the same task. Even if the unit price remains identical, the final bill may increase rather than decrease.

This illustrates that in LLM selection and cost management, “looking at the price list” is only the first step; “calculating the actual ledger” is the key. This article will break down the components of LLM API costs and provide an actionable evaluation method and optimization checklist.

2. Cost Formula: Three Variables Determine Your Bill

The cost of a single model invocation is fundamentally determined by the following formula:

Task Cost = Uncached Input Tokens × Input Unit Price + Cached Input Tokens × Cache Unit Price + Output Tokens × Output Unit Price

The three variables are:

1. Token Unit Price: The number on the price list. It is the most visible and easily mistaken for the whole picture.
2. Token Consumption: The actual number of input and output tokens used to complete a given task. This is the most hidden variable and has the highest fluctuation.
3. Cache Hit Rate: The percentage of input that hits the prefix cache, directly determining the actual discount on the input portion.

Most teams focus solely on the first variable during selection, while the second variable is often the primary cause of budget overruns.

Why Do New-Generation Models Consume More Output Tokens?

This is not an accident but a byproduct of the capability improvement path:

• Explicit Reasoning Chains: New-generation models generally possess stronger “thinking” capabilities. Before providing a final answer, they generate longer reasoning processes, which are also billed as output tokens.
• Increased Tool Call Rounds: Models are more inclined to decompose tasks into multiple steps, repeatedly calling tools and verifying results. The intermediate outputs of each round accumulate costs.
• More Detailed Responses: More capable models tend to provide more complete explanations and cover more edge cases, naturally increasing the output length per response.

In other words, one of the ways a model “gets smarter” is by being willing to “think more, do more, and say more”—and all three directly impact token consumption.

3. Illustrative Calculation: Why the Bill Rises When the Unit Price Remains Unchanged

Let’s use a simplified example to illustrate (the numbers below are fictional for illustration and do not represent any real product’s pricing):

Assume a model service is priced at: $1.00/Million tokens for input, $4.00/Million tokens for output, and $0.25/Million tokens for cached input (equivalent to a discount of over 80% on the cached portion).

Scenario: A typical task with 20,000 input tokens (of which 15,000 are a stable prefix that can be cached) and an output of 2,000 tokens from the older model.

• Old version, no cache: 20,000 × 1/1M + 2,000 × 4/1M = 0.020 + 0.008 = $0.028
• Old version, with cache: 5,000 × 1/1M + 15,000 × 0.25/1M + 2,000 × 4/1M = 0.005 + 0.00375 + 0.008 = $0.01675

Now the model is upgraded. The unit price remains the same, but the output tokens per task increase by about 20% (corresponding to the magnitude observed in industry benchmarks), reaching 2,400 tokens:

• New version, no cache: 0.020 + 2,400 × 4/1M = $0.0296, which is about 5.7% more expensive than the old version.
• New version, with cache: 0.005 + 0.00375 + 0.0096 = $0.01835, which is still about 9.6% more expensive than the old version.

The conclusion is clear: Unchanged unit price ≠ Unchanged spend. When the growth of output tokens exceeds a certain threshold, the actual cost will rise. An increase in the cache hit rate can offset this growth to some extent. Evaluating a model upgrade requires calculating costs using real traffic replay, not just comparing price lists.

4. Capability Evaluation: Translating “Benchmark Scores” into “Task Performance”

Comprehensive Indices Are Useful, But Require Proper Reading

Multiple third-party evaluation agencies regularly publish comprehensive LLM intelligence indices, aggregating various benchmark test results into a single weighted score. The value of such indices lies in providing a horizontally comparable coordinate system: how much a new model has improved over the previous generation, and where it stands among models in the same tier.

However, they have limitations:

• A composite score masks individual differences. Two models with similar total scores might have distinct strengths in coding, long-context processing, or Agent task stability.
• There is a deviation between benchmark datasets and real-world business distributions. A high benchmark score does not guarantee good performance in your specific business scenario.

A More Practical Approach: The Capability-Cost 2D Perspective

For developers, a more effective evaluation framework places “capability” and “cost” within the same two-dimensional plane:

• Compare Costs at the Same Capability Tier: When several models have benchmark scores in the same tier, the cost per task (the average amount spent to complete a standard task) becomes the decisive factor. Industry benchmark data shows that models with similar capability scores can have a cost-per-task difference of up to 20%—a substantial gap at scale.
• Compare Capabilities at the Same Cost Tier: When the budget is fixed, choose the most capable model at that price point.
• Focus on the Efficiency Frontier: Plot candidate models on a scatter plot using “Capability Score” vs. “Cost per Task”. Models located on the efficiency frontier (strongest capability at the same cost, or lowest cost at the same capability) are the rational choices.

Furthermore, capability evaluation should align with your business: if the core scenario is code generation, focus on programming benchmarks and pass rates on real coding tasks; if the core scenario involves long-horizon Agent tasks, emphasize multi-step task completion rates and intermediate step stability; if the scenario is security-sensitive, defensive security capabilities should also be factored in.

5. Caching: The Most Underestimated Cost Lever

The principle of prefix caching is not complex: if the input prefix of the current request exactly matches a recent one, the service provider can directly reuse the previous computation results. This portion of the input is billed at a significantly lower price—cache prices for many model services can be as low as 20% of the original price, equivalent to a discount of over 80%.

High-value caching scenarios include:

• Long System Prompts: System prompts containing role definitions, constraint rules, and few-shot examples are often thousands of tokens long and highly stable, making them the most typical caching targets.
• Multi-turn Conversations: Conversation history naturally forms a progressively increasing prefix structure. The more turns there are, the greater the caching benefit.
• Fixed RAG Context: When retrieved document snippets are reused within the same session, they can be designed as a stable prefix.
• Agent Loops: Tool descriptions and environment state descriptions repeatedly carried in Agent tasks.

The key engineering principle is: Place stable content at the front, and dynamic content at the back. A cache hit is only possible if the request prefix matches at the byte level; a single character change (including mixing timestamps or random IDs into the prefix) will cause the cache to miss. Therefore, prompt templates should be deliberately partitioned: stable content like system instructions, tool definitions, and reference documents should be placed at the very beginning, while mutable content like user input and retrieval results should be placed at the end.

6. Actionable Selection and Evaluation Process

Step 1: Build a Representative Task Set. Extract 50 to 200 tasks from real business operations, covering a typical difficulty distribution (simple Q&A, medium complexity, complex reasoning/Agent tasks). Avoid using only public benchmark sets or artificially created samples.

Step 2: Replay and Record Usage. Run the same task set against each candidate model. Record the usage data for each invocation (input tokens, output tokens, cache hit tokens). Most model services include these fields in their API responses.

Step 3: Calculate Task Cost and Quality Score. Calculate the cost of each task using the formula in Section 2. Simultaneously, provide a quality score using human review or automated evaluation.

Step 4: Calculate “Cost Per Unit of Quality” and Visualize. The core metric is not “who is cheaper” or “who is stronger,” but “how much money is required to achieve the same level of quality.”

An illustrative cost calculation script:

def call_cost(prompt_tokens, completion_tokens, cached_tokens,
              price_in, price_out, price_cache):
    """Calculate single call cost in USD based on price per million tokens"""
    uncached = max(prompt_tokens - cached_tokens, 0)
    return (uncached * price_in
            + cached_tokens * price_cache
            + completion_tokens * price_out) / 1_000_000
# Example: Compare the actual cost of the same task on old vs. new model versions
old = call_cost(20000, 2000, 15000, 1.0, 4.0, 0.25)
new = call_cost(20000, 2400, 15000, 1.0, 4.0, 0.25)
print(f"Old: ${old:.5f}  New: ${new:.5f}  Increase: {(new/old-1):.1%}")

In real-world projects, this logic should be integrated into a unified API gateway to automatically aggregate daily cost reports by task dimension. Mainstream cloud platforms (such as Baidu AI Cloud) generally provide API usage statistics and monitoring alert capabilities. You can directly leverage these general capabilities to build a cost dashboard rather than reconciling accounts manually after the fact.

7. Cost Optimization Checklist

Given a fixed model and pricing, there is still significant room for optimization on the engineering side:

1. Control Output Length: Explicitly require concise answers in your instructions; use structured formats like JSON to constrain results; reasonably set the max output token limit to prevent runaway long responses.
2. Prompt Slimming: Regularly clean up redundant content in system prompts, and keep the number of few-shot examples at the point of diminishing returns.
3. Cache-Friendly Design: Rearrange prompts following the “stable in front, dynamic in back” principle. Avoid concatenating volatile content like timestamps or random numbers at the beginning of the prompt.
4. Model Routing: Not all requests require the most powerful model. Establish a routing layer that sends “simple tasks to lightweight models and complex tasks to flagship models.” This can significantly reduce overall costs with almost no loss in quality.
5. Batch Processing for Offline Tasks: Use batch APIs for time-insensitive tasks, which usually provide additional discounts.
6. Budgets and Alerts: Set token usage and monetary budgets for different business lines, coupled with monitoring alerts to prevent billing disasters caused by abnormal traffic (such as infinite Agent retry loops).

8. Conclusion: The Next Phase of Competition is “Unit Cost at Equal Intelligence”

Returning to the initial question: the model is upgraded, the unit price remains the same, so why might the bill be more expensive? Because the true determinant of cost is the product of “unit price × token consumption × cache efficiency,” and capability improvements are often accompanied by an increase in token consumption.

This also explains the shift in the industry’s competitive focus: as the capability scores of top-tier models gradually converge, the marginal value of simply competing on benchmark scores is declining. “Who can push the cost per task lower at the same level of intelligence” is becoming the new watershed. For users, this means the selection methodology must be upgraded: from looking at price lists, to looking at benchmark scores, to using real task replays to calculate the “cost per unit of quality.” The more meticulously you calculate this ledger, the more从容 you will be during the rapid iteration cycles of LLMs.