LLM API Selection: Calculating the Real Cost per Task Beyond Token Prices

1. The Cost Reality Behind “More Capability, Same Price”

In recent years, large language models have been iterating at a markedly faster pace, and “improved capability with unchanged API pricing” has become a common release strategy. For developers, this lowers the barrier to adoption — you can plug a new version into your existing call chain without requesting new budget. But an API bill is determined by two variables together: unit price and consumption. After a model upgrade, longer reasoning chains, exposed thinking processes, and more rounds of tool calls all push up the number of output tokens per task. A pattern widely observed across the industry: a new-generation model may produce double-digit percentage more output tokens than its predecessor. If the unit price stays flat while consumption rises, actual spending goes up, not down.

So evaluating a model upgrade takes more than reading the sticker price of “X dollars per million tokens.” You have to return to the fundamental question: how much does it actually cost to complete my task? That is the core question this article explores.

2. Understanding the API Billing Model: Three Price Tiers and Cache Discounts

Mainstream LLM APIs today generally use a three-tier billing structure:

1. Input price — charged on all prompt tokens sent to the model, including the system prompt, conversation history, retrieved documents, and every other part of the context.
2. Output price — charged on model-generated content, typically much higher than the input price, commonly 2–4x in the industry. The reason lies in the compute characteristics of autoregressive generation: every token produced requires full reads and writes of the KV cache, memory bandwidth becomes the bottleneck, and inference cost is inherently higher.
3. Cached input price — a steep discount on the input portion that hits the prefix cache. On some platforms the discount can exceed 80%, meaning you pay only about 20% of the standard input price.

The principle behind prefix caching is straightforward: when multiple requests share the same prefix (system prompt, tool definitions, few-shot examples, long documents), the server can reuse the KV cache already computed for that portion and skip redundant prefill work, which is why it bills at a discounted rate. The higher your hit rate, the lower your effective input cost.

This gives us a general per-request cost formula:

cost = (Tin − Tin_hit) × Pin + Tin_hit × Pcache + Tout × Pout

where each T is a token count and each P is the corresponding price per million tokens. This formula is the foundation for all the cost analysis below.

3. Capability Evaluation: How to Read and Use Composite Indices

Third-party evaluation organizations typically maintain cross-vendor benchmark suites and publish a composite intelligence index that makes it easy to compare capability trends across models. Used correctly, such an index serves two purposes:

• Longitudinal comparison across versions: see at a glance how a model family evolves between generations. If a single upgrade lifts the composite score by several points and moves the model into the same score range as its same-tier competitors, the generational leap is substantial.
• Horizontal tiering: models with similar scores can be treated as the same capability tier. Once capability is roughly equal, price, token consumption, and ecosystem support become the primary decision variables.

Know the limits, though. Benchmark suites cover general-purpose tasks that may not match your workload distribution, and leaderboard scores can diverge from real-world engineering performance (stability, format compliance, hallucination rate). A composite index is only good for initial screening; the final call must come from re-testing on your own task set.

Recent industry trends show model upgrades concentrating in three areas: code generation and repair, long-horizon agent tasks (multi-step planning, tool calling, self-correction), and security and adversarial robustness. If your business falls mainly into these areas, the payoff from a generational upgrade will be especially pronounced.

4. The Core Metric: Cost per Task

Definition: cost per task is the total API fee incurred to complete one task end to end, under a fixed task definition, fixed prompts, and a unified grading standard.

It combines “unit price” and “consumption” into a single directly comparable number — a metric much closer to the real bill than unit price alone. Four variables shape it: input token count (driven by task complexity and your context strategy), output token count (driven by model verbosity, thinking length, and tool-call rounds — the variable most prone to runaway growth), cache hit rate (driven by how stable your prompt structure is), and the three unit prices themselves.

Here is an illustrative calculation (the numbers are purely hypothetical and do not represent any vendor’s real pricing):

def task_cost(tokens_in, tokens_out, hit_ratio,
              p_in, p_out, p_cache):
    """Return the cost per task in USD; prices are per million tokens."""
    hit = tokens_in * hit_ratio
    miss = tokens_in - hit
    return (miss * p_in + hit * p_cache) / 1e6 \
         + tokens_out * p_out / 1e6
# Illustrative prices: input 1.5 / output 6.0 / cache 0.3 (USD per million tokens)
P_IN, P_OUT, P_CACHE = 1.5, 6.0, 0.3
# Previous-generation model: 2,000 output tokens
old = task_cost(8_000, 2_000, 0.6, P_IN, P_OUT, P_CACHE)
# New-generation model: same prices, but 20% more output (2,400 tokens)
new = task_cost(8_000, 2_400, 0.6, P_IN, P_OUT, P_CACHE)
print(f"Previous gen: ${old:.4f} / New gen: ${new:.4f} "
      f"/ Change: {(new/old-1)*100:+.1f}%")

Run it and you will confirm: even with unit prices completely unchanged, a 20% increase in output tokens pushes cost per task up by roughly 15%–20%, depending on how much of the total cost output accounts for. That is the math behind “same unit price, bigger bill.” The flip side also holds: if the new model is capable enough to cut retries and rework, total cost can come down — so you must measure it, not guess.

5. Cost Estimation and Optimization Practices

1. Build a task baseline set

Pull 20–50 representative tasks from real production workloads, covering typical scenarios such as short Q&A, long-document understanding, code tasks, and multi-step agent workflows. Fix the prompts, temperature parameters, and grading criteria so the benchmark is repeatable. Re-run it on every model upgrade or vendor switch to keep the comparison apples-to-apples.

2. Log four key numbers

For every request, record input tokens, output tokens, cache hit rate, and whether the task succeeded. Aggregate them into two core metrics — cost per task and task success rate — and plot a cost-versus-quality scatter chart. The value equation becomes obvious at a glance.

3. Use caching as a structural cost lever

Put everything that never changes at the very front of your prompt: system prompt, tool schemas, few-shot examples, retrieved long documents. The more stable the content and the further forward it sits, the higher your prefix cache hit rate. In multi-turn agent applications, keep the conversation prefix from being rewritten wherever possible. Also note that caches usually have a time-to-live (TTL); once entries expire, the discount disappears, so weigh the benefit against your request cadence.

4. Rein in output length

• Set a sensible max_tokens ceiling as a backstop against abnormally long outputs.
• Require structured output (e.g., JSON) so the model doesn’t pad answers with long explanatory prose.
• State explicitly in the prompt: “Give the result directly; do not restate the question or walk through the process.”
• For models with deep-thinking capabilities, assess whether every task really needs extended thinking — turn it off where you can and keep it short where you can’t.

5. Route across model tiers

Not every request needs the flagship model. Use intent classification or rule-based routing: send format conversion and simple extraction to lightweight models; reserve complex reasoning, code generation, and multi-step planning for the flagship. In practice, lightweight models often absorb more than half of traffic, which noticeably lowers overall cost.

6. Monitor, budget, and attribute

Aggregate token consumption and spend by project and by feature, and set threshold alerts. Mainstream LLM service platforms (for example, Baidu AI Cloud’s model services) generally provide common supporting capabilities such as usage statistics, monitoring alerts, and batch inference. When integrating, examine the completeness of these capabilities first, so that every dollar spent is attributable and traceable.

6. A Selection Decision Checklist

• Capability clears the bar: shortlist with a composite index, then validate on your own task set, paying special attention to code and agent performance.
• Judge cost per task, not unit price: measure with your baseline set, and stay alert to hidden price hikes of the “longer output” variety.
• Cache fit: workloads with stable prefixes and long contexts benefit most from cache discounts.
• Output controllability: does the model support structured output and thinking-length control?
• Engineering support: context window, concurrency limits, SDK maturity, observability.
• Deployment options: open weights or private deployment options affect long-term cost and data compliance.
• Avoid lock-in: abstract the model access layer and maintain the ability to switch vendors.

7. Conclusion

As composite capability scores among leading models converge, competition is shifting from “who is smarter” to “who delivers intelligence more cheaply.” For developers, that means the evaluation process must move in step: from watching unit prices to computing per-task costs, from reading leaderboards to running your own baselines, from passively accepting the bill to actively governing token consumption. Translating the “more for the same price” narrative into your own business’s cost-per-task curve is where the real moat of model selection lies.