# How to Choose an LLM API: Per-Task Cost and Optimization
As LLM APIs become more capable and more comparable, choosing a model should not depend only on benchmark rankings or headline per-token prices. This article explains how to compare the real cost of completing a business task by breaking down input, output, and cached token billing, avoiding misleading cost assumptions, improving cache hit rates, and building a reusable per-task cost model.
How to Choose an LLM API: Per-Task Cost and Optimization
1. The focus has shifted: choose an LLM API based on per-task cost, not only rankings
When evaluating an LLM API, teams often start with capability leaderboards: reasoning scores, coding performance, multi-turn stability, long-context understanding, and so on. However, once mainstream models reach a similar capability level, engineering teams face a more practical question: if two models can both complete the task, why is one bill higher?
This is the core shift in choosing an LLM API. The decision moves from comparing absolute capability to comparing the cost per task after capability has reached a usable level. Capability determines whether a model can do the job. Cost determines whether it is suitable for large-scale deployment. Once a model has crossed the business capability threshold, paying a large premium for a small score advantage often brings little additional value.
Therefore, the evaluation focus should move from “price per million tokens” to “how much does it cost to complete one business task?” This article does not discuss specific vendors. Instead, it provides a general approach: how to read billing components, how to avoid traps where a lower per-token price leads to a higher total cost, how to use caching, and how to build a reusable cost accounting model.
2. Understand the billing structure first: input, output, and cache are separate costs
Most LLM APIs use tokens as the minimum billing unit, but looking at only one price is not enough. A complete bill should be broken down into at least three categories:
- Input tokens: prompts, system instructions, tool definitions, historical context, and other content sent to the model.
- Output tokens: model-generated answers, reasoning traces, code, structured results, and other generated content.
- Cached tokens: when a request prefix matches previously cached content, the matching portion can be billed at a lower rate.
One important detail is that total input tokens usually consist of two parts: uncached input tokens and cache-hit tokens. In other words:
Total input tokens = Uncached input tokens + Cache-hit tokens
During billing, the cache-hit portion is usually not charged at the normal input price. Instead, it is charged at a cache price. If a provider displays cache hits on the input side, it is important to confirm whether the discount has already been applied, so that the cost is not calculated twice.
Another key point is that output pricing is usually higher than input pricing. For example, input may cost just over USD 1 per million tokens, while output may cost more than USD 4 per million tokens. A cache hit may be priced at only 15% to 20% of the normal input price, which corresponds to roughly 80% to 85% savings. For this reason, the structure of token usage matters more than a single price label.
3. Key misconception: a cheaper per-token price does not mean a cheaper per-task cost
Many teams are influenced by statements such as “the price did not increase” or “input is cheaper.” However, the real cost depends on how many tokens are needed to complete a task and whether retries are required.
A more accurate total cost formula is:
Total cost per task ≈ (Uncached input tokens × Input price + Cache-hit tokens × Cache price + Output tokens × Output price) × Calls or retries for the task
Several points need special attention:
- “Uncached input tokens” does not include cache-hit tokens.
- The input side should be split into uncached input and cache-hit tokens to avoid double counting.
- The multiplication outside the parentheses represents the total number of calls or retries needed to complete the same business task.
- If a business task contains multiple steps, each step should be calculated separately and then summed.
For example, suppose two models have similar per-call pricing, but Model A produces longer outputs and requires more retries to improve the success rate of complex tasks. Even if the per-token price stays the same, the total token consumption and number of calls increase, so the cost per task rises.
This is why a cheaper per-token price does not always mean a lower cost per task. When evaluating cost efficiency, teams should focus on the business-level metric: the average cost to complete one task, rather than only the price per million tokens.
4. Interpret capability scores correctly: scores change, but business testing is the final check
Public benchmark scores can still be useful as an initial screening tool, but they should not replace testing with real business data. There are three reasons:
- Scores reflect capability, not cost. Two models may have similar scores, but one may produce shorter and cheaper outputs, while the other may be more verbose.
- Scores can change with model versions, evaluation datasets, prompts, and testing methods. A high score in one evaluation does not guarantee stable performance on your own business samples.
- A high score may come from a longer reasoning process. If your workload is sensitive to latency or cost, that type of model may not be the best fit.
A more reliable approach is to use public scores to shortlist models that meet the capability threshold, then run small-scale tests using your own business samples. The focus of these tests should not be only “can it answer?” It should be “what is the average input, output, cache, and retry cost required to complete one task?”
5. Cache optimization: improving cache hit rate is often more effective than lowering the per-token price
Context caching is one of the most underestimated cost levers. The principle is simple: if multiple requests share the same prefix, such as a fixed system prompt, tool definitions, example samples, or long documents, the cached portion can be billed at a lower price.
Before optimizing caching, teams should understand several common limitations:
- Minimum cache length: Some providers require the prefix to reach a minimum token count before it can be cached.
- Cache lifetime: Cache entries may be retained for minutes to hours. After expiration, they need to be written again.
- Prefix invalidation conditions: Timestamps, random IDs, user IDs, tool schemas, system prompts, message order, delimiters, or whitespace changes can all cause cache misses.
- Provider differences: Automatic caching, explicit cache markers, charges for cache creation, and whether cache can be shared across requests all need to be verified against the target provider’s documentation.
The code below only demonstrates the algorithm. When calculating real costs, replace the example prices with the target provider’s actual prices:
# Demo: input-side cost comparison at different cache hit ratesinput_price = 1.40 # Input price per million tokens (illustrative)cache_price = 0.26 # Cache-hit price per million tokens (illustrative)def input_cost(tokens, hit_rate):cached = tokens * hit_ratefresh = tokens * (1 - hit_rate)return cached * cache_price + fresh * input_pricefor rate in [0.0, 0.5, 0.8]:tokens = 1_000_000cost = input_cost(tokens, rate)miss_tokens = tokens * (1 - rate)baseline = tokens * input_pricesaved = (baseline - cost) / baselineprint(f"Hit rate {rate:.0%}: "f"uncached tokens {miss_tokens:.0f}, "f"cost {cost:.3f} USD, "f"saved {saved:.1%}")
As the results show, increasing the cache hit rate from 0% to 80% can significantly reduce input-side costs. To improve the cache hit rate, teams can take the following engineering steps:
- Put stable content at the beginning of the prompt, such as system instructions, tool descriptions, and fixed examples.
- Put dynamic content later, such as user questions, timestamps, and random IDs.
- Keep the prefix byte-for-byte identical. Avoid changes caused by spaces, newlines, JSON field ordering, or other small differences.
- Prioritize caching for long documents, multi-turn conversations, and batch tasks.
- Monitor cache hit rate instead of only checking whether requests succeed.
6. Build your own per-task cost accounting framework
Instead of passively accepting a vendor’s price list, teams should incorporate cost into their own evaluation process. A practical framework can be built in four steps.
1. Build a representative task set and group tasks into buckets
Do not evaluate only a small number of simple samples. It is better to create buckets based on business dimensions:
- Input length: short prompts, medium context, and long documents.
- Output length: short answers, structured results, and long-form generation.
- Task types: question answering, classification, extraction, code generation, and multi-turn conversation.
- Retry difficulty: tasks that succeed on the first try, tasks that often require retries, and tasks that easily trigger length limits.
2. Collect complete telemetry fields
Each request should record at least the following fields:
request_id, task_id, model, prompt_version, input_tokens_total,cache_hit_tokens, cache_miss_tokens, output_tokens, retry_count,latency_ms, status_code, error_type, cost_estimate
The key is to break costs down into an explainable level of detail: how many tokens are input, how many are cache hits, how many are output, how many retries occurred, and what caused failures, such as timeouts, format errors, or unacceptable content.
3. Define the cost calculation method
Use the same formula to keep comparisons consistent:
Total cost per task = Sum over steps[(Uncached input tokens × Input price + Cache-hit tokens × Cache price + Output tokens × Output price) × Calls or retries]
Important notes:
- Total input tokens should not be double counted.
- Cache-hit tokens should be priced separately.
- Retry costs should be included in the total cost of the task.
- Multi-step tasks should be calculated step by step and then summed.
4. Add weights and thresholds
When making a decision, teams can quantify and compare capability, cost, latency, and retry rate:
Candidate model Capability threshold Cost per task Latency threshold Retry threshold Weight suggestion ConclusionModel A Pass Low Medium Low 0.4 Preferred for cost-sensitive workloadsModel B Pass Medium Low Low 0.4 Preferred for low latencyModel C Fail Low — — — Eliminate
This changes model selection from a subjective judgment such as “it feels smarter” to a process that is calculable, reproducible, and comparable.
7. Practical recommendations for production: control retries, caching, and anti-patterns
Finally, here are several actionable recommendations for engineering teams:
- Use capability scores for initial screening and real task testing for the final decision. Do not let leaderboards make the choice for you.
- Treat cost per task as a core KPI instead of focusing only on the price per million tokens.
- Set a retry budget for failed requests. Failed requests may still incur input or partial output charges, and retries can amplify costs. Teams should set maximum retry counts, timeouts, idempotency mechanisms, and alert thresholds.
- Control output length. Set reasonable maximum generation lengths, constrain overly verbose reasoning, and avoid letting output tokens become a major cost driver.
- Check caching strategies by default. For repeated prefixes, enable caching where appropriate and monitor prefix stability and cache hit rate.
- Avoid common anti-patterns: switching to a cheaper model that lacks sufficient capability and therefore increases retries; placing dynamic fields in the prefix and causing cache misses; and failing to monitor retry rates, which can lead to uncontrolled spending.
As differences in model capability continue to narrow, engineering competitiveness will increasingly come down to who can achieve the same result at a lower cost. By making token efficiency, cache hit rate, retry budgets, and cost dashboards part of the evaluation process, teams can better control costs when deploying at scale.
