LLM API Cost Accounting and Optimization Practices
This article explains why LLM API cost control should be treated as an engineering task, breaks down token-based pricing for input, output, and cache-read tokens, shows how to estimate single-call and monthly costs, and presents practical optimization strategies such as context trimming, cache reuse, model routing, batch processing, output control, and budget monitoring.
1. Why cost accounting should be an engineering capability
Many teams ignore LLM API costs during prototyping. The issue often becomes visible only after launch, when traffic grows and bills increase rapidly. Unlike traditional compute billing based on requests or runtime, LLM API billing is primarily based on token throughput—the basic unit the model processes for text. This means total cost depends not only on how many times the API is called, but also on how much content is sent into and generated from the model on each call.
For this reason, cost estimation should not be left until financial reconciliation. It should be treated as a first-class consideration during system design. The following sections first explain the billing model, then cover estimation and optimization.
2. Billing model: three token categories and three pricing dimensions
Most mainstream LLM service bills are composed of three parts. Understanding the differences among them is essential for controlling cost.
Input cost (input / prompt tokens): This includes user prompts, system instructions, retrieved context, conversation history, and any other text sent to the model. It is the main cost source in long conversations and retrieval-augmented generation (RAG) scenarios because it grows with context length.
Output cost (output / completion tokens): This covers the text generated by the model. Generation is decoded token by token, which consumes more compute than one-time input encoding. As a result, output tokens usually have a higher unit price than input tokens, often several times higher.
Cache-read cost (cache read tokens): When multiple requests share the same prefix, such as a fixed system prompt, a long document, or few-shot examples, the provider can reuse intermediate computed state, such as the KV cache, and charge a lower price for the cached input tokens. The more stable and reusable the shared prefix is, the more savings can be achieved.
A key rule is: optimizing input context often provides greater benefit than optimizing output length, because input tokens are usually larger in volume and can be further reduced through caching and trimming.
3. Worked example: from a single call to a monthly budget
To make the calculation reproducible, this section uses hypothetical prices. The values are examples only and do not represent any specific vendor’s pricing.
Assume a service has the following unit prices:
- Input: A billing units per million tokens
- Output: B billing units per million tokens
- Cache read: C billing units per million tokens
Consider a scenario where a mid-sized codebase is analyzed. A single request uses about 500,000 input tokens (0.5 million), and the expected output is 20,000 tokens (0.02 million) of modification suggestions.
The single-call cost formula is:
Single-call cost = (Input tokens in millions × Input price per million tokens)+ (Output tokens in millions × Output price per million tokens)= (0.5 × A) + (0.02 × B)
One point that is easy to overlook is that although output is only 1/25 of the input volume, its cost share can become closer to that of input because the output unit price is usually higher. In the example, if the input price is 0.8 and the output price is 2.8, the single-call cost is about 0.4 + 0.056 = 0.456 billing units. If the input price is 0.4 and the output price is 1.4, the cost is about 0.2 + 0.028 = 0.228 billing units. In both cases, lowering the unit price reduces cost nearly proportionally.
Monthly estimation can be calculated by multiplying by call frequency:
Monthly cost ≈ Single-call cost × Monthly call volume × (1 + overhead factor)
The overhead factor covers extra consumption from retries, failed replays, prompt iteration, and similar cases. In engineering practice, reserving 15% to 30% is usually reasonable.
4. Promotional pricing windows: why recalculation matters
A common pricing strategy in the industry is to offer a discounted rate for a limited period when a new model or new capability is launched, such as charging a percentage of the regular price for several weeks. The value of such a window is not simply taking advantage of a lower price. It provides a low-risk period for migration and load testing.
A rational approach is to run a real cost assessment and quality evaluation with production-like traffic during the promotional period. Measure the full picture of “unit price × actual context distribution,” then decide whether long-term use is economical. Once the promotional period ends, prices return to normal levels. If teams only discover then that overly long contexts or cache misses will double the bill, the migration decision may need to be reversed. In other words, a promotional period is the best time to make decisions with real data, not a time to blindly increase usage.
5. Systematic cost-optimization strategies
The unit price is determined by the provider, but the total amount and structure of tokens can be controlled. The following strategies are ordered by return on investment.
1. Context trimming and prompt simplification
Input tokens are usually the largest variable. Regularly audit retrieved fragments, conversation history, and system prompts inserted into the context. Remove redundant content, merge duplicates, and limit the number of retrieved items. For long documents, summarize first or use segmented retrieval, and feed only the parts strongly related to the current task.
2. Make full use of cache reuse
Place stable content, such as system role definitions, tool descriptions, few-shot examples, and long document prefixes, at the beginning of the prompt. This allows multiple requests to share the same prefix and benefit from lower-priced cache reads. Do not insert changing content, such as timestamps or random IDs, at the beginning of the prompt, because this can frequently invalidate the cache.
3. Model routing
Not all tasks require the most powerful model. Use lightweight, lower-cost models for simple tasks such as classification, extraction, and formatting, and reserve stronger models for complex reasoning or long-chain tasks. A routing layer can significantly reduce costs with little or no loss in effectiveness.
4. Batching and asynchronous processing
For offline or non-real-time tasks, such as batch summarization or nighttime data processing, prefer asynchronous or batch interfaces. Many providers offer lower rates or higher throughput quotas for these types of calls.
5. Output-length control
Clearly constrain generation length. Use structured output, such as limited fields, instead of long free-form text. This saves output tokens and makes it easier for downstream programs to consume the results.
6. Budget monitoring and alerts
Connect token usage to monitoring and alerting systems. Set daily or monthly budget thresholds by project or API endpoint, and trigger alerts or rate limiting when thresholds are exceeded. Logging services can record input, output, and cache-read token details for each call, making costs attributable and drillable.
6. Conclusion
The cost of LLM API calls is determined mainly by two factors: unit price and the total amount and structure of tokens. Unit price is an external variable, while context design, cache hit rate, model selection, task batching, and budget governance are internal levers that developers can control. By moving cost estimation earlier into the design phase, using real data to drive model selection and migration decisions, and applying systematic optimization methods, teams can find a sustainable balance between quality and cost. For teams planning to deploy LLM capabilities in production, the same methodology applies to mainstream LLM service providers. The core principle is the same: measure accurately first, then optimize.
