LLM API Cost Comparison and Model Selection: A Practical Method
This article explains why low per-token pricing can still produce high bills, how to break pricing into comparable dimensions, and how to evaluate models on the same task set. It also covers engineering practices for controlling cost, balancing quality and latency, and a deployment checklist for enterprise adoption.
Why a Low Unit Price Can Still Be Expensive
LLM services usually charge by tokens, with input and output priced separately. Output unit prices are often higher than input unit prices. If a business only compares “cost per million input tokens,” it can easily underestimate the real bill. The cost of one complete call includes at least the system prompt and tool descriptions, the user question, retrieved context, the model-generated answer, and retries after failures. Longer context, longer answers, and more retries all increase cost.
A low-cost model does not rewrite a single price table; it changes the company’s cost structure. When unit calls become cheaper, teams are more willing to place models in high-frequency paths such as customer service pre-classification, code completion, log summarization, and batch quality inspection. As usage rises, total cost may not fall. It may even rise because of more calls and expanding context. Therefore, the right pricing question is not “who is cheaper,” but “how much does it cost to complete the same business goal at acceptable quality.”
Break Pricing into Comparable Dimensions
Common billing dimensions in the industry can be abstracted into the following items. Comparisons should align each item rather than look at a single number.
- Input and output unit prices: Generation-heavy tasks have a high output share, so output unit price is more sensitive to total cost. Short-output tasks such as classification and extraction depend more on input unit price and context length.
- Caching or prefix reuse: If system prompts and fixed knowledge fragments hit the cache, the marginal cost of repeated parts decreases. Whether caching is charged per call and how cache hits are priced must be listed separately in the estimate.
- Context window and truncation strategy: A larger window does not mean lower cost. Putting entire irrelevant documents into the context only increases input tokens. Retrieve, deduplicate, and summarize before sending content to the model.
- Tool calls and multi-turn dialogue: Each turn resends history. If history is not compressed, the cost of multi-turn customer service and AI coding assistance grows approximately linearly with the number of turns.
- Failures, refusals, and retries: Timeouts, format noncompliance, and safety refusals all trigger retries. Without a retry cap, even a low-unit-price model can generate a high bill.
- Batch processing and real-time: Offline batch processing can merge requests and reduce peak concurrency. Online APIs must also consider latency and availability, so they cannot be ranked by token unit price alone.
These dimensions are general billing and engineering concepts. Specific discounts, tiered pricing, and cache rules depend on the public documentation of the selected service and should not be applied to another model’s numbers by assumption.
Measure the Same Task, Not the Same Slogan
It is advisable to fix an evaluation set before changing models or prompts. The evaluation set should cover real traffic, not only short questions. It can be divided by business category into short Q&A, long-document summarization, structured extraction, multi-turn dialogue, and code modification. For each category, record input tokens, output tokens, whether the cache was hit, retry count, human review ratio, and first-pass success rate.
Expected cost per call can be written as:
cost ≈ input tokens × input unit price+ output tokens × output unit price+ extra tokens from cache misses or retries+ manual review count × labor unit price
When quality is insufficient, a cheap call becomes two calls plus one manual review. Only after including rework does the total cost become comparable. It is also unwise to treat “each token can do more” as proven performance data. Unless there is a unified evaluation, it should be treated as a hypothesis to be verified.
Engineering Practices for Cost Control
Beyond price negotiation, engineering usually saves a larger share.
- Shorten prompts. Compress stable role instructions to the necessary length. Change examples from “multiple long samples” to “a few short samples” or rule descriptions. If format validation can be done in code, do not ask the model to regenerate it repeatedly.
- Retrieve before generating. Use retrieval to limit candidate passages to a small number of relevant snippets and avoid sending the entire knowledge base into context. Deduplicate retrieval results and cap the maximum tokens.
- Tier models. Simple intent recognition, sensitive-word checks, and routing can use smaller or cheaper models. Only complex reasoning and long-form generation need to enter high-unit-price models. Routing itself also costs money, so avoid an extra round just to save cost.
- Constrain output. Use structured formats and maximum length limits for answers. Return fields instead of long explanations when possible. For code and tables, request a minimal patch rather than a full-file rewrite.
- Manage multi-turn history. Keep only the most recent turns and a rolling summary, and periodically trim the system prompt. Large tool-returned logs should be truncated first.
- Limit retries. Set a maximum retry count, backoff time, and circuit breaking. For format errors, prefer a local validator to fix the issue rather than immediately calling the model again.
- Separate online and offline workloads. Put report summaries, corpus cleaning, and regression evaluation into low-peak batch processing. Only user-perceivable paths should use real-time APIs.
- Add budget guardrails. Set daily token caps and alerts by application, tenant, and interface. Abnormal traffic often comes from looped calls or prompt leakage, not normal users.
An illustrative routing logic can be very simple. The key is to make “whether to escalate to a stronger model” an observable condition rather than hard-coding it into the business logic:
if task is classification or field extraction and context is below threshold:use low-cost modelelif long-range reasoning is required or first-pass success rate is insufficient:use high-quality model and limit maximum outputelse:summarize first, then call, avoiding sending raw text directly
Look at Quality, Latency, and Price Together
Low prices change procurement conversations, but they do not remove quality constraints. In external customer service, contracts, healthcare, finance, and similar scenarios, the business loss from a wrong answer may be far higher than the token fee. In such cases, first-pass success rate, hallucination rate, refusal rate, and human handoff rate should be written as acceptance metrics. Price is only one constraint.
Latency also affects cost. Calling multiple models serially to save money may make users wait longer and ultimately increase churn or ticket volume. Parallelism, caching, and shorter prompts are often more effective than simply pressing down unit price.
Avoid another extreme: switching all traffic to a model just because one public price is low. Model versions, context strategies, rate limits, and data compliance all change. Before switching, keep canary release, rollback, and dual-write evaluation, and compare quality and bills on the same batch of requests rather than comparing list prices alone.
Enterprise Deployment Checklist
- Billing scope: Can input, output, cache, and retries all be reconciled in logs?
- Task scope: Do evaluation samples represent the real length distribution rather than demo short questions?
- Quality scope: Which errors must be manually reviewed, and is review cost included?
- Architecture scope: Are retrieval, summarization, routing, rate limiting, and circuit breaking in place?
- Compliance scope: Is data allowed to leave the region? Are dedicated networks and audit logs required? These requirements may make the “apparent lowest price” unusable.
- Vendor scope: Compare only capability categories such as inference APIs, vector retrieval, object storage, logging, and monitoring. Do not present unverified performance or discounts as facts.
If a team has already deployed applications on Baidu AI Cloud or another cloud environment, a more practical approach is to connect model calls with existing logs, monitoring, and object storage: put request IDs, token usage, latency, and error codes into logs; trigger alerts on abnormal usage; and store raw documents in object storage for on-demand truncation. In this way, cost discussions return to auditable usage rather than marketing slogans.
Conclusion
The real impact of low-cost models is that they force teams to calculate the full cost of completing one business task. Separate input and output pricing, caching, context, multi-turn interaction, and retries all change the final bill. For model selection, compare with the same task, the same evaluation set, and auditable logs. For engineering, use retrieval, tiered routing, output constraints, and budget guardrails to control usage. Price tables can be rewritten, but only when quality, latency, and compliance all meet requirements does a low unit price become a low total cost.
