Choosing an LLM: How to Evaluate Cost-Effectiveness and Total Cost
This guide explains why a model's per-token price is not a reliable basis for choosing an LLM. It covers how to estimate total cost per qualified task, which dimensions to evaluate, how to build a reproducible test set, how to compare models by effective unit cost, and which engineering practices and common pitfalls affect long-term spending.
1. Why a Low Unit Price Does Not Mean a Low Total Cost
Many people compare models by looking first at the price per million tokens. That number is only part of the cost. The real cost of a single business request breaks down into the following items:
- Direct billing cost: Input tokens and output tokens are billed at their own unit prices.
- Retry cost: When output is unsatisfactory, fails format parsing, or times out, the system must call the model again. This can double the cost or increase it even further.
- Manual review cost: When output quality is unstable, the staff time needed for proofreading often costs much more than the API fees.
- Latency loss: Slow responses can affect conversion rates, user retention, or the efficiency of internal workflows. This loss rarely appears on a bill, but it is real.
- Migration cost: Prompts, output parsing logic, and evaluation systems are all tied to a specific model. Switching models requires re-adapting them.
For this reason, a better way to measure cost is the “total cost per task,” not the “price per call.” The following approximate formula can serve as a starting point:
Total cost per task = (input tokens × input price + output tokens × output price) × average number of calls+ manual review cost + converted latency loss + amortized migration cost
Each parameter in the formula must come from actual testing. Do not rely only on the listed prices.
2. Building the Evaluation Dimensions
When selecting a model, evaluate it on at least six dimensions.
1. Capability fit. Determine whether the model can complete the target task reliably, such as information extraction, code generation, text classification, or multi-turn dialogue. Base capability metrics on your own business data, not on general leaderboard scores. Leaderboard scores indicate the approximate general level of a model, but they do not directly reflect how it will perform in your business.
2. Unit task cost. Convert cost into the cost of each qualified task completed. This is the most direct basis for comparison.
3. Latency and throughput. Measure time to first token, full response time, and throughput under concurrency separately. Interactive scenarios are sensitive to time to first token, while batch processing depends more on throughput. Tail latency, such as P95 and P99, usually reflects the real user experience better than the average.
4. Stability and availability. Consider service availability, rate-limiting policies, error rates, and how often versions are updated. A version change can shift output style or format, so this risk should be part of the evaluation.
5. Data and compliance requirements. When sensitive data is involved, confirm that data storage location, retention period, and access control meet your organization’s requirements. If this requirement is not met, the model cannot be used, no matter how well it scores on other metrics.
6. Migration and lock-in cost. A unified model-calling abstraction layer lowers the cost of switching models in the future. The investment in this layer usually pays off over a long period of operation.
3. Building a Reproducible Evaluation Set
The evaluation set is the foundation of model selection. If it does not represent real business use, every later comparison will be distorted. You can build one by following these steps:
- Sample from real traffic. Cover common requests, edge cases, and high-risk scenarios, and keep the proportions close to the online distribution.
- Stratify by difficulty. Track easy, medium, and hard samples separately. This shows where a model’s capability breaks down.
- Define clear scoring criteria. For structured tasks, use automated metrics such as field accuracy and format compliance rate. For open-ended tasks, write detailed scoring guidelines and have several reviewers score independently.
- Use blind evaluation. Reviewers should not know which model produced each output. This reduces subjective bias.
- Fix randomness settings. Apply the same sampling parameters, such as temperature, across all tests so the results are comparable.
The following sample code shows an evaluation pipeline. model_client is an abstract interface. In practice, replace it with your team’s client wrapper, and confirm the prices against the latest official documentation:
import timeimport statisticsdef evaluate_model(model_client, dataset, scorer, price_in, price_out):"""Evaluate a single model and return summary results for quality, cost, and latency."""records = []for item in dataset:start = time.perf_counter()resp = model_client.generate(item["prompt"], temperature=0)latency = time.perf_counter() - startscore = scorer(resp.text, item["reference"])cost = resp.input_tokens * price_in + resp.output_tokens * price_outrecords.append({"score": score,"cost": cost,"latency": latency,"passed": score >= item.get("pass_threshold", 0.8),})pass_rate = sum(r["passed"] for r in records) / len(records)total_cost = sum(r["cost"] for r in records)latencies = sorted(r["latency"] for r in records)p95 = latencies[int(len(latencies) * 0.95) - 1]return {"pass_rate": pass_rate,"avg_cost_per_task": total_cost / len(records),"effective_cost": (total_cost / len(records)) / max(pass_rate, 1e-9),"median_latency": statistics.median(latencies),"p95_latency": p95,}
The effective_cost (effective unit cost) is the key metric in this code because it combines cost with pass rate. A model with a low unit price but a 50% pass rate can cost more in practice than a model with a higher unit price and a 90% pass rate.
4. Comparing Models by Effective Unit Cost
Suppose there are two candidate models. The figures below only illustrate the calculation method. They do not represent data from any actual product.
| Metric | Model A | Model B |
|---|---|---|
| Cost per call (CNY) | 0.010 | 0.025 |
| Task pass rate | 60% | 92% |
| Effective unit cost (CNY per qualified task) | 0.0167 | 0.0272 |
By unit price, Model A costs more than twice as little as Model B. However, when measured per qualified task, Model B costs only about 60% more. If manual review time is included, the gap may shrink further or even reverse.
This shows that model selection should use “the cost of each qualified result” as the primary basis, with unit price as a reference only.
5. Reducing Costs Through Engineering
Even after a model is selected, several engineering methods can still reduce costs.
Streamline prompts. Remove repeated instructions and redundant examples, and keep only the information the model needs to complete the task. Some services support caching of stable prompt prefixes, which can lower the billing for repeated content. The exact rules depend on the provider’s documentation.
Control output length. Specify the output format and a maximum length to prevent the model from generating irrelevant content. Structured output, such as JSON with fixed fields, also reduces retries caused by parsing failures.
Route requests by tier. Choose a model based on task difficulty. Send simple tasks to a smaller model and escalate complex tasks to a larger one. The following simplified routing example shows the idea:
def route_request(task):"""Select a model tier based on task characteristics and return the model identifier."""if task.get("requires_reasoning") or task.get("input_length", 0) > 8000:return "large-tier"if task.get("type") in {"classification", "extraction"}:return "small-tier"return "medium-tier"
Tiered routing should be used together with regression testing. After a routing rule goes live, keep monitoring the pass rate of each tier to make sure simple tasks are not assigned to a model that lacks the required capability.
Cache results. For highly repetitive requests, such as fixed FAQ questions, cache model outputs to reduce duplicate calls.
Use asynchronous and batch processing. For tasks with low real-time requirements, batch submission makes better use of throughput and reduces scheduling overhead.
6. Common Mistakes
Looking only at unit price. As explained above, unit price does not reflect pass rate, retries, or manual review cost.
Using general leaderboards instead of business evaluation. Leaderboards measure general capability. Business scenarios often differ greatly in terminology, format, and data distribution.
Ignoring tail latency. The average latency may look normal while P99 is high, which can noticeably degrade the experience for some users. Monitor percentile metrics as well.
Ignoring version drift. After a model version is updated, output style, format, and refusal behavior may change. Keep a fixed regression test set and run it automatically after every version change.
Treating model selection as a one-time decision. Model prices, capabilities, and service policies all change over time. Reevaluate periodically, typically when a version is updated or the budget changes.
7. Implementation Recommendations
- Build a model-calling abstraction layer. Business code should depend only on a unified interface. When switching models, only the configuration needs to change, which noticeably reduces migration cost.
- Set cost budgets and alerts. Track call volume and spending by project or tenant, connect them to monitoring and alerting, and notify the owner when a threshold is exceeded.
- Maintain evaluation assets. Store evaluation sets, scoring scripts, and results under version control so that the team can reuse and audit them.
- Formalize the selection process. Define the requirements input, evaluation criteria, decision owner, and reevaluation schedule so that selection does not depend on individual experience.
In summary, LLM selection should center on business tasks and rely on effective unit cost, latency distribution, and stability for overall judgment. Only a reproducible evaluation system can support consistent, sound decisions in an environment where models change quickly.
