LLM Model Selection: A Four-Metric Evaluation Framework for Accuracy, Latency, Token Usage, and Cost
This article presents a practical framework for evaluating large language model upgrades and comparing model options using four metrics: accuracy, average latency, average token consumption, and cost per call. It walks through an example version-to-version comparison, explains how to compare models within the same cost tier, provides a weighted scoring method, and offers engineering recommendations for private evaluation sets, controlled testing, model routing, and Agent regression tests. The example data is illustrative only and is intended to demonstrate the evaluation and calculation process rather than represent any vendor’s official results.
1. A Note on the Data
The numbers in this article are example data. They are used to demonstrate the evaluation method and calculation process, and they do not represent any vendor’s official results. Readers who want to cite these conclusions should remeasure them with their own evaluation sets and API configurations.
All cost figures include the calculation process so they can be checked. The example cost calculation counts only output tokens; input tokens and cache discounts are not included. This simplification affects the absolute cost and should be corrected in real evaluations.
2. Four Core Evaluation Metrics
- Accuracy: The percentage of questions the model answers correctly. It reflects the model’s performance on the target capability.
- Average latency: The average time from sending a request to receiving the complete result. It directly affects user waiting time and system throughput.
- Average token consumption: The number of input and output tokens consumed per call. Reasoning models often generate longer thinking traces, which can significantly increase output tokens.
- Cost per call: The cost calculated according to billing rules, usually expressed as the amount per 1,000 calls for easier comparison.
Cost should be calculated separately for input and output:
Cost per call = input tokens × input unit price + output tokens × output unit price
In actual billing, teams also need to confirm several details: whether cache hits receive a discount, whether tiered pricing or batch discounts apply, and whether failed retries are billed. The example cost in this article counts only the output portion; input and cache discounts are not included. This simplification should be accounted for in production evaluations.
Accuracy represents benefit, while latency and token usage represent cost. Monetary cost converts both into spending. These dimensions must be evaluated together. Looking only at accuracy can easily overstate the value of an upgrade.
3. Version-to-Version Comparison: Capability Gains and Trade-offs
Assume that an old version and a new version are tested on a Chinese-language evaluation set of about 15,000 questions. The results are as follows (example data; output unit price is CNY 1 per million tokens):
| Metric | Old version | New version | Change |
|---|---|---|---|
| Overall accuracy | 68.1% | 71.5% | +3.4 pp |
| Reasoning and math | 66.7% | 72.7% | +6.0 pp |
| Medical and mental health | 77.6% | 83.6% | +6.0 pp |
| Finance | 77.7% | 84.2% | +6.5 pp |
| Education | 46.9% | 52.0% | +5.1 pp |
| Language and instruction following | 64.6% | 69.3% | +4.7 pp |
| Agent and tool use | 69.9% | 68.1% | -1.8 pp |
| Average token consumption per call | 2,851 | 3,922 | +37.6% |
| Average latency per call | 59 s | 96 s | +~63% |
| Cost per 1,000 calls | CNY 2.85 | CNY 3.92 | +37.6% |
The cost calculation is as follows. The old version consumes 2,851 output tokens per call. For 1,000 calls, that is 2.851 million tokens. At CNY 1 per million tokens, the cost is about CNY 2.85. The new version consumes 3,922 tokens per call. For 1,000 calls, that is 3.922 million tokens, or about CNY 3.92. When the unit price is unchanged, the cost increase is exactly the same as the token increase: about 37.6%.
Several changes stand out.
First, overall accuracy improves by 3.4 percentage points, which is a clear generational upgrade. Second, reasoning and math improve by 6.0 percentage points, consistent with the industry direction of extending reasoning chains to raise the performance ceiling. Third, domains with high knowledge density and high error costs, such as healthcare and finance, also show clear gains. Fourth, instruction-following improves, which directly reduces prompt engineering and debugging work at the application layer.
The fifth point requires caution. Agent and tool-use capability drops by 1.8 percentage points. One possible explanation is that stronger reasoning makes the model prefer to think before acting. In scenarios that require fast, multi-turn tool calls, the decision chain becomes longer, leading to overthinking. This explanation still needs to be verified with dedicated Agent tests; it cannot be inferred from the total score alone.
The trade-offs are also real. Token consumption increases by about 38%, latency by about 63%, and cost by about 38%. Teams need to judge whether this trade-off is worthwhile for their business type. Offline analysis and complex question answering are usually accuracy-sensitive and latency-tolerant, so they can often accept this cost. High-frequency, low-latency online services should be cautious, because increased latency may directly affect user retention.
4. Cross-Model Comparison: Options Within the Same Cost Tier
A version-to-version comparison answers “should we upgrade?” A cross-model comparison answers “is there a better option?” The approach is to first group models by measured cost tier, then compare accuracy and latency within the same tier.
Within the same tier, if two models have similar accuracy, the faster model usually has an advantage. If one model has higher accuracy but costs twice as much as another, teams need to weigh the decision against the cost of errors. For example, when an incorrect answer can trigger customer complaints or compliance risk, the extra cost may be justified. Conversely, if errors only affect the polishing of a small amount of content, there is no need to pay a high price for a small accuracy gain.
This article does not list specific competitor numbers because the example data cannot be verified, and citing it directly would mislead readers. Teams should use their own data to compare models within the same tier.
To put accuracy, speed, and cost into a single scoring system, teams can use the following formula:
score = w1 × normalized accuracy + w2 × normalized speed + w3 × normalized cost
The three weights sum to 1. The normalization method is to divide each metric by the maximum value of that metric among the candidate models. For latency and cost, lower values are better, so take the reciprocal first and then apply the same normalization. Weights should be set according to the business, for example:
- Customer service QA: w1 = 0.6, w2 = 0.2, w3 = 0.2; accuracy first.
- Real-time chat: w1 = 0.3, w2 = 0.5, w3 = 0.2; response speed first.
- High-concurrency batch tasks: w1 = 0.3, w2 = 0.2, w3 = 0.5; cost first.
5. Practical Recommendations for Engineering Teams
Build a private evaluation set. Stratified-sample 500 to 2,000 examples from real business logs, covering major scenarios and typical hard cases, and label the expected outputs. This reflects actual performance better than public leaderboards.
Control variables. When comparing models, keep temperature, maximum output length, system prompt, and concurrency consistent, and record the model version and test date. When configurations differ, conclusions often come from configuration differences rather than model differences.
Record all four metric categories. For every evaluation, record accuracy, average latency, input and output token counts, cost per call, and cache hit rate. Prices may change by tier or batch, so check the latest price sheet before each evaluation and avoid using stale data.
Route by scenario. Use stronger reasoning models for complex questions and lightweight models for simple requests to significantly reduce overall cost. Below is a simplified routing example. The model names and thresholds must be replaced with actual configuration:
DEEP_MODEL = "deep-reasoning-model" # Placeholder name; replace with actual model IDLIGHT_MODEL = "lightweight-model"DEFAULT_MODEL = "default-model"COST_BUDGET_PER_CALL = 0.001 # Per-call budget cap in CNY; example valuedef route(query_complexity: str, budget_per_call: float) -> str:if query_complexity not in ("low", "medium", "high"):return DEFAULT_MODEL # Unknown complexity: fall back to default modelif query_complexity == "high":return DEEP_MODELif budget_per_call < COST_BUDGET_PER_CALL:return LIGHT_MODELreturn DEFAULT_MODEL
Routing effectiveness depends on the accuracy of complexity classification. The classification logic itself also needs separate evaluation; otherwise, routing may assign hard questions to lightweight models.
- Run dedicated Agent regression tests. Build a dedicated test set for tool calling, and measure the number of calls, number of steps, and tool-call success rate. Do not assume that a higher total score means every dimension improved. The Agent capability regression in this example shows why.
6. Conclusion
In the example evaluation, the new version improves accuracy by 3.4 percentage points, but token consumption, latency, and cost increase by about 38%, 63%, and 38%, respectively, while Agent capability declines slightly. These conclusions apply only to the example data and configuration in this article and should not be directly extrapolated to other businesses.
Teams should build their own four-metric evaluation system, make weighted decisions based on the cost of errors in their business, and use model routing to balance capability and cost.
