LLM Evaluation in Practice: Balancing Accuracy, Latency, and Cost
This article explains how to evaluate large language models beyond a single accuracy score. It covers the four core dimensions of LLM evaluation—accuracy, response latency, token consumption, and cost—and how to interpret each correctly. You'll learn a repeatable workflow for deciding whether a model upgrade is worth it, why cross-model comparisons should stay within the same cost tier, and how to build a reusable evaluation framework. Complete with worked examples, an upgrade decision function, and an evaluation harness, this guide helps you turn model selection from a subjective debate into an auditable engineering decision. All numbers shown are illustrative only.
1. Why LLM Evaluation Can’t Rely on Accuracy Alone
When large language models move into production, LLM evaluation should never be reduced to a single accuracy score. Leading a public leaderboard doesn’t mean a model fits your specific workload. Choosing a model requires looking at four dimensions at the same time: whether the answer is correct (accuracy), how fast it responds (response latency), how verbose it is (token consumption), and how expensive it is to call (cost).
These four metrics are in natural tension. A model may generate longer chains of thought to improve accuracy, which drives up both token consumption and latency. On the other hand, a solution tuned for extremely low latency may sacrifice reasoning depth. So a sound LLM evaluation doesn’t just rank models—it answers a different question: given your cost budget and latency constraints, which model offers the best overall capability?
Note: All figures in this article—68%, 59 seconds, 96 seconds, 2851, 3922, ¥37, ¥52, and so on—are illustrative examples, not measured results. They exist only to explain the evaluation method. Real results depend on your own evaluation set, concurrency conditions, scoring criteria, and billing rules.
2. How to Interpret the Four Core Metrics
| Metric | How to collect it | Calculation | Risk |
|---|---|---|---|
| Accuracy | Scorer, rule matching, manual spot checks | Correct samples / total samples | Inconsistent scoring criteria make results incomparable |
| Response latency | Client-side or server-side instrumentation | Mean, P95, P99, time to first token | Looking only at the average ignores the long tail |
| Token consumption | API usage or local tokenizer | Input tokens + output tokens | Not separating input from output underestimates cost |
| Cost | Token consumption combined with unit price | Cost per 1,000 calls | Looking only at unit price ignores per-call consumption |
1. Accuracy: Look at the Breakdown, Not Just the Total
Accuracy is the most intuitive metric, but looking only at the overall score hides structural differences. Consider a typical Chinese-language general-capability evaluation over a set of tens of thousands of test questions. Suppose one version improves accuracy from 68% to 71%—a 3-point gain that may also push its ranking noticeably higher. That conclusion matters, but what’s truly valuable for engineering is the per-category breakdown:
- Reasoning and math: hypothetically rising from 66% to 72%, the largest gain;
- Educational tasks: hypothetically up 5 points from 46%, showing this was a clear weakness before;
- Vertical domains such as healthcare and finance: hypothetically up about 6 points each;
- Language and instruction following: hypothetically up about 4 to 5 points;
- Agents and tool calling: possibly regressing, going down instead of up.
Breaking results down by category reveals a key fact: an overall improvement is often a mix of gains and trade-offs. If a capability your business relies on heavily—say, tool calling—regresses, then a higher total score can’t justify the upgrade. The first principle of evaluation is this: weight categories by your own business distribution, not by the generic weights on a leaderboard.
2. Response Latency: Look Beyond the Average at Tail Latency
Average response time is the most common way to report latency, but for production systems the average badly masks long-tail problems. Suppose average latency for a call rises from 59 seconds to 96 seconds—that only tells you things got slower overall. Under high concurrency, however, it’s the P95 and P99 tail latencies that determine user experience and timeout rates.
When evaluating, collect all of the following:
- Average latency;
- P95 latency;
- P99 latency;
- Time to first token (TTFT) in streaming scenarios;
- Full response latency.
If a model supports streaming output, TTFT often affects perceived responsiveness more than full response latency. For interactive scenarios such as customer support, Copilots, and code completion, prioritize TTFT; for offline scenarios such as long reports and batch summarization, prioritize full response time and throughput.
3. Token Consumption: The Real Variable in the Cost Model
Many teams look only at unit price and overlook the fact that token consumption is what actually drives cost. When the output price stays the same, if a single call goes from consuming 2851 tokens to 3922—an increase of roughly 37%—the actual cost rises noticeably even if the unit price doesn’t change at all. This usually means the new model uses a more complex reasoning mechanism, such as longer chains of thought or more intermediate steps, trading tokens for accuracy.
When evaluating, track “unit price” and “per-call consumption” separately:
- Input token count;
- Output token count;
- Input unit price;
- Output unit price;
- Per-call cost;
- Cost per 1,000 calls.
Counting only total tokens without separating input from output easily underestimates cost, because for many models the output price is higher than the input price.
4. Cost: Combine the First Three into One Comparable Number
Finally, multiply token consumption by unit price to get a per-call cost, then convert it to “cost per 1,000 calls.” This step translates technical metrics into financial terms.
If the unit price is quoted per 1,000 tokens, the formula is:
cost per 1,000 calls = avg input tokens / 1000 × input unit price + avg output tokens / 1000 × output unit price
If the unit price is quoted per single token, first convert it to a per-1,000-token price, then calculate.
Suppose the old version costs about ¥37 per 1,000 calls and the new version rises to ¥52—an increase of roughly 40%. At this point you need to weigh it against business value: accuracy up 3 points, latency increased, cost up 40%. Whether that trade is worth it depends on how sensitive the scenario is to accuracy. High-value decisions and healthcare, finance, and legal-compliance scenarios may accept higher cost for higher accuracy; high-concurrency customer support, batch summarization, and lightweight Q&A may care more about latency and cost.
3. Version Iteration Evaluation: Deciding Whether an Upgrade Is Worth It
When facing a model upgrade, use the following process to make a quantitative decision:
- Fix the test set: old and new versions must be evaluated on exactly the same set of questions, otherwise the comparison is invalid;
- Fix the parameters: keep temperature, top_p, maximum output length, system prompt, and retry policy as consistent as possible;
- Compare by category, not by total: compare category by category, identify improvements and regressions, and weigh them against business priorities;
- Compute the combined cost: derive the cost per 1,000 calls rather than comparing unit prices alone;
- Assess the latency impact: quantify how latency changes affect timeout rates and concurrency capacity;
- Make a weighted decision: use relative change rates to compute upgrade gains, avoiding directly adding quantities with different units.
Below is an upgrade decision function you can adapt directly. It uses relative change rates so that accuracy, latency, and cost all convert into dimensionless gain values, avoiding the unit confusion of mixing “accuracy + 1/second + 1/yuan.”
def should_upgrade(old, new, weights):"""old/new: metric dicts for each version, e.g. containing accuracy, latency_p95, cost_per_1kweights: non-negative weights, ideally normalized, e.g.:{"accuracy": 0.5, "latency": 0.2, "cost": 0.3}Returns:bool: whether an upgrade is recommendedfloat: combined gain score; greater than 0 means overall better"""eps = 1e-9# An accuracy improvement is a positive gaingain_accuracy = (new["accuracy"] - old["accuracy"]) / max(old["accuracy"], eps)# Lower latency is a positive gaingain_latency = (old["latency_p95"] - new["latency_p95"]) / max(old["latency_p95"], eps)# Lower cost is a positive gaingain_cost = (old["cost_per_1k"] - new["cost_per_1k"]) / max(old["cost_per_1k"], eps)score = (weights.get("accuracy", 0) * gain_accuracy+ weights.get("latency", 0) * gain_latency+ weights.get("cost", 0) * gain_cost)return score > 0, score
The advantages of this approach are:
- Every metric is converted to a relative change, so units are consistent;
- Latency and cost don’t need to be inverted;
- All weights are non-negative gain weights;
- When accuracy, latency, and cost all get worse, the combined gain is negative;
- You can set an upgrade threshold based on business needs, for example requiring a combined gain greater than 2% before upgrading.
This turns “should we upgrade?” from a subjective argument into a number that can be recomputed and reviewed.
4. Horizontal Comparison: Only Within the Same Cost Tier
When comparing across models, the biggest mistake is pitting different cost tiers against each other. Comparing the accuracy of a model that costs tens of yuan per 1,000 calls against one that costs hundreds of yuan is meaningless, because the money isn’t in the same ballpark. The right approach is to first bucket models by cost, then compare within each bucket.
Start by building a same-cost-tier comparison table template:
| Cost tier | Model | Accuracy | P95 latency | Cost per 1,000 calls | Conclusion |
|---|---|---|---|---|---|
| ¥50 tier | Model A | 71% | 6.2s | ¥48 | Balanced accuracy and cost, recommended |
| ¥50 tier | Model B | 71% | 9.5s | ¥51 | Similar accuracy, but higher latency |
| ¥50 tier | Model C | 69% | 5.8s | ¥45 | Low cost, low latency, good for high-frequency use |
| ¥100 tier | Model D | 76% | 8.0s | ¥98 | High accuracy, good for high-value scenarios |
The data in the table above is an example template; replace it with your own evaluation results when filling it in.
Within the same ¥50 tier, if multiple models have the same accuracy—say, all 71%—then the deciding factor shifts to latency: among models with equal accuracy and equal cost, the one that responds faster is the better choice. Conversely, if multiple models have similar latency, then compare accuracy, token consumption, and stability.
This kind of “compare latency within the same cost” or “compare accuracy within the same latency” Pareto comparison is the core of selection decisions. Evaluation conclusions are also scenario-dependent: a combination that leads on general Chinese-language tasks may not win in English or in a specific vertical domain. The language and domain distribution of your evaluation set must match real traffic, otherwise leaderboard rankings have limited reference value.
5. A Reusable LLM Evaluation Framework Skeleton
A practical LLM evaluation framework should cover at least: a fixed test set, accuracy judging, latency statistics, token statistics, cost combination, tail-latency protection, and concurrency load-testing notes. Below is an example that’s closer to a production environment.
import timeimport mathimport statisticsdef percentile(sorted_values, p):"""Nearest-rank percentile calculation, avoiding out-of-bounds errors fromint(n * p) or distortion when the sample size is small."""if not sorted_values:return Nonen = len(sorted_values)index = min(max(math.ceil(p * n) - 1, 0), n - 1)return sorted_values[index]def evaluate(client, test_set, price_in_per_1k, price_out_per_1k):"""The object returned by client.call(prompt, stream=True) must contain:- text: model output text- usage: optional, containing prompt_tokens and completion_tokens- first_token_latency: optional, time to first token in streaming scenariosprice_in_per_1k / price_out_per_1k:input and output price per 1,000 tokens."""records = []for case in test_set:# Count input tokens before the call onlyprompt_tokens = count_tokens(case.prompt, mode="input")start = time.perf_counter()resp = client.call(case.prompt, stream=True)# In streaming scenarios, record time to first token; for non-streaming,# the first returned chunk can be used as a substitutettft = getattr(resp, "first_token_latency", None)latency = time.perf_counter() - start# Prefer the usage returned by the API; otherwise count output text separatelyusage = getattr(resp, "usage", None)if usage is not None:prompt_tokens = usage.prompt_tokenscompletion_tokens = usage.completion_tokenselse:completion_tokens = count_tokens(resp.text, mode="output")# price_in_per_1k / price_out_per_1k are both per-1,000-token pricescost = (prompt_tokens / 1000.0 * price_in_per_1k+ completion_tokens / 1000.0 * price_out_per_1k)records.append({"correct": judge(resp.text, case.golden),"latency": latency,"ttft": ttft,"prompt_tokens": prompt_tokens,"completion_tokens": completion_tokens,"tokens": prompt_tokens + completion_tokens,"cost": cost,})n = len(records)if n == 0:return {}latencies = sorted(r["latency"] for r in records)ttfts = sorted([r["ttft"] for r in records if r["ttft"] is not None])return {"accuracy": sum(r["correct"] for r in records) / n,"avg_latency": statistics.mean(latencies),"p95_latency": percentile(latencies, 0.95),"p99_latency": percentile(latencies, 0.99),"avg_ttft": statistics.mean(ttfts) if ttfts else None,"p95_ttft": percentile(ttfts, 0.95) if ttfts else None,"avg_tokens": statistics.mean(r["tokens"] for r in records),"cost_per_1k": statistics.mean(r["cost"] for r in records) * 1000,}
Things to note when using this framework:
count_tokens,judge,price_in_per_1k, andprice_out_per_1kmust be replaced according to your actual tokenizer, scoring rules, and billing basis;- Input tokens are counted before the call, output tokens after the call;
- If the API returns
usage, prefer the token counts inusage; - When the sample size is small, P95 and P99 tail latencies are easily distorted; it’s recommended that the evaluation set cover the real traffic distribution and grow to a statistically meaningful sample size;
- If you care about production behavior, load-test under your target concurrency or target QPS, distinguishing single-request latency, concurrency queueing latency, and resource bottlenecks;
- In streaming scenarios you must record TTFT, because perceived user experience often comes from how fast the first token appears, not the full response time.
6. Common Questions and Common Pitfalls
Common Pitfalls in LLM Evaluation
- Looking only at the total, not the breakdown: an upgrade may cause a regression in some critical capability, so verify category by category;
- Looking only at unit price, not consumption: token consumption is the real cost variable; always combine it into a cost per 1,000 calls;
- Reporting only the average, ignoring the long tail: production systems need P95 and P99 tail latencies;
- Comparing across tiers: comparisons are only meaningful within the same cost tier;
- Replacing the business set with a generic leaderboard: the language and domain distribution of the evaluation set must align with real traffic;
- Ignoring scoring criteria: accuracy depends heavily on how you score; inconsistent criteria make horizontal data incomparable;
- Ignoring streaming experience: measuring only full response latency without measuring time to first token easily leads to misjudging the interactive experience;
- Ignoring concurrency effects: good results from sequential single-machine calls don’t mean stability under high concurrency.
Choosing Scoring Criteria
- Strict matching: suited to multiple choice, classification, rule extraction, and fixed-format output;
- Semantic similarity: suited to summarization, question answering, intent understanding, and text rewriting;
- Model-based review: suited to open-ended generation, but requires a fixed judge model, prompt, and scoring rubric;
- Manual spot checks: suited to high-risk scenarios, complaint-heavy scenarios, and compliance-sensitive scenarios.
Whichever criteria you use, keep them consistent within the same round of evaluation. Otherwise, accuracy between different models or versions isn’t comparable.
Frequently Asked Questions
How do I do LLM evaluation?
First define a business test set, then fix the prompts and parameters, then record accuracy, latency, token consumption, and cost at the same time, and finally make a weighted decision based on business priorities.
How do I choose a model?
First bucket models by cost per 1,000 calls, then compare accuracy, P95 latency, and stability within the same tier. At equal cost, prefer the model with lower latency and more stable results.
How is token cost calculated?
Track input and output separately. If the API returns usage, prefer it; if not, count with a tokenizer before and after the call. Finally combine it into a cost per 1,000 calls.
What should I pay attention to when evaluating an upgrade?
You must use the same test set, the same parameters, and the same scoring criteria. Beyond the total score, also look at per-category regressions, P95/P99 latency, changes in output length, cost increase, and how well it matches business weights.
7. Summary
The essence of LLM evaluation is finding the point that best matches your business goals among four mutually constraining dimensions: accuracy, response latency, token consumption, and cost. Whether an upgrade is worth it doesn’t depend on leaderboard ranking, but on whether the per-category capabilities align with business weights, whether the cost increase stays within budget, and whether the latency change is acceptable.
Master this approach—break down by category, compare within a tier, weigh with Pareto trade-offs, and evaluate terminal cost—and you’ll turn model selection from a guess-based argument into an engineering decision backed by evidence. For LLM evaluation, the most important thing isn’t producing a pretty total score, but building an evaluation system that can be recomputed, reused, and aligned with business goals.
