# LLM Upgrade Evaluation Methodology: Balancing Accuracy, Latency, and Cost
When a new LLM version launches, the headline benchmark score tells only part of the story. This article explains why aggregate scores can hide domain-level regressions and how to build a test set and control variables so that versions can be compared fairly. It also covers how to measure accuracy (including per-domain results and bootstrap confidence intervals), latency (time to first token versus full response time, with P50/P95/P99), token consumption, cost per call, and the marginal cost per accuracy point gained. A Python evaluation script shows how to put these metrics into practice.
1. Why a Version Upgrade Is More Than the Overall Score
When a new model version is released, vendors usually highlight improvements on a few specific metrics. The real question engineering teams need to answer is how much benefit the new version will bring in their own business scenarios, and what that benefit costs. Accuracy is only part of the benefit. Response time, token consumption, and cost per call together determine the actual value of a single call.
A common mistake is to look only at the overall score. That score is a weighted average across multiple sub-domains, so a drop in one area can be masked by gains in others. For example, if your product relies on tool calling and that capability regresses, execution will still fail in production even when the overall score goes up. Evaluation therefore needs to look at both the overall score and the per-domain results.
2. Building a Comparable Evaluation Foundation
2.1 Build a Unified Test Set
The test set determines how much you can trust the evaluation results. Build it according to these principles:
- Cover business scenarios: Divide domains according to your actual request distribution, such as Q&A, summarization, math reasoning, code generation, tool calling, and structured output.
- Sufficient sample size: Make sure each key domain has enough samples so that small-sample noise is not mistaken for a change in capability.
- Clear labeling standards: Agree in advance on how correct answers are judged, and distinguish between exact match, semantic equivalence, and format compliance.
- Keep failure samples: Count samples with timeouts, errors, or invalid output formats in the statistics rather than discarding them.
2.2 Control Variables
The baseline and candidate versions must run under the same conditions. These conditions include prompt templates, system instructions, sampling temperature, maximum output length, context length, and retry policy. Otherwise, differences in the metrics may come from configuration changes rather than from the model itself.
A new version may respond differently to the same prompt. It is therefore advisable to redo prompt tuning on the new version before comparing it with the old one, and to record the results of both groups separately.
3. Core Metrics and How to Calculate Them
3.1 Accuracy
Accuracy is defined as:
Accuracy = number of correctly judged samples / number of valid samples
Two points deserve attention. First, agree in advance on the definition of “valid samples.” Treating timeouts and format errors as failures and excluding them from the count can lead to very different conclusions. Second, report per-domain accuracy alongside overall accuracy. If sample sizes vary significantly across domains, weight them by business importance and provide a confidence interval. For example, repeatedly resample the accuracy difference between the two versions using the bootstrap method to check whether the difference goes beyond random variation.
3.2 Response Time
Average latency can hide tail issues. A small fraction of extremely slow requests can noticeably hurt the user experience while barely moving the average. Report P50, P95, and P99 along with the average.
Two types of latency should also be distinguished:
- Time to First Token (TTFT): The time from sending a request to receiving the first output token. It directly affects perceived speed in streaming output scenarios.
- Full Response Time: The time from sending a request to receiving the complete response. It affects throughput for batch tasks.
For online conversational applications, focus mainly on TTFT and P95. Offline tasks should pay more attention to full response time and throughput.
3.3 Token Consumption
Count input tokens and output tokens separately. If a new version uses a longer internal reasoning process, output tokens will increase noticeably. In that case, check whether the provider’s billing includes tokens generated during reasoning, and how those tokens appear on the bill.
3.4 Cost per Call
Cost per call is calculated as follows:
Cost per call = (average input tokens × input price + average output tokens × output price) / 1,000,000
Cost per 1,000 calls = cost per call × 1,000
Use the provider’s official pricing, which is stated per million tokens, and note the currency and the pricing date.
The following example uses hypothetical numbers: an average of 2,000 input tokens and 2,000 output tokens, an input price of 1 CNY per million tokens, and an output price of 4 CNY per million tokens.
- Cost per call = (2000 × 1 + 2000 × 4) / 1,000,000 = 0.01 CNY
- Cost per 1,000 calls = 10 CNY
Suppose output tokens increase by 30% and everything else stays the same. Output tokens then rise to 2,600, and the cost per call becomes (2000 × 1 + 2600 × 4) / 1,000,000 = 0.0124 CNY. That equals 12.4 CNY per 1,000 calls, a 24% increase. When prices stay the same, the cost increase comes entirely from higher token consumption. The price gap between input and output also amplifies the impact of changes in output tokens.
3.5 Marginal Cost
Whether a version upgrade is worthwhile depends mainly on how much extra cost each percentage point of accuracy improvement requires:
Marginal cost = increase in cost per 1,000 calls / increase in accuracy (percentage points)
This metric lets you compare accuracy gains and cost on the same scale. If the marginal cost in a core domain is far above what the business can tolerate, a full switch is not advisable even if the overall score improves.
4. Example Evaluation Script
The code below shows the basic structure of an evaluation framework. Replace call_model and is_correct with your actual model-calling function and scoring logic.
import statisticsimport timefrom collections import defaultdictdef percentile(values, p):"""Return the approximate p-th percentile of a sequence."""ordered = sorted(values)idx = round(p / 100 * (len(ordered) - 1))return ordered[max(0, min(idx, len(ordered) - 1))]def evaluate(call_model, is_correct, samples, price_in, price_out):"""call_model(prompt) -> dict containing text, input_tokens, output_tokensis_correct(text, sample) -> bool, judged against task-specific rulesprice_in / price_out: unit price per million tokens"""correct = defaultdict(int)total = defaultdict(int)latencies, in_tok, out_tok = [], [], []for s in samples:start = time.perf_counter()try:resp = call_model(s["prompt"])except Exception:# Failed samples count toward the total and are not removedresp = {"text": "", "input_tokens": 0, "output_tokens": 0}latencies.append(time.perf_counter() - start)in_tok.append(resp["input_tokens"])out_tok.append(resp["output_tokens"])total[s["domain"]] += 1if is_correct(resp["text"], s):correct[s["domain"]] += 1n = len(samples)mean_in = statistics.mean(in_tok)mean_out = statistics.mean(out_tok)cost_per_call = (mean_in * price_in + mean_out * price_out) / 1_000_000return {"accuracy": sum(correct.values()) / n,"domain_accuracy": {d: correct[d] / total[d] for d in total},"latency_p50": percentile(latencies, 50),"latency_p95": percentile(latencies, 95),"avg_tokens": mean_in + mean_out,"cost_per_1k_calls": cost_per_call * 1000,}def compare(baseline, candidate, domain_tolerance=0.0):"""Compare two versions on accuracy, cost, tail latency, and regressed domains."""regressed = [d for d, base_acc in baseline["domain_accuracy"].items()if candidate["domain_accuracy"].get(d, 0.0) < base_acc - domain_tolerance]acc_delta_pp = (candidate["accuracy"] - baseline["accuracy"]) * 100cost_delta = candidate["cost_per_1k_calls"] - baseline["cost_per_1k_calls"]marginal_cost = cost_delta / acc_delta_pp if acc_delta_pp > 0 else float("inf")return {"accuracy_delta_pp": acc_delta_pp,"cost_delta_pct": cost_delta / baseline["cost_per_1k_calls"] * 100,"latency_p95_delta_pct": (candidate["latency_p95"] / baseline["latency_p95"] - 1) * 100,"marginal_cost_per_pp": marginal_cost,"regressed_domains": regressed,}
Three design choices in the script are worth noting. First, failed requests are counted in the total number of samples so that averages are not artificially inflated. Second, per-domain results are output separately, which makes regressions hidden by the overall score easy to spot. Third, the comparison function returns the marginal cost directly, which makes decision-making easier.
5. Interpreting the Results
When you have the comparison report, start with regressed_domains. If a core domain has regressed, do not fully switch to the new version, even if the overall score has risen. Next, check accuracy_delta_pp together with its confidence interval to see whether the improvement is real rather than random variation. Then compare marginal_cost_per_pp with what the business can afford. If the value is extremely large, it can mean that cost has risen much faster than accuracy, or that the accuracy gain is small or even negative.
Finally, look at latency_p95_delta_pct in the context of the workload. For online services, a large increase in P95 latency can hurt the user experience even when accuracy and cost look acceptable. Only when there are no regressed core domains, the accuracy gains are reliable, the marginal cost is acceptable, and tail latency stays within limits should you switch to the new version.
