# LLM Upgrade Evaluation Methodology: Balancing Accuracy, Latency, and Cost

    When a new LLM version launches, the headline benchmark score tells only part of the story. This article explains why aggregate scores can hide domain-level regressions and how to build a test set and control variables so that versions can be compared fairly. It also covers how to measure accuracy (including per-domain results and bootstrap confidence intervals), latency (time to first token versus full response time, with P50/P95/P99), token consumption, cost per call, and the marginal cost per accuracy point gained. A Python evaluation script shows how to put these metrics into practice.

    1. Why a Version Upgrade Is More Than the Overall Score

    When a new model version is released, vendors usually highlight improvements on a few specific metrics. The real question engineering teams need to answer is how much benefit the new version will bring in their own business scenarios, and what that benefit costs. Accuracy is only part of the benefit. Response time, token consumption, and cost per call together determine the actual value of a single call.

    A common mistake is to look only at the overall score. That score is a weighted average across multiple sub-domains, so a drop in one area can be masked by gains in others. For example, if your product relies on tool calling and that capability regresses, execution will still fail in production even when the overall score goes up. Evaluation therefore needs to look at both the overall score and the per-domain results.

    2. Building a Comparable Evaluation Foundation

    2.1 Build a Unified Test Set

    The test set determines how much you can trust the evaluation results. Build it according to these principles:

    • Cover business scenarios: Divide domains according to your actual request distribution, such as Q&A, summarization, math reasoning, code generation, tool calling, and structured output.
    • Sufficient sample size: Make sure each key domain has enough samples so that small-sample noise is not mistaken for a change in capability.
    • Clear labeling standards: Agree in advance on how correct answers are judged, and distinguish between exact match, semantic equivalence, and format compliance.
    • Keep failure samples: Count samples with timeouts, errors, or invalid output formats in the statistics rather than discarding them.

    2.2 Control Variables

    The baseline and candidate versions must run under the same conditions. These conditions include prompt templates, system instructions, sampling temperature, maximum output length, context length, and retry policy. Otherwise, differences in the metrics may come from configuration changes rather than from the model itself.

    A new version may respond differently to the same prompt. It is therefore advisable to redo prompt tuning on the new version before comparing it with the old one, and to record the results of both groups separately.

    3. Core Metrics and How to Calculate Them

    3.1 Accuracy

    Accuracy is defined as:

    Accuracy = number of correctly judged samples / number of valid samples

    Two points deserve attention. First, agree in advance on the definition of “valid samples.” Treating timeouts and format errors as failures and excluding them from the count can lead to very different conclusions. Second, report per-domain accuracy alongside overall accuracy. If sample sizes vary significantly across domains, weight them by business importance and provide a confidence interval. For example, repeatedly resample the accuracy difference between the two versions using the bootstrap method to check whether the difference goes beyond random variation.

    3.2 Response Time

    Average latency can hide tail issues. A small fraction of extremely slow requests can noticeably hurt the user experience while barely moving the average. Report P50, P95, and P99 along with the average.

    Two types of latency should also be distinguished:

    • Time to First Token (TTFT): The time from sending a request to receiving the first output token. It directly affects perceived speed in streaming output scenarios.
    • Full Response Time: The time from sending a request to receiving the complete response. It affects throughput for batch tasks.

    For online conversational applications, focus mainly on TTFT and P95. Offline tasks should pay more attention to full response time and throughput.

    3.3 Token Consumption

    Count input tokens and output tokens separately. If a new version uses a longer internal reasoning process, output tokens will increase noticeably. In that case, check whether the provider’s billing includes tokens generated during reasoning, and how those tokens appear on the bill.

    3.4 Cost per Call

    Cost per call is calculated as follows:

    Cost per call = (average input tokens × input price + average output tokens × output price) / 1,000,000

    Cost per 1,000 calls = cost per call × 1,000

    Use the provider’s official pricing, which is stated per million tokens, and note the currency and the pricing date.

    The following example uses hypothetical numbers: an average of 2,000 input tokens and 2,000 output tokens, an input price of 1 CNY per million tokens, and an output price of 4 CNY per million tokens.

    • Cost per call = (2000 × 1 + 2000 × 4) / 1,000,000 = 0.01 CNY
    • Cost per 1,000 calls = 10 CNY

    Suppose output tokens increase by 30% and everything else stays the same. Output tokens then rise to 2,600, and the cost per call becomes (2000 × 1 + 2600 × 4) / 1,000,000 = 0.0124 CNY. That equals 12.4 CNY per 1,000 calls, a 24% increase. When prices stay the same, the cost increase comes entirely from higher token consumption. The price gap between input and output also amplifies the impact of changes in output tokens.

    3.5 Marginal Cost

    Whether a version upgrade is worthwhile depends mainly on how much extra cost each percentage point of accuracy improvement requires:

    Marginal cost = increase in cost per 1,000 calls / increase in accuracy (percentage points)

    This metric lets you compare accuracy gains and cost on the same scale. If the marginal cost in a core domain is far above what the business can tolerate, a full switch is not advisable even if the overall score improves.

    4. Example Evaluation Script

    The code below shows the basic structure of an evaluation framework. Replace call_model and is_correct with your actual model-calling function and scoring logic.

    1. import statistics
    2. import time
    3. from collections import defaultdict
    4. def percentile(values, p):
    5. """Return the approximate p-th percentile of a sequence."""
    6. ordered = sorted(values)
    7. idx = round(p / 100 * (len(ordered) - 1))
    8. return ordered[max(0, min(idx, len(ordered) - 1))]
    9. def evaluate(call_model, is_correct, samples, price_in, price_out):
    10. """
    11. call_model(prompt) -> dict containing text, input_tokens, output_tokens
    12. is_correct(text, sample) -> bool, judged against task-specific rules
    13. price_in / price_out: unit price per million tokens
    14. """
    15. correct = defaultdict(int)
    16. total = defaultdict(int)
    17. latencies, in_tok, out_tok = [], [], []
    18. for s in samples:
    19. start = time.perf_counter()
    20. try:
    21. resp = call_model(s["prompt"])
    22. except Exception:
    23. # Failed samples count toward the total and are not removed
    24. resp = {"text": "", "input_tokens": 0, "output_tokens": 0}
    25. latencies.append(time.perf_counter() - start)
    26. in_tok.append(resp["input_tokens"])
    27. out_tok.append(resp["output_tokens"])
    28. total[s["domain"]] += 1
    29. if is_correct(resp["text"], s):
    30. correct[s["domain"]] += 1
    31. n = len(samples)
    32. mean_in = statistics.mean(in_tok)
    33. mean_out = statistics.mean(out_tok)
    34. cost_per_call = (mean_in * price_in + mean_out * price_out) / 1_000_000
    35. return {
    36. "accuracy": sum(correct.values()) / n,
    37. "domain_accuracy": {d: correct[d] / total[d] for d in total},
    38. "latency_p50": percentile(latencies, 50),
    39. "latency_p95": percentile(latencies, 95),
    40. "avg_tokens": mean_in + mean_out,
    41. "cost_per_1k_calls": cost_per_call * 1000,
    42. }
    43. def compare(baseline, candidate, domain_tolerance=0.0):
    44. """Compare two versions on accuracy, cost, tail latency, and regressed domains."""
    45. regressed = [
    46. d for d, base_acc in baseline["domain_accuracy"].items()
    47. if candidate["domain_accuracy"].get(d, 0.0) < base_acc - domain_tolerance
    48. ]
    49. acc_delta_pp = (candidate["accuracy"] - baseline["accuracy"]) * 100
    50. cost_delta = candidate["cost_per_1k_calls"] - baseline["cost_per_1k_calls"]
    51. marginal_cost = cost_delta / acc_delta_pp if acc_delta_pp > 0 else float("inf")
    52. return {
    53. "accuracy_delta_pp": acc_delta_pp,
    54. "cost_delta_pct": cost_delta / baseline["cost_per_1k_calls"] * 100,
    55. "latency_p95_delta_pct": (
    56. candidate["latency_p95"] / baseline["latency_p95"] - 1
    57. ) * 100,
    58. "marginal_cost_per_pp": marginal_cost,
    59. "regressed_domains": regressed,
    60. }

    Three design choices in the script are worth noting. First, failed requests are counted in the total number of samples so that averages are not artificially inflated. Second, per-domain results are output separately, which makes regressions hidden by the overall score easy to spot. Third, the comparison function returns the marginal cost directly, which makes decision-making easier.

    5. Interpreting the Results

    When you have the comparison report, start with regressed_domains. If a core domain has regressed, do not fully switch to the new version, even if the overall score has risen. Next, check accuracy_delta_pp together with its confidence interval to see whether the improvement is real rather than random variation. Then compare marginal_cost_per_pp with what the business can afford. If the value is extremely large, it can mean that cost has risen much faster than accuracy, or that the accuracy gain is small or even negative.

    Finally, look at latency_p95_delta_pct in the context of the workload. For online services, a large increase in P95 latency can hurt the user experience even when accuracy and cost look acceptable. Only when there are no regressed core domains, the accuracy gains are reliable, the marginal cost is acceptable, and tail latency stays within limits should you switch to the new version.