LLM Version Evaluation: A Four-Dimensional Metric Framework and Practical Deployment Methods

    When a new LLM version ships, vendors usually describe the improvement qualitatively, but engineering teams need decision-ready evidence rather than a leaderboard rank. This article proposes a four-dimensional evaluation framework covering quality, latency, token consumption, and cost, and explains why each dimension cannot substitute for another. Using a public Chinese-language benchmark of roughly 15,000 questions as an example, it shows how a 3.4-point accuracy gain came with a 37.6% rise in tokens, about 40% higher measured cost, and roughly 63% longer latency, while agent and tool-calling accuracy actually regressed 1.8 points. It also covers cost-tiered cross-model comparison, a reproducible evaluation pipeline with frozen configurations and structured per-call records, upgrade gates treated as hard constraints, scenario-based selection advice, and a checklist of common pitfalls.

    1. The Starting Point: “The New Version Is Stronger” Should Not Be Taken for Granted

    Every time a model version is released, the official materials usually describe the improvements qualitatively. But for engineering teams, the real question is not “is it stronger?” Three more specific questions matter:

    1. Stronger on which capability dimension: A single aggregate leaderboard score hides structural differences across domains, while the distribution of real production traffic is usually long-tailed.
    2. What does the improvement cost: The token consumption, latency increase, and measured cost from a larger inference budget directly determine whether the unit economics still hold.
    3. Are there local regressions: While the overall average rises, a critical capability (tool calling, instruction following) may have regressed, and that is exactly the part production is most sensitive to.

    So the goal of evaluation is not to produce a ranking, but to produce decision-ready evidence: a quantified report that can support a “switch / canary / keep” decision.

    2. The Four-Dimensional Metric System: Why You Must Look at All Four at Once

    A single-dimension evaluation almost inevitably leads to misjudgment. A complete version-comparison system needs at least four classes of metrics, and none can substitute for another.

    2.1 Quality: The Question Set Behind the Accuracy

    Accuracy is the only “outcome metric,” but its interpretability depends entirely on question-set design. You need to specify:

    • Question-set size and representativeness: Sample size determines the width of the confidence interval. At a scale of tens of thousands of questions (the example below uses about 15,000), differences within 1 percentage point usually fall inside statistical noise and should not drive selection.
    • Domain split: Sub-domains such as reasoning and math, education, healthcare and mental health, finance, language and instruction following, and agents and tool calling should be counted and reported separately. Aggregate scores are only for gauging trends, not for capacity planning.
    • Scoring criteria: Strict matching for multiple-choice questions, key-point recall for open-ended questions, and test-case pass rate for code questions cannot be merged into a single total score. The scorer version must be frozen together with the question-set version.

    2.2 Latency: The End-to-End Mean Is Almost Always Wrong

    The “average latency” commonly seen in evaluation reports is a statistic with high information loss. Real user experience is determined by the tail of the distribution:

    • Time to first token (TTFT) determines the feel of interactive scenarios;
    • Overall throughput (tokens/s) determines the cost and experience of long-output scenarios;
    • P50 / P95 / P99 determine timeout policies and retry rates;
    • The mean is only suitable for rough cross-version comparison.

    In practice, collect both the “queue wait” and the “pure inference” timings separately, otherwise load-test conclusions will be masked by server-side queuing.

    2.3 Token Consumption: Put “Thinking” on the Ledger

    A new generation of models generally introduces an explicit reasoning process, so the structure of output tokens changes. If the accounting does not distinguish them, you end up with the “tokens went up 37% but you can’t tell where” situation. At a minimum, split into three segments:

    1. total_tokens = prompt_tokens + completion_visible_tokens + reasoning_tokens

    where reasoning_tokens is the main source of version differences. It has three knock-on effects:

    • Higher cost: When billed by actual usage, cost is roughly linear in tokens.
    • Context crowding: Thinking content consumes window budget, squeezing the space available for dialogue history and tool return values.
    • Longer call chains: Each call becomes slower, and in serial orchestration (multi-round tool calls, chain reasoning) this is multiplied.

    2.4 Cost: An Unchanged Unit Price Does Not Mean Unchanged Cost

    This is the most easily overlooked point. Many version comparisons say “the unit price stays the same,” implying cost is unchanged. But if the tokens consumed per call increase, measured cost must rise. A suggested formula:

    1. def cost_per_1k_calls(avg_prompt_tokens, avg_completion_tokens,
    2. price_in_per_m, price_out_per_m):
    3. """Unit: CNY / 1k calls. Price in CNY / M token."""
    4. per_call = (avg_prompt_tokens * price_in_per_m
    5. + avg_completion_tokens * price_out_per_m) / 1_000_000
    6. return per_call * 1000

    A complete ledger should also include hidden costs: the call volume amplified by failed retries, the storage and logging overhead of long outputs, and the labor cost of a higher manual-review rate.

    3. Reading a Set of Public Benchmark Data

    A public Chinese-language general-capability benchmark (about 15,000 questions) provides a typical old-vs-new version comparison that works well as a methodology example:

    Metric Previous version New version Change
    Overall accuracy 68.1% 71.5% +3.4 pp
    Leaderboard rank 16 5 up 11 places
    Avg latency (per call) 59 s 96 s ~+63%
    Avg tokens (per call) 2851 3922 +37.6%
    Measured cost (CNY/1k calls) 37.6 52.5 ~+40%
    Output unit price 14.0 CNY/M token 14.0 CNY/M token unchanged

    3.1 Conclusion 1: The Capability Gain Is Bought with Reasoning Budget

    The output-side unit price is unchanged, yet measured cost rose by about 40%, roughly the same magnitude as the 37.6% rise in token consumption. This means the cost change is almost entirely explained by “longer thinking,” not by a pricing change. In other words, from a billing standpoint this improvement is buying accuracy with more tokens, the classic “test-time compute scaling” path.

    A simple back-of-envelope calculation from the public data: the average generation latency per token rose from about 20.7 ms to about 24.5 ms, meaning the total latency increase (63%) is clearly higher than the token increase (37.6%). This shows that the decoding efficiency of long reasoning output also dropped slightly, so the perceived degradation in serial long-chain scenarios is worse than the single-call number suggests.

    3.2 Conclusion 2: The Ranking Jump Is Driven by Leaderboard Density and Cannot Be Converted Directly to Capability

    Accuracy rose by only 3.4 points, yet the rank went from 16 to 5, which reflects how densely packed the top region is rather than a large absolute capability gain. Do not treat ranking as evidence of quality during selection; go back to absolute accuracy and performance on your own business question set.

    3.3 Conclusion 3: The Domain Structure Is “Broad Gains + a Single Regression”

    Sub-domain Previous version New version Change
    Finance 77.7% 84.2% +6.5
    Healthcare & mental health 77.6% 83.6% +6.0
    Reasoning & math 66.7% 72.7% +6.0
    Education 46.9% 52.0% +5.1
    Language & instruction following 64.6% 69.3% +4.7
    Agent & tool use 69.9% 68.1% -1.8

    The last row deserves the most attention. A regression in tool calling despite stronger reasoning is an explainable engineering phenomenon:

    • Output structure gets crowded out: As the chain of thought grows, the parameter format of tool calls is more likely to deviate from a strict schema, especially when multiple consecutive calls are needed.
    • Context budget gets consumed: Historical tool return values must be retained; once thinking occupies the window, less information remains available and planning quality drops.
    • Decisions get “heavier”: Longer internal reasoning tends to choose more conservative or more complex action sequences, which paradoxically adds failure points.

    The conclusion is direct: if your main production scenario is agents or tool orchestration, you must run a dedicated regression on tool-call success rate, parameter validity, and average number of calls before upgrading; an aggregate score alone is not enough.

    4. Cross-Model Comparison: Align by Cost, Not by Model

    A common mistake in cross-model comparison is ranking models from different price tiers in the same accuracy table. A more sensible approach is to first bucket by measured cost per 1k calls, then compare within the same tier.

    In one bucketing result at the ~50 CNY tier, the new version (52.5 CNY/1k calls) had the same accuracy as another reasoning-intensive model (53.5 CNY/1k calls), both at 71.5%, but the latter averaged 72 s against the former’s 96 s. This example shows two things:

    1. Same score, different price; same price, different speed is the norm, and cost tiers must be built on measured usage.
    2. At the same accuracy, a latency difference translates directly into a capacity difference. Suppose a single request needs 4 serial calls; the gap between 96 s and 72 s is amplified along the end-to-end chain by queuing effects beyond the raw 96 s vs 72 s figures, which pushes up the required number of concurrent replicas and indirectly raises the true TCO.

    So a 3D Pareto chart is recommended over a 2D ranking: measured cost on the x-axis, accuracy on the y-axis, and P95 latency as point size. Only candidates on the Pareto frontier are worth advancing to the next round of testing.

    One caution: this kind of public benchmark focuses mostly on Chinese general capability. Performance in English, minority languages, ultra-long context, code generation, and multimodal tasks cannot be inferred from it and must be re-tested with your own business sets.

    5. Engineering Practice: A Reproducible Evaluation Pipeline

    The credibility of an evaluation conclusion depends on reproducibility. Below are the variables that must be fixed in practice.

    5.1 Freezing the Runtime Configuration

    1. from dataclasses import dataclass
    2. @dataclass
    3. class EvalRunConfig:
    4. model_alias: str # Version alias, e.g. "candidate-v2"
    5. prompt_version: str # Prompt template version, version-controlled with the code
    6. dataset_version: str # Dataset snapshot ID, ensures cross-version comparability
    7. temperature: float = 0.0 # Can be set to 0.3 for subjective tasks, but keep consistent across versions
    8. top_p: float = 1.0
    9. max_tokens: int = 4096 # Too small truncates the reasoning process and distorts scores
    10. timeout_s: int = 180 # New-version latency is higher, so relax the timeout accordingly
    11. retries: int = 1 # Retries must be counted in cost accounting
    12. seed: int = 20260101
    13. cache: bool = False # Disable cache for eval, otherwise metrics are distorted
    14. concurrency: int = 8 # Concurrency affects queuing and P95, so it must be fixed

    Three common traps: a max_tokens set too low truncates the new version’s reasoning and unfairly lowers accuracy; reusing the old version’s timeout records normal long reasoning as failures; and enabling cache invalidates all token and latency statistics.

    5.2 Collection Layer: One Structured Record per Call

    1. record = {
    2. "case_id": case.id,
    3. "domain": case.domain, # Domain label, used for granular attribution
    4. "model_alias": cfg.model_alias,
    5. "prompt_tokens": resp.usage["prompt_tokens"],
    6. "completion_tokens": resp.usage["completion_tokens"],
    7. "reasoning_tokens": resp.usage.get("completion_tokens_details", {}).get("reasoning_tokens", 0),
    8. "ttft_ms": first_token_ts - request_start_ts,
    9. "e2e_ms": finished_ts - request_start_ts,
    10. "status": "ok", # ok / timeout / error / truncated
    11. "score": judge(case, resp.text), # Computed by an independent judge
    12. "cost": compute_cost(resp.usage),
    13. }

    With this detail, all metrics from the first four sections can be recomputed on demand: accuracy is filtered by status == "ok" and then aggregated by domain; latency is computed directly as P50/P95/P99; and cost can be summed per call or broken down to domain granularity to find “which scenarios are burning money.”

    5.3 Decision Layer: Setting Upgrade Gates

    1. GATES = {
    2. "overall_accuracy_min_delta": 0.0, # The overall score must not drop
    3. "agent_tool_success_min_delta": 0.0, # Critical sub-domain: strict zero regression
    4. "cost_per_1k_max": 60.0, # Measured cost cap (CNY/1k calls)
    5. "p95_latency_ms_max": 150_000, # P95 latency cap
    6. "min_sample_per_domain": 200, # Min sample per domain, to avoid noise-driven wins
    7. }
    8. def should_promote(cur: Metrics, base: Metrics, gates: dict) -> dict:
    9. report = {}
    10. for domain, m in cur.by_domain.items():
    11. b = base.by_domain[domain]
    12. delta = m.accuracy - b.accuracy
    13. key = "agent_and_tool_use" if domain == "agent_and_tool_use" else "default"
    14. floor = gates[f"{key}_accuracy_min_delta"]
    15. report[domain] = {
    16. "delta": round(delta, 4),
    17. "pass": delta >= floor and m.n >= gates["min_sample_per_domain"],
    18. }
    19. report["cost_pass"] = cur.cost_per_1k <= gates["cost_per_1k_max"]
    20. report["latency_pass"] = cur.p95_latency_ms <= gates["p95_latency_ms_max"]
    21. report["promote"] = (all(v["pass"] if isinstance(v, dict) else v
    22. for v in report.values() if isinstance(v, (dict, bool)))
    23. and report["cost_pass"] and report["latency_pass"])
    24. return report

    The core idea of gating is to make “capability regression” and “cost overrun” hard constraints, and treat “capability improvement” as a bonus. This avoids accepting the degradation of a critical business domain just to chase a higher overall score.

    5.4 A Checklist of Common Pitfalls

    1. Drawing conclusions from a single run: The new version has a longer reasoning chain, so output stability needs repeated runs to take the mean and variance.
    2. Reporting only the mean, not the distribution: Long-tail requests determine online timeout and retry-storm rates.
    3. Ignoring prompt coupling: New versions often need prompt-template adjustments to show their capability, but “tuning prompts for the new task” and “comparing directly with the old prompt” are two different sets of conclusions and should be reported separately.
    4. Question-set contamination: Public question sets easily enter training corpora, inflating absolute scores; trend comparison is more reliable.
    5. Inconsistent cost accounting: Treating an unchanged unit price as unchanged cost is the most typical risk in this comparison.
    6. Ignoring serial amplification: On multi-step agent chains, the per-step latency and token increases are amplified by the number of rounds.

    6. Scenario-Based Selection Recommendations

    Using the method above, a decision matrix can guide the version strategy:

    • Deep-reasoning scenarios (complex computation, finance and healthcare knowledge Q&A, offline batch processing): accuracy first, high latency tolerance. In the example data, these scenarios gained more than 6 points, and a 37% token increase is an acceptable cost, so upgrade directly and dilute latency through batching and asynchronous task scheduling.
    • Interactive scenarios (customer service, writing assistants, tutoring): these need a P95 latency constraint. Canary the rollout, first compare P95 and time to first token with shadow traffic, then decide whether to take full traffic.
    • Agent and tool-calling scenarios (workflow orchestration, coding assistants, automated ops): this is the highest-risk category for upgrading. If the dedicated tool-call success rate regresses, overall capability gains cannot compensate, so keep the old version as a fallback route.
    • High-concurrency, low-unit-price scenarios (content moderation, tagging and extraction): these are cost-sensitive. The 40% increase from 37.6 to 52.5 CNY/1k calls may directly break the budget, so the unit economics must be recalculated.

    A more robust engineering practice is dual-version coexistence plus scenario-based routing: abstract model calls behind a unified interface, push routing rules and fallback strategies through a config center so that different tasks hit different versions, and collect token, latency, and cost data uniformly at the gateway, feeding into a logging service and monitoring alerts for a real-time dashboard. Such evaluation and traffic-scheduling pipelines are usually deployed on cloud container platforms and object storage, with evaluation artifacts (question-set snapshots, detail records, reports) stored and versioned for traceability.

    7. Conclusion

    The gains from a model version upgrade are never free. The data above reveals a general pattern: a 3.4-point accuracy gain and an 11-place leaderboard jump correspond to a 37.6% increase in token consumption, about 40% higher measured cost, roughly 63% longer latency, and a 1.8-point regression in agents and tool calling.

    So the right way to approach a version upgrade is to treat it as a constrained optimization problem: use your own business question set as the sole judge, the four-dimensional metrics as constraints, gate rules as release conditions, and canary routing as a fallback. Do this, and “is the new version better?” is no longer a subjective question, but an engineering problem that can be answered with data.