Evaluating LLM Version Upgrades: Capability Gains vs. Cost and Latency
This article presents a reproducible method for evaluating LLM version upgrades that goes beyond leaderboard rankings. Using an open-source Chinese-language benchmark of roughly 15,000 questions, it compares an old and a new model version across accuracy, average latency, average token consumption, and measured cost per 1,000 calls, and shows how a 3.4-point accuracy gain came with about 62.7% higher latency, 37.6% more tokens, and 39.6% higher cost. It examines which domains improved and which regressed (Agent and tool calling declined), explains why two models with equal accuracy at the same cost tier are separated mainly by speed, and covers the limits of such benchmark conclusions. It then lays out a methodology for running your own evaluation, including cost-corrected accuracy, guardrail metrics, staged rollout with rollback, and sample code for per-case collection, metric aggregation, and tiered model routing.
1. The Problem: A Version Switch Recalculates More Than Scores
Model vendors release new versions periodically, and the developer’s real question is never “is the new version stronger?” but three more specific ones:
- Does the part that got stronger actually land on the business I care about?
- Can the business absorb the extra latency and cost that come with it?
- Did some capability that used to work well quietly regress during the upgrade?
Answering these requires far more than “the ranking went up.” A rank is a relative quantity, disconnected from cost and latency. A more meaningful approach is to measure, under a fixed question set and fixed call parameters, the accuracy, average latency, average token consumption, and measured cost per 1,000 calls, then compare vertically (old vs. new version) and horizontally (within the same cost tier).
The sample used here comes from an open-source Chinese LLM capability benchmark of about 15,000 questions, covering reasoning, education, medical, financial, language instruction, Agent, and tool calling, and testing only Chinese-language scenarios. The models below are referred to as the “old version” and “new version.”
2. The Overall Picture: A Reproducible Set of Per-Call Metrics
| Metric | Old version | New version | Change |
|---|---|---|---|
| Overall accuracy | 68.1% | 71.5% | +3.4 points, rank up from 16th to 5th |
| Avg. latency (per call) | 59 s | 96 s | +62.7% |
| Avg. tokens (per call) | 2851 | 3922 | +37.6% |
| Cost per 1,000 calls | CNY 37.6 | CNY 52.5 | +39.6% |
The output price stayed at CNY 14.0 per million tokens in both versions. So the entire increase in call cost comes from the higher average token usage per call.
These figures yield three conclusions:
- The accuracy gain is real but limited. A 3.4-point absolute gain moved the overall rank up 11 places, which shows how tightly packed the top of the leaderboard is and how a small improvement can shift rank substantially. So you cannot infer the magnitude of a capability gain from rank alone.
- The cost increase closely matches the token increase. 37.6% vs. 39.6%, less than 2 points apart. When the unit output price is unchanged, the bill is essentially determined by average token usage. The per-output unit price is only a “list price”; the measured cost per 1,000 calls is the real cost.
- What really deserves attention is latency. The relative increase in latency is more than 25 points higher than that of tokens. From this we can derive the average generation speed:
| Version | Avg. latency | Avg. tokens | Avg. generation speed (tokens/s) |
|---|---|---|---|
| Old version | 59 s | 2851 | ≈48.3 |
| New version | 96 s | 3922 | ≈40.9 |
The new version is slower per token, at roughly 85% of the old version’s speed. This suggests the latency growth is not driven by token count alone. A plausible explanation is that the new version uses a more complex generation or reasoning mechanism, though the exact cause should follow the vendor’s release notes and actual load testing.
Core conclusion: this upgrade trades clear increases in latency, tokens, and cost for an improvement in overall accuracy.
3. Where the Gains Land: Which Domains Benefit, Which Are Sacrificed
Breaking the evaluation down by domain, the changes are not consistent across dimensions:
| Domain | Old version | New version | Change |
|---|---|---|---|
| Finance | 77.7% | 84.2% | +6.5 |
| Medical & mental health | 77.6% | 83.6% | +6.0 |
| Reasoning & math | 66.7% | 72.7% | +6.0 |
| Education | 46.9% | 52.0% | +5.1 |
| Language & instruction following | 64.6% | 69.3% | +4.7 |
| Agent & tool calling | 69.9% | 68.1% | -1.8 |
(Sorted by change, descending.)
1. Reasoning and math: the most direct gain. A 6.0-point improvement indicates a clear strengthening of the new version’s multi-step derivation and calculation ability. This is consistent with the rise in token usage and latency: such tasks require longer reasoning chains, and a more complex mechanism naturally produces more output tokens and longer waits.
2. Finance, medical, and mental health: the largest gains, but two possibilities must be separated. The two improved by 6.5 and 6.0 points respectively, and their absolute levels are also the highest (both above 83%). Whether the gains come from indirect transfer of reasoning ability or from simultaneous improvements in domain knowledge and instruction understanding cannot be distinguished from the total score alone. It is better to re-run a domain-specific regression on the subset before drawing conclusions.
3. Education: a sizable gain, but the weak spot remains. It rose from 46.9% to 52.0%, yet it is still the only domain below 60%, approaching a “roughly one in two calls answered incorrectly” level. Such scenarios are better served by a two-stage “generate + verify” approach than by trusting a single output directly.
4. Language and instruction following: the real gain may be underestimated. The 4.7-point increase is the smallest of all dimensions. Since the total number of questions is fixed, the change per domain can be treated as a contribution to the same overall shift. The larger a domain’s share, the larger its contribution, roughly 4.7 × p (where p is the domain’s share of questions). Its increase is close to the overall level (3.4) yet it still ranks last, which implies this domain has a relatively small share of questions. For applications centered on dialogue, summarization, copywriting, and structured output, this increase may already be close to their entire benefit.
5. Agent and tool calling: the only regression, and the key signal of this evaluation. This is the only domain that declined (-1.8 points), which the original attributes to “a trade-off in the course of overall performance improvement.” A more rigorous reading is that this is a clear capability substitution: the new version has allocated part of its capability budget to reasoning and knowledge tasks. The impact depends on the share of each question type and can be estimated as:
Contribution to overall score ≈ (questions in domain / total questions) × domain's changeLet the Agent domain's share be p and the overall gain be 3.4 points, then:Contribution of all other domains = 3.4 + 1.8 × pE.g., when p = 10%, the Agent regression offsets about 0.18 points,wiping out more than 5% of the overall gain.
For teams building agents, RAG question answering, or workflow automation, the selection accuracy of tool calls directly determines whether a task completes in one pass. A few points of higher error rate get amplified layer by layer in flows that involve “multiple calls, accumulated step by step.” Therefore, the domains where the old version was strongest should be mandatory tests before the new version goes live.
4. Cross-Comparison at the Same Cost Tier: When Scores Tie, Speed Decides
Within the same benchmark, the new version’s measured cost is CNY 52.5 per 1,000 calls, placing it in the same tier as a lightweight model from a certain international vendor (CNY 53.5). The two share the same overall accuracy of 71.5%, and the new version ranks slightly higher, but its average latency is 96 s versus the competitor’s 72 s.
The specific numbers shift with versions and question-set updates; what this comparison really reveals is the evaluation logic:
- When accuracy is equal, the selection decision shifts to constraints. Once the gap narrows to CNY 1 per 1,000 calls and the scores are identical, success often comes down to latency, stability, context length, and ecosystem compatibility. A 96 s latency means a single call approaches two minutes, and a synchronous interface can no longer support real-time interaction.
- Cost and performance are not monotonically related. At nearly equal cost (52.5 vs. 53.5 CNY) and identical scores, the two still rank 5th and 6th. Looking only at rank and not cost leads to the opposite conclusion. Selection must use a multi-dimensional view of “accuracy + latency + cost,” not a single-score leaderboard.
- At equal scores, the latency gap amplifies with call volume. If a business’s average chain requires 3 serial calls, that is 96 s × 3 ≈ 288 s, already beyond most users’ acceptable wait, unless it switches to asynchronous tasks or streaming responses.
- Cross-comparison conclusions apply only under the stated conditions. This benchmark emphasizes Chinese scenarios; switch language or vertical domain and the ordering could reverse entirely.
5. The Limits of the Conclusion: What the Data Supports and What It Doesn’t
Before making decisions based on this, understand the boundaries of these figures:
- Scenario boundary: the conclusions cover only general Chinese-language capability and cannot be extrapolated directly to English, multilingual, or highly specialized vertical domains.
- Score structure: 68.1% → 71.5% represents only the average correct rate. If the business truly cares about the hardest 5% of samples, an overall gain may mask stalled or even regressed tail performance; the by-difficulty and by-type breakdowns must be examined.
- Limits of averages: 96 s is an average. Production should assess experience with P95/P99 latency, timeout rate, and error rate; the mean hides long-tail problems.
- Single-point comparison: it is usually hard for the new and old versions to be retested at the same moment with identical data and configuration, so results may include evaluation-environment differences. Key conclusions should be re-checked on your own data.
- Question-set risk: public question sets carry the risk of data contamination, meaning the test questions or near-variants have already entered the model’s training data. Once contamination exists, scores are systematically inflated, so evaluation on your own scenarios is irreplaceable.
6. Methodology: Running a Version-Upgrade Evaluation on Your Own Scenarios
6.1 Build your own evaluation set
- Sample to match your business distribution: sample by the proportion and difficulty distribution of real online requests, ideally no fewer than 300–500 items, and label each with its domain and difficulty.
- Keep input/output baselines: fix prompts and generation parameters (such as temperature), and fully record the input, output, latency, and token usage of every call to ensure comparability across versions and reproducibility of results.
- Keep the judging method consistent: use exact match for objective questions and fixed scoring rules or model-based scoring for open-ended ones. The old and new versions must use the same judging method; otherwise score differences may come from the judging step itself.
6.2 Design multi-dimensional metrics and compare with cost-corrected accuracy
Beyond accuracy and latency, also track token consumption, cost per 1,000 calls, and output-format validity (e.g., whether JSON parses correctly). For cross-version comparison, introduce a “cost-corrected accuracy”:
Cost-corrected accuracy = accuracy ÷ cost per 1,000 calls
Reference calculation:
| Version | Accuracy | Cost per 1,000 calls | Cost-corrected accuracy (% / CNY) |
|---|---|---|---|
| Old version | 68.1% | CNY 37.6 | ≈1.81 |
| New version | 71.5% | CNY 52.5 | ≈1.36 |
The new version’s accuracy is about 5.0% higher, but after cost correction it is actually about 25% lower. Put another way, each 1-point gain in accuracy costs about 11.6% more on average. This does not mean the new version is worse; it is a reminder that whether it is worth it depends on whether the business can turn the extra accuracy into real returns, for example by lowering the manual-review rate or reducing retries, where such gains can cover the added cost.
6.3 Set guardrail metrics
Set a non-regressing floor for each version’s weakest domain. In the reference sample, the new version’s Agent and tool-calling ability regressed, so one can set “Agent subset accuracy not below the old version” as a guardrail condition and pause the ramp-up if it is not met.
6.4 Gradual rollout and rollback
- Offline evaluation: compare candidate versions one by one on the evaluation set.
- Shadow traffic: mirror a small portion of online requests to both the new and old versions, recording only without returning, to observe performance under the real distribution without affecting users.
- Ramp up by percentage: scale from 5% → 20% → 50% gradually while monitoring accuracy proxy metrics (user feedback rate, manual-review correction rate, task completion rate), P95 latency, error rate, and cost.
- One-click rollback: wrap a unified entry point at the calling layer so switching models by configuration rolls back instantly, avoiding tight coupling between the version upgrade and business code.
6.5 Front-load cost engineering
Part of the rise in latency and cost can be absorbed through engineering means:
- Task routing: send simple requests to a lightweight model and reserve the heavy-reasoning version for hard problems.
- Context trimming: compress retrieved snippets and merge duplicate information.
- Cache reuse: reuse results for repeated or similar requests to lower average token usage.
- Output-length control: set a reasonable maximum output token count for tasks that don’t need long output.
- Streaming responses: return the first segment early to improve the waiting experience, and set timeouts and retry caps at the calling layer.
A common practice on cloud platforms such as Baidu AI Cloud is to wrap model calls as an independent service, write each call’s input, output, latency, and token usage to a logging service, connect success rate, latency, and cost to monitoring and alerting, and archive evaluation data and results in object storage. This keeps different versions on the same pipeline and collection methodology and enables fast localization and rollback when metrics go abnormal. The evaluation set and its results should also be brought into configuration management and versioned, so every upgrade decision is traceable.
7. Sample Code: Collection, Accounting, and Routing
The code below only illustrates the method and includes no specific vendor’s interface or parameters.
7.1 Per-case collection: record latency, tokens, and cost
import time# Prices are in CNY per million tokens; configure per the actual billing rules of the chosen model servicePRICE_INPUT_PER_M = ... # input pricePRICE_OUTPUT_PER_M = ... # output pricedef run_case(client, model_id: str, messages: list, gold) -> dict:"""Make one call for a single case; record latency, token usage, and cost."""t0 = time.perf_counter()resp = client.chat.completions.create(model=model_id,messages=messages,temperature=0, # fix generation parameters for reproducibility and comparability across versionsmax_tokens=1024, # cap output length so a few long outputs don't inflate average tokens)elapsed = time.perf_counter() - t0u = resp.usagecost = (u.prompt_tokens / 1e6 * PRICE_INPUT_PER_M+ u.completion_tokens / 1e6 * PRICE_OUTPUT_PER_M)answer = resp.choices[0].message.contentreturn {"elapsed": elapsed,"total_tokens": u.total_tokens,"cost": cost,"correct": judge(answer, gold), # the judge function must be identical across old and new versions}
7.2 Metric aggregation: produce a horizontally comparable evaluation report
def percentile(values, q):ordered = sorted(values)if not ordered:return 0.0k = (len(ordered) - 1) * qlo, hi = int(k), min(int(k) + 1, len(ordered) - 1)return ordered[lo] + (ordered[hi] - ordered[lo]) * (k - lo)def summarize(rows, benchmark_n: int = 1000):"""Summarize the core metrics of one evaluation; rows is the list of results returned by run_case."""n = len(rows)acc = sum(r["correct"] for r in rows) / ncost_per_1k = sum(r["cost"] for r in rows) / n * benchmark_nlatencies = [r["elapsed"] for r in rows]return {"accuracy": round(acc * 100, 1),"p50_latency": round(percentile(latencies, 0.50), 2),"p95_latency": round(percentile(latencies, 0.95), 2),"avg_tokens": round(sum(r["total_tokens"] for r in rows) / n, 1),"cost_per_1k_calls": round(cost_per_1k, 2),# cost-corrected accuracy: correct rate gained per unit of cost (% / CNY)"acc_per_yuan": round(acc * 100 / cost_per_1k, 3),}
7.3 Tiered routing: trade off capability against cost
def choose_model(task) -> str:"""Choose a model by task type and difficulty so not every request hits the expensive version."""# Verified Agent / tool-calling scenarios: prioritize no capability regressionif task.category in {"agent_tool", "function_call"}:return task.preferred_verified_model# Weak spots like education: use the reasoning-heavy version and add verification at the app layerif task.category == "education":return MODELS["reasoning"]# Multi-step reasoning and math: route to the reasoning-heavy versionif task.difficulty >= 3:return MODELS["reasoning"]# All other routine requests: use the light, low-cost versionreturn MODELS["light"]
8. Conclusion
In the reference sample, the new version traded roughly 37.6% more tokens, 62.7% more latency, and 39.6% more cost for a 3.4-point gain in accuracy. It improved markedly in reasoning, finance, medical, and education, but regressed in Agent and tool calling; in a same-cost cross-comparison, it also exposed a weakness in latency. Such a result is neither a “complete upgrade” nor a “regression.”
For developers, a more reliable approach is to treat each version upgrade as a quantifiable engineering decision:
- Build a scenario-specific evaluation set that records accuracy, latency (including P95), token consumption, and cost per 1,000 calls, and compare with cost-corrected accuracy;
- Set guardrail metrics for the weakest domains to prevent strong capabilities from being quietly sacrificed in an upgrade;
- Absorb some of the added cost through task routing, caching, and streaming responses;
- Validate effects with staged rollout and shadow traffic, and always keep a one-click rollback capability.
