LLM Version Upgrades: Evaluation Metrics and Model Selection Decisions

    When a new LLM version launches, the total leaderboard score rarely tells the whole story. Drawing on a public Chinese-language evaluation of a next-generation model against its predecessor (about 15,000 questions), this article shows how a 3.4-point accuracy gain came with roughly 63% higher latency, 37.6% more tokens per call, and about 40% more cost per thousand calls—and how one domain, Agent and tool calling, actually regressed. It presents a four-dimension evaluation framework spanning quality, efficiency, cost, and stability, explains why cost can rise even when the output unit price stays flat, and compares candidates within the same cost tier. It closes with a selection framework built on per-scenario strategies, task routing, gray release, and observability, so teams can upgrade where it pays off and stay reversible where it does not.

    1. Introduction: An Upgrade Is Not a “Total Score Game”

    As foundation models iterate quickly, news of a “next-generation release” arrives almost constantly. For developers and enterprise engineering teams, the hard part is rarely “is the new version stronger.” It is “is the part that got stronger worth the extra latency and cost.” A single version jump usually brings three changes at once: expanded capability boundaries, longer reasoning, and a higher per-call cost. If you decide to upgrade based only on a leaderboard’s total score, you may discover after launch that long questions respond more slowly, the end-of-month bill jumps noticeably, and previously stable tool-calling paths start misbehaving.

    This article uses one public Chinese-language evaluation of a next-generation model against its predecessor as a sample (about 15,000 test questions). It compares the two versions on accuracy, response time, token consumption, and cost, then discusses two engineering questions: how to design a metric set that covers “quality—efficiency—cost,” and how to use it to decide which traffic can be upgraded and which should wait.

    2. Breaking Down the Metrics: What to Measure in a Version Upgrade

    2.1 A Four-Dimension Metric Framework

    A single total score only reflects “average capability” and cannot explain the side effects of an upgrade. A more complete approach splits metrics into four dimensions, each tied to a specific business risk:

    • Quality dimension: overall accuracy, per-domain accuracy. The total answers “is it better overall,” while per-domain data answers “which tasks benefit and which may regress.”
    • Efficiency dimension: average latency, and where possible P95 and P99 latency. Averages hide the long tail, and the long tail is exactly what users feel in interactive products.
    • Cost dimension: output unit price, average token consumption, cost per thousand calls. All three matter. An unchanged unit price does not mean unchanged cost, because total cost is the product of unit price and actual consumption.
    • Stability dimension: call success rate, structured-output compliance rate, performance under long context. These directly determine whether an upgrade can be safely wired into existing engineering pipelines.

    2.2 Core Formula: Why Cost Decouples from Output Unit Price

    To understand the data later, first establish the general billing logic:

    1. Cost per call = (input tokens × input unit price + output tokens × output unit price) / 1,000,000
    2. Cost per 1,000 calls ≈ Cost per call × 1000 × (1 + retry rate)

    If a model trades a longer reasoning process for accuracy, input and output tokens inflate together, and cost rises accordingly. Dividing “cost per thousand calls by total tokens per thousand calls” lets you back out the blended unit price:

    • Previous generation: 37.6 ÷ (2851 × 1000 ÷ 1,000,000) ≈ 13.2 yuan/million tokens
    • New generation: 52.5 ÷ (3922 × 1000 ÷ 1,000,000) ≈ 13.4 yuan/million tokens

    The two are essentially identical. This shows the new version’s roughly 40% cost increase comes almost entirely from higher token consumption, not from a higher call unit price. This is good news: cost can be partially recovered through prompt trimming, output-length constraints, and task routing, rather than passively accepting a rising bill. The retry rate in the cost formula is also easy to overlook. Once average latency grows from 59 to 96 seconds, requests near the timeout threshold are more likely to fail and retry, so the actual bill may exceed the static estimate.

    3. Old vs. New: An Improvement with a Price

    3.1 Overall Performance

    Metric Previous generation New generation Change
    Test questions ~15,000 ~15,000 —
    Overall accuracy 68.1% 71.5% +3.4 pp
    Overall rank 16th 5th up 11 places
    Avg. latency (per call) 59s 96s ~+63%
    Avg. tokens (per call) 2851 3922 +37.6%
    Output unit price 14.0 yuan/million tokens 14.0 yuan/million tokens unchanged
    Cost per 1,000 calls 37.6 yuan 52.5 yuan ~+40%

    (The output unit price is the output-side price under the evaluation’s pricing basis. Source: ReLE Chinese LLM benchmark, https://github.com/jeinlee1991/chinese-llm-benchmark.)

    In short, this is an upgrade that trades time, tokens, and cost for accuracy. Accuracy rises 3.4 percentage points, a relative increase of about 5.0%, at the price of roughly 63% more latency and about 40% more cost per thousand calls.

    3.2 Per-Domain Changes: Gains Are Concentrated, and the Single Regression Is Worth Watching

    Domain Previous New Change
    Finance 77.7% 84.2% +6.5 pp
    Healthcare & mental health 77.6% 83.6% +6.0 pp
    Reasoning & math 66.7% 72.7% +6.0 pp
    Education 46.9% 52.0% +5.1 pp
    Language & instruction following 64.6% 69.3% +4.7 pp
    Agent & tool calling 69.9% 68.1% -1.8 pp
    • Broad gains, with reasoning tasks benefiting most. Improvements concentrate on knowledge-intensive, multi-step reasoning tasks such as finance, healthcare, math, and education, rising 5–6.5 percentage points; even the weakest domain, education, climbs from 46.9% to 52.0%, though it remains a clear weak spot for the new version. “Language & instruction following” improves 4.7 percentage points, indicating the new version follows prompts somewhat better. The overall pattern is “the stronger the reasoning, the greater the gain.”
    • Agent & tool calling is the only regression. It drops 1.8 percentage points, indicating the overall improvement comes with a trade-off. Tool calling requires the model to output an exact format and pick the right tool within a limited number of turns; a longer reasoning process may introduce format drift or off-track flows. This is a reasonable inference based on public data; the specific cause still needs validation against your own business cases.
    • This regression deserves special attention. Tool calling is often a core step in agent-style applications, where a missing field or type error breaks downstream parsing and the impact is amplified. For such systems, per-domain data carries more decision value than the total score.

    3.3 Latency and Cost: Two Different Kinds of Pressure

    Latency pressure is a user-experience problem. Average latency grows from 59 to 96 seconds, about 63% slower. By average latency, roughly 61% of calls used to return within 60 seconds before the upgrade; afterward that drops to about 39%, and the rest hit the timeout threshold. Interactive products that keep the old timeout settings may see widespread timeouts. P95 and P99 latency are usually longer than the mean, so timeout thresholds, retry strategies, and even the front-end interaction model (such as streaming output) all need to be re-evaluated.

    Cost pressure is a budget problem. The output unit price stays at 14.0 yuan/million tokens, but each call now consumes about 1,071 more tokens on average, and the cost per thousand calls rises from 37.6 to 52.5 yuan. Taking 100,000 calls per day as an example, the monthly cost rises by about 45,000 yuan. Whether this increase is affordable depends on whether the new version’s accuracy gains in core scenarios translate into real business value—for example, less manual review and a lower error rate.

    4. Cross Comparison: Choosing the Best Option Within the Same Cost Tier

    An evaluation should not stop at the vertical version comparison. Only by placing the new version alongside other candidates in the same cost tier can you judge whether an upgrade is “worth it.”

    • Leading overall rank within the tier. The new generation’s cost is 52.5 yuan per thousand calls, placing it in the 50-yuan tier; a very close candidate, Model A, sits at 53.5 yuan, in the same tier. Both score 71.5% accuracy, but the new version’s rank (5th) is slightly higher than Model A’s (6th). With nearly equal pricing and identical scores, the rank difference suggests the new version may do better on other composite metrics (such as balance across domains) and deserves a closer look.
    • Latency becomes the key differentiator. Model A’s average latency is about 72 seconds, roughly 25% faster than the new version’s 96 seconds. For real-time interactive applications, this gap directly affects timeout rates and user experience. If the business is latency-sensitive and accuracy requirements are already met, a faster model may be the safer choice.

    This yields a practical principle: selecting within a tier means comparing multi-dimensional vectors, not a single scalar. When total scores are close, latency, token consumption, and fit with your own business often determine the final payoff more than differences after the decimal point.

    5. A Selection Decision Framework: From “Upgrade Everything” to “Allocate by Need”

    5.1 Scenario-Based Strategy

    Use case Key metric Upgrade advice Main risk & mitigation
    Real-time Q&A, smart assistants P95/P99 latency Keep the previous or a lightweight version by default; route only complex questions to the new version Higher timeout rate; set tiered timeouts and streaming responses
    Offline analysis, report generation Accuracy Upgrade first Higher cost; absorb latency with batch processing
    Domain-specific (finance, healthcare, education, etc.) Per-domain accuracy Prioritize finance, healthcare, and reasoning tasks; validate education tasks with your own data General benchmarks do not fully represent business data; build a vertical evaluation set
    Agent & tool calling Tool-call success rate, structured-output compliance Regression-test thoroughly before upgrading; strengthen format validation and retries Format drift; validate with JSON Schema and retry on failure
    Simple classification, information extraction Cost per call Keep a lightweight model Unnecessary high-cost calls

    5.2 Task Routing: Reserve Expensive Reasoning for Requests That Need It

    1. import re
    2. # Rules can be replaced with confidence scores or task-classification results based on business statistics
    3. ROUTING_RULES = [
    4. (r"证明|推导|为什么|计算|分析原因", "flagship"), # Multi-step reasoning → new generation
    5. (r"提取|分类|打标|格式转换|翻译", "lightweight"), # Simple tasks → lightweight model
    6. ]
    7. def route_model(prompt: str, default: str = "legacy") -> str:
    8. for pattern, model in ROUTING_RULES:
    9. if re.search(pattern, prompt):
    10. return model
    11. return default
    12. if __name__ == "__main__":
    13. cases = [
    14. "证明:任意三个连续整数之积必能被6整除",
    15. "从下面这段合同中提取甲乙方名称和签署日期",
    16. "请结合上下文总结这段文字的中心思想",
    17. ]
    18. for q in cases:
    19. print(route_model(q), "|", q)

    Suppose 70% of online requests are simple tasks; after routing, the new generation handles only 30% of calls. Even though the new generation consumes 37.6% more tokens per call, the overall token increase is only about 11.3%, letting you preserve most of the reasoning gain while sharply containing the cost increase.

    5.3 Gray Release and Observability: An Upgrade Must Be Reversible

    • Dual-run comparison: Use shadow traffic to let the old and new versions process the same requests in parallel, comparing output quality, latency, and token consumption case by case, then scale up gradually by percentage after evaluating the differences.
    • Observability: Use common cloud capabilities such as logging services and monitoring/alerting to continuously collect success rate, percentile latency, token usage, and cost by scenario, and set budget alerts.
    • Fallback strategy: When the timeout rate, failure rate, or format-validation failure rate exceeds a threshold, automatically fall back to the old version or a lightweight model, so the new version’s long-tail problems do not reach users directly.

    6. Conclusion

    In this case, the new generation makes clear progress in accuracy, per-domain capability, and overall rank, but the small regression in Agent and tool-calling ability, along with about 63% more latency and about 40% higher cost, reminds us that upgrading requires trade-offs. More important than “whether to upgrade” is “where to upgrade”: keep a lightweight model for latency-sensitive scenarios; prioritize upgrades for reasoning-intensive scenarios such as finance and healthcare, and re-check them against your own data; strengthen format validation and regression testing for tool-calling paths.

    Once you build an evaluation system covering quality, efficiency, cost, and stability, and add task routing, gray release, and continuous monitoring, every model version iteration can turn into a measurable, reversible, cost-controlled productivity gain.