How to Evaluate Large Language Models: Core Metrics, Version Comparison, and Reading the Results

    Choosing a large language model is rarely as straightforward as picking a database or a message queue, because there is no clean spec sheet to compare. Models differ in where they excel: context length, reasoning, tool calling, code generation, and so on. Vendor announcements usually emphasize that "multiple capabilities improved," while real business workloads can tell a very different story. The triangle of price, speed, and quality also means no model is universally best. Rather than taking marketing claims at face value, build an evaluation process that is measurable, reproducible, and comparable. This guide stays product-agnostic and lays out a general framework you can use to see past the headlines when selecting a model.

    Introduction: Why a Rational Evaluation Method Matters

    Choosing a large language model is rarely as straightforward as picking a database or a message queue, where you can line up a clean spec sheet. Different models put their emphasis on different things: context length, reasoning, tool calling, code generation, and more. Official release notes typically stress that “multiple capabilities have improved,” but in your actual workloads the behavior can differ substantially. The trade-off triangle of price, speed, and quality also means no single model is best at everything. Instead of passively accepting marketing language, build an evaluation method that is quantifiable, reproducible, and comparable. This article is not about any specific product; it abstracts a general-purpose evaluation framework to help you see the real picture during selection.

    1. Four Core Dimensions of Evaluation

    A complete evaluation system should cover at least four orthogonal dimensions, which answer “is it right, is it fast, is it economical, and is it affordable.”

    1.1 Accuracy: the foundation of answer quality

    Accuracy measures how closely model output matches the expected answer, and it is the base of every evaluation. Do not rely on a handful of demo questions. Build a test set that covers your real business scenarios and is large enough to be statistically meaningful. The larger the sample, the stronger the statistical significance: once random fluctuation is averaged out, the conclusions become more trustworthy. In practice, a single evaluation run should involve questions in the tens of thousands, so that small differences between versions or between models carry reference value.

    1.2 Latency: the invisible threshold for user experience

    Latency reflects how fast a single call is. It directly shapes the feel of interactive applications such as AI coding tools and customer service systems. Record the duration of every call and pay attention to the average, but also add percentile metrics such as P95 and P99. An average alone easily hides the stalls caused by long-tail requests.

    1.3 Token consumption: efficiency and hidden cost

    Token consumption answers “how economical is the answer.” For the same task, one model may need a longer reasoning chain or more turns before it produces an answer, and therefore burn more tokens. Token consumption affects not only cost but also speed, since the two are positively correlated: the longer the output, the longer the wait.

    1.4 Total cost: the numbers must be viewed together

    Total cost cannot be judged by unit price alone. It should be calculated by combining token consumption with call volume. A common estimate is “cost per thousand calls,” which equals average tokens per call × 1,000 × unit price. A cheap but verbose model does not necessarily end up cheaper overall.

    2. Putting the Method to Work with a Sample Evaluation Loop

    An evaluation is essentially an automated pipeline. Below is a vendor-neutral skeleton that collects all four metrics in one pass:

    1. import time
    2. def evaluate_model(call_api, test_cases):
    3. """Run a batch evaluation of a model and report the four core metrics."""
    4. total = len(test_cases)
    5. correct = 0
    6. latencies, token_list = [], []
    7. for case in test_cases:
    8. start = time.perf_counter()
    9. resp = call_api(case.prompt) # Call the model API under test
    10. latencies.append(time.perf_counter() - start)
    11. token_list.append(resp.usage.total_tokens)
    12. if resp.matches(case.expected): # Compare against the expected answer
    13. correct += 1
    14. latencies.sort()
    15. p95 = latencies[int(total * 0.95)] if total > 20 else latencies[-1]
    16. return {
    17. "accuracy": correct / total,
    18. "avg_latency": sum(latencies) / total,
    19. "p95_latency": p95,
    20. "avg_tokens": sum(token_list) / total,
    21. }

    The key point: the test set is fixed, the call parameters are fixed, and the evaluation environment is fixed. Only when variables are controlled are results from different runs comparable.

    3. Comparing Versions: Understanding What an “Upgrade” Actually Changed

    The most frequent evaluation scenario is a comparison between an old and a new version. Consider a set of sample numbers to illustrate how to read the results. Suppose that in one evaluation run the new version improves accuracy from 66.6% to 68.7% (about +2.1 percentage points) and climbs from 20th to 10th in the ranking; average latency drops from 98 s to 59 s (about -39.8%); average tokens per call fall from 2,944 to 2,851 (about -3.2%); and cost per thousand calls goes down from CNY 38.8 to CNY 37.6 (about -3.1%).

    Three conclusions follow from these numbers:

    • A small accuracy gain (+2.1 percentage points) combined with a marked jump in ranking suggests the improved overall position comes more from stronger relative competitiveness than from a surge in absolute capability.
    • A latency reduction of nearly 40% is the most noticeable part of this upgrade and delivers the most value to interactive applications.
    • With the unit price unchanged, lower token consumption together with faster responses yields a modest cost improvement, progress through steady steps rather than a deep price cut.

    A reminder: these figures only demonstrate how to read results. A real evaluation must be based on measurements from your own test set.

    4. Domain Breakdown: A Single Overall Score Hides Structural Weaknesses

    One aggregate score cannot reflect the structure of a model’s capabilities, so evaluation must be broken down by domain. Consider a hypothetical set of domain results:

    • Law and public administration: up from 72.7% to 80.3% (+7.6), the most significant improvement;
    • Education: up from 41.2% to 46.9% (+5.7);
    • Healthcare and mental health: up about 3.7 percentage points to 77.6%;
    • Reasoning and mathematics: up about 3.6 percentage points to 65.8%;
    • Language and instruction following: a slight decline from 66.2% to 64.6% (-1.6);
    • Agents and tool calling: down from 72.7% to 69.9% (-2.8).

    These numbers reveal the pattern most easily overlooked in evaluation: a version upgrade is often a trade-off rather than a across-the-board improvement. While the overall score rises, some individual domains decline. So selection should not rely on the aggregate ranking alone; it should be checked against the priorities of your own workload. If your scenarios depend heavily on tool calling or strict instruction following, a decline in those two areas can be a hard blocker even when the overall score leads.

    5. Reading the Gap Between Official Claims and Independent Test Results

    When official materials emphasize that “agent capabilities improved significantly” while an independent test shows a decline in agents and tool calling, that gap itself is informative. Possible explanations: the vendor’s test set is distributed differently from your business traffic, or the optimization targeted other metrics. The way to handle it is to keep retesting with a test set close to your own workload, translating marketing statements into verifiable metrics and letting data rather than copy drive the decision.

    6. Reusable Evaluation Practices

    1. Build the test set around your business: use real, anonymized business samples to form the question bank, avoiding the bias of generic demo questions.
    2. Control variables: fix the prompt templates, sampling parameters such as temperature, and the calling environment so that horizontal comparison holds.
    3. Look at percentiles, not just the mean: add P95/P99 latency and watch for long-tail stalls that hurt interactive experience.
    4. Do the full math: put accuracy, latency, token usage, and unit price in one table, and compute cost per task rather than looking at unit price alone.
    5. Run continuous regression: re-run the fixed evaluation set after every version release to build a comparable historical trend.

    Conclusion

    There is no “standard answer” in model selection, only the answer that best matches your scenario. Instead of being led by a single ranking or a marketing script, build your own evaluation pipeline covering accuracy, latency, token consumption, and cost, break down capability by domain, and let reproducible data do the talking. Once you can state clearly where a model is strong, where it is weak, and how much it costs in your scenario, selection stops being a matter of intuition and becomes a measurable, defensible engineering decision. That is the most reliable footing when new versions keep arriving.