How to Evaluate Large Language Models: Core Metrics and Model Selection Methods

    Choosing a large language model for code generation, agent orchestration, or content creation takes more than trusting a single "significant capability improvement" claim. This article lays out a structured evaluation framework built on four core dimensions—accuracy, latency, token consumption, and cost—and shows how to read the data behind a version upgrade without mistaking a ranking jump for doubled capability. It also covers per-domain analysis, the trade-offs inherent in capability improvements, how to build a reusable automated evaluation pipeline, and practical guidance for selecting the model that best balances accuracy, speed, cost, and your business scenario.

    Introduction: Why Evaluation Matters More Than a Single Benchmark Score

    In real engineering projects, selecting a large language model for code generation, agent orchestration, or text creation is rarely a matter of trusting a single claim like “significant capability improvements” from an official release. What actually determines the user experience is a set of metrics that can be quantified, reproduced, and tracked continuously. Teams often fall into two traps during selection: blindly chasing leaderboard rankings while ignoring how well a model matches the actual business scenario, or focusing only on accuracy while underestimating how latency and token consumption amplify overall cost and degrade human-AI interaction.

    A structured, reproducible evaluation methodology is therefore far more valuable than relying on a one-off benchmark run. This article builds on widely used evaluation frameworks to break down the key dimensions of model evaluation and shows how to understand the real gains and potential trade-offs behind a version upgrade.

    1. Four Core Dimensions of an Evaluation System

    A mature model evaluation system needs to cover at least four main lines: capability, speed, efficiency, and cost. These four are interrelated, yet they often constrain one another.

    Accuracy: The First Yardstick of Model Capability

    Accuracy reflects the proportion of correct outputs a model produces on a given task set. It is central to capability assessment, but also the metric most easily misread. A responsible evaluation must specify three things:

    • Test set size and composition: An evaluation might cover around 15,000 questions across domains such as law, education, finance, healthcare, mathematical reasoning, instruction following, and tool calling. Sample size determines the statistical confidence of the conclusions, while the distribution of questions determines their scope of applicability.
    • Objectivity of the judging criteria: Is it human annotation, rule-based matching, or cross-evaluation by a stronger model? Each judging approach introduces systematic bias.
    • Drill-down into sub-domains: An overall accuracy figure can mask structural weaknesses; the real value lies in the per-domain data.

    Response Latency: The Key to Interaction Experience

    The average time per call directly affects the user’s subjective experience. For highly interactive scenarios such as real-time dialogue and code completion, latency is often a more sensitive pain point than accuracy. In a typical version upgrade, average time per call dropped from about 98 seconds to 59 seconds, an improvement of nearly 40%—an optimization that can improve user experience more noticeably than a two or three percentage-point gain in accuracy. During evaluation, monitor both the average and the tail (such as P95 and P99 latency), because tail latency is the real culprit behind timeouts and collapsing experiences.

    Token Consumption: The Hidden Cost Amplifier

    The number of tokens consumed per call is the bridge between “capability” and “cost.” Excessive consumption often means the model tends to produce redundant output or generates unnecessary intermediate steps during reasoning. Reducing average consumption from about 2,944 to 2,851 (a decrease of about 3.2%) may look minor, but at the scale of millions of calls, the cumulative resource savings are considerable. When assessing token efficiency, analyze it alongside accuracy: the higher the accuracy obtained per token, the better the model’s cost-performance.

    Call Cost: Where Cost-Effectiveness Lands

    Cost is the combined result of the first three dimensions. With the output unit price held constant, lower token consumption and higher speed together reduce the overall cost. For example, the cost per thousand calls dropped from 38.8 to 37.6, a decrease of about 3.1%. The important point to stress is this: the unit price is just a number on paper. You should combine it with the actual call volume of your business and convert it into a monthly or even annual total cost of ownership (TCO) before making a rational budget decision.

    2. How to Read the Evaluation Data of a Version Upgrade

    Taking a typical model iteration as an example, evaluation data often takes this comparative structure:

    1. Dimension Old version New version Change
    2. Overall accuracy 66.6% 68.7% +2.1 pp
    3. Overall ranking #20 #10 ↑10 places
    4. Avg. latency/call 98s 59s -39.8%
    5. Avg. tokens/call 2944 2851 -3.2%
    6. Cost/1,000 calls 38.8 37.6 -3.1%

    The key to reading this table is: do not simply equate “a jump in ranking” with “doubled capability.” Moving from 20th to 10th place reflects relative position among comparable models, while the 2.1 percentage-point accuracy gain is the change in absolute capability. Combining the two leads to the objective conclusion of “steady overall progress.” At the same time, the near-40% speed improvement and the simultaneous decline in tokens and cost show that this upgrade achieved positive optimization on both the “capability” and “efficiency” fronts, making it a high-quality iteration.

    3. Per-Domain Evaluation: Overlooked Capability Divergence

    A rising overall score does not mean every domain is getting better. A rigorous evaluation always performs a domain drill-down. Common divergences include:

    • Law and administrative affairs: A substantial jump from 72.7% to 80.3%, a gain of 7.6 percentage points, the most prominent improvement.
    • Education: An increase from 41.2% to 46.9% (+5.7%), clear progress, but the absolute score remains low, indicating considerable room for further optimization.
    • Healthcare and mental health: Up 3.7 percentage points to 77.6%.
    • Reasoning and mathematical computation: Up 3.6 percentage points to 65.8%.
    • Finance: Up 0.8 percentage points to 77.7%.
    • Language and instruction following: A slight decline from 66.2% to 64.6% (-1.6%).
    • Agents and tool calling: A decline from 72.7% to 69.9% (-2.8%).

    The core message of this breakdown table is that capability assessment must match the business scenario. If your core need is legal document processing, the gains from this upgrade are significant. But if your business heavily depends on strict instruction following or tool calling, be wary of the real impact of the metric declines.

    4. The “Trade-off” Phenomenon in Capability Improvements

    A thought-provoking phenomenon is that while official messaging often emphasizes “enhanced agent capability,” the “tool calling” domain actually declined slightly in the evaluation. This is not a contradiction; rather, it reveals the rule of capability trade-offs that large models face under limited resources.

    There is a mutually competing relationship among a model’s capacity, context window (for example, expanding from 128K to 200K tokens), reasoning depth, and tool-calling strategy. Optimizing long-context handling may compromise precise responses to short instructions; strengthening the reasoning chain may introduce extra token overhead. This tells us:

    1. There is no single “best at everything” model—only the choice that “best fits the scenario”;
    2. The value of an evaluation report lies precisely in revealing these imperfect trade-offs rather than painting a picture of total victory;
    3. In engineering practice, you should assign weights to key metrics for the specific business rather than chasing the highest total score.

    5. Building a Reusable Automated Evaluation Pipeline

    To make evaluation continuously valuable, it must be engineered and automated. A reusable evaluation pipeline typically has four stages:

    1. # Evaluation pipeline (pseudocode)
    2. def run_eval_pipeline(model_endpoint, dataset):
    3. results = []
    4. for case in dataset:
    5. t0 = time.now()
    6. resp = model_endpoint.call(case.prompt)
    7. latency = time.now() - t0
    8. results.append({
    9. "domain": case.domain,
    10. "correct": judge(case.expected, resp.output),
    11. "latency": latency,
    12. "tokens": count_tokens(resp.usage),
    13. })
    14. return aggregate(results) # Aggregate accuracy, latency, tokens, cost
    15. def aggregate(results):
    16. return {
    17. "accuracy": mean([r["correct"] for r in results]),
    18. "avg_latency": mean([r["latency"] for r in results]),
    19. "p95_latency": percentile([r["latency"] for r in results], 95),
    20. "avg_tokens": mean([r["tokens"] for r in results]),
    21. }

    Several best practices are worth following when putting this pipeline into practice:

    • Fix the test set and judging logic: Comparisons between versions are only meaningful when the inputs are consistent.
    • Record both average and tail metrics: P95 and P99 latency often reflect the real experience better than the average.
    • Integrate evaluation into continuous integration: Whenever a model version or prompt changes, automatically run a regression pass to prevent capability regression.
    • Score by business weights: Configure differentiated weights based on how sensitive the scenario is to accuracy, latency, cost, and tokens.

    6. Selection Recommendations and Engineering Practice Summary

    Back to the original question—with a limited budget, how do you choose a model that truly fits you? Based on the evaluation framework above, here are the conclusions:

    1. Calculate the total cost first, then the unit price: Convert token consumption, call volume, and latency into the real monthly cost, rather than fixating on a single phrase like “cheap.”
    2. Set weights by scenario: Code completion prioritizes latency, professional Q&A prioritizes accuracy, and batch processing prioritizes token efficiency. There is no universal optimum.
    3. Value the “efficiency dividend” of version iterations: When an upgrade brings improvements in accuracy, speed, and cost at the same time, the cost-performance of migration is usually high.
    4. Keep the ability to roll back: Encapsulate the model call interface through an abstraction layer so that business code is decoupled from any specific model, making it easy to switch flexibly between different model services.

    For teams that need to compare, canary-release, and perform failover switching across multiple models, mature cloud model services (such as the model services offered by Baidu AI Cloud, along with general capabilities like monitoring and alerting and logging services) can be used to build a unified evaluation and scheduling foundation, reducing engineering complexity and letting you focus your energy on the business value itself.

    In one sentence: Evaluating large language models is not about choosing the “strongest” one, but about choosing the one that achieves the best balance among accuracy, speed, cost, and your business scenario. Master the four-dimensional metric system and the per-domain analysis method, and you will have the confidence to make rational selection decisions.