How to Evaluate Large Language Models: Metrics Framework, Domain Breakdown, and Version Comparison

1. Why LLM Evaluation Is Harder Than Before

Evaluating an LLM in everyday question answering is relatively intuitive: you check whether the answer is correct and fast enough. Once you move into coding assistance, tool calling, and multi-turn agent workflows, however, complexity multiplies. These tasks are multi-step, context-dependent, and often require external tools to work together. A single “correct” response does not mean the overall task has been completed. A model may be strong at individual tasks yet fail a long-chain task because of lost context, poor tool selection, or interrupted reasoning.

Sound evaluation must therefore go beyond isolated capabilities and assess task-level capability. It should measure not only how accurate the model’s answers are, but also its efficiency and cost across the real invocation chain. This article lays out a complete evaluation methodology to help developers and enterprise users build an assessment system that is quantifiable, comparable, and reproducible.

2. Building a Quantifiable Evaluation Framework

A credible evaluation report is typically built around four core dimensions: quality, user experience, efficiency, and cost. None can be left out.

2.1 Accuracy (quality)

Accuracy is the most fundamental metric. It measures how closely a model’s output matches the reference answer and must be based on a test set large enough for statistical significance, since conclusions drawn from too few samples lack statistical meaning. With a test set of roughly 15,000 questions, for example, a final accuracy of 68.7% provides enough data to support per-domain analysis.

The scoring method also needs to be defined: exact match, semantic match, or scoring by human raters or another model. The scoring method you choose can significantly affect how comparable the final numbers are.

2.2 Response time (user experience)

The average time per invocation directly shapes how the product feels to use. In coding assistance, a model that takes more than ten extra seconds per turn noticeably disrupts the flow of interaction. Illustratively, if a model’s average response time drops from 98 seconds to 59 seconds (a reduction of about 40%), the improvement in high-frequency interaction is especially obvious.

When measuring latency, filter out network jitter and concurrent request queuing. Otherwise, the data will reflect fluctuations in the pipeline rather than the model’s own inference speed.

2.3 Token consumption (efficiency)

Token consumption measures the amount of text the model processes to complete the same task, including both input context and model output. It indirectly reflects the model’s “expressive efficiency”: models that explain something with fewer tokens often follow instructions better and produce less redundant output. Illustratively, if average consumption per invocation drops from 2,944 to 2,851 (about 3.2%), the model is producing less unnecessary output while maintaining quality.

2.4 Cost per invocation (cost)

Cost is the combined result of the three metrics above. When the output price per unit stays the same, lower token consumption and shorter response times together reduce the cost per invocation. Illustratively, the cost per 1,000 invocations fell from 38.8 to 37.6 yuan (about 3.1%), a direct financial return on efficiency improvements.

Importantly, cost must be assessed together with accuracy. Cutting tokens at the expense of accuracy is not a meaningful optimization.

3. Evaluation Methodology: Test Set Design and Domain Breakdown

The value of evaluation results depends on how well the test set is designed. Follow these principles:

• Large enough: tens of thousands of samples produce more stable conclusions for each segment.
• Reflective of real-world distribution: question types should mirror actual business scenarios, rather than piling on obscure edge cases.
• Splittable by domain: classify the test set by business domain so you can pinpoint the model’s capability boundaries.

Common domain splits include:

• Law and administrative services
• Education and training
• Finance
• Healthcare and mental health
• Reasoning and mathematical computation
• Language and instruction following
• Agents and tool calling

Once the test set is split this way, evaluation is no longer a single overall score but a “capability profile” that clearly shows the strengths and weaknesses in each area.

4. A Framework for Comparing Model Versions

After a model version upgrade, the question most often overlooked is “should we upgrade at all?” Looking only at the overall score tends to lead to one-sided conclusions, so the right approach is a multi-dimensional comparison. For a new-versus-old version switch, you can apply a framework like the following:

• Overall quality: accuracy rises from 66.6% to 68.7% (an increase of 2.1 percentage points), a steady improvement.
• Ranking: in head-to-head comparisons with competitors, moving from 20th place to 10th indicates a marked gain in relative competitiveness. Ranking is not the same as absolute capability, though, so you still need to go back to accuracy itself to judge.
• Efficiency: whether response time and token consumption improve in parallel determines whether the upgrade makes the model “faster and leaner.”
• Cost: the change in unit cost driven by the overall efficiency gains.

The core idea is to break “a higher overall score” down into “which capabilities improved, which declined, and how efficiency and cost changed.” This prevents a single number from masking structural problems.

5. Uneven Gains Across Domains and Why They Happen

It is worth noting that not every domain improves in sync after a model upgrade, and uneven gains and losses are the norm. In one illustrative upgrade:

• Law and administrative services: up from 72.7% to 80.3% (a 7.6-percentage-point gain), the most pronounced improvement.
• Education: up from 41.2% to 46.9% (a 5.7-percentage-point gain).
• Healthcare and mental health: up 3.7 percentage points to 77.6%.
• Reasoning and mathematical computation: up 3.6 percentage points to 65.8%.
• Finance: up 0.8 percentage points to 77.7%.

Meanwhile, some domains declined:

• Language and instruction following: down from 66.2% to 64.6% (a 1.6-percentage-point drop).
• Agents and tool calling: down from 72.7% to 69.9% (a 2.8-percentage-point drop).

This divergence reveals a “seesaw effect” in model optimization: when training resources are tilted toward certain capabilities, others may be relatively squeezed. This is especially relevant when an official announcement claims “enhanced agent capabilities” while evaluation shows a decline in tool calling. Such a gap between marketing claims and measured results calls for an evaluation that is based on data rather than promotional messaging. The value of an evaluation report lies precisely in presenting these discrepancies honestly to readers.

6. Balancing Capability and Efficiency

The judgment that matters most in evaluation is recognizing trade-offs rather than chasing a clean sweep across the board. Here are a few typical ones:

• Accuracy vs. speed: longer chains of thought and more reasoning steps can improve accuracy, but they drive up response time and token consumption.
• Context window vs. performance: expanding the context window from 128K to 200K can, in theory, support more complex, longer-chain tasks. But a larger context is not useful for every task and may bring attention dilution and higher costs. Whether to expand it depends on your target use cases.
• Quality vs. cost: a low-cost model may be good enough for some tasks, while an expensive flagship model is not necessarily the most cost-effective choice for every task type.

As a result, there is no universal answer to “which model is better.” The meaningful conclusion is always “which model is more suitable for a given task type and a given cost budget.”

7. Engineering Recommendations for Developers

If you are selecting a model for your project or validating an upgrade, the following practical path can help:

1. Define your task boundaries: determine whether your core scenario is code completion, multi-turn conversation, or long-chain agents. Each scenario carries different weights.
2. Build a reproducible evaluation set: construct a small regression set from real business samples and pair it with public benchmarks. This creates a two-tier evaluation that combines external comparability with internal realism.
3. Fix the evaluation environment: standardize scoring methods, concurrency levels, and network conditions so results stay comparable across runs.
4. Focus on long-chain success rate: for agent tasks, track the overall completion rate of multi-step tasks rather than step-level accuracy.
5. Track continuously with monitoring: after launch, use logging services and monitoring and alerting capabilities to keep collecting accuracy, response time, and token consumption from real invocations. This closes the loop between offline and online evaluation.

8. Building an Evaluation Platform

When evaluation moves from one-off manual testing to routine regression testing, it needs a supporting platform. From an engineering perspective, it is typically organized like this:

# Evaluation pipeline (pseudocode)
def evaluate(model_endpoint, test_set):
    results = {"correct": 0, "latency_sum": 0, "token_sum": 0}
    for item in test_set:
        resp = call_model(model_endpoint, item.prompt)      # shared model invocation wrapper
        if judge(resp.answer, item.gold):                   # shared scoring method
            results["correct"] += 1
        results["latency_sum"] += resp.latency_ms
        results["token_sum"] += resp.input_tokens + resp.output_tokens
    n = len(test_set)
    return {
        "accuracy": results["correct"] / n,
        "avg_latency_s": results["latency_sum"] / n / 1000,
        "avg_tokens": results["token_sum"] / n,
    }

At the infrastructure level, this kind of pipeline typically relies on several general-purpose cloud capabilities:

• Object storage to archive test sets and evaluation results
• Container platforms to run elastically scalable batch evaluation jobs
• Message queues to decouple tasks and absorb peak loads
• Log services to retain details of every invocation
• Monitoring and alerting to flag metric anomalies in real time

General-purpose capabilities from platforms such as Baidu AI Cloud can serve as building blocks for this kind of evaluation and regression system. They help teams turn evaluation from “ad-hoc scripts” into a sustainable engineering asset.

Conclusion

The real difficulty in evaluating LLMs is not producing a few impressive numbers. It is building an analytical framework that reflects real-world tasks, pinpoints weaknesses, and guides model selection. Combining four dimensions (accuracy, response time, token consumption, and cost) with domain breakdown and version comparison turns “is this model good enough?” into a quantifiable, traceable engineering decision.