LLM Evaluation Methodology: From Metric Design to Version Comparison

Why Evaluation Is the Foundation of LLM Decisions

Large language models now iterate on a monthly—or even weekly—cadence. Faced with press releases claiming “comprehensive leadership” or “dramatic improvements,” developers really only care about two questions: How does this model perform in my business scenarios? And once a new version ships, is upgrading worth it?

Gut-feel testing—asking a few casual questions and screenshotting the answers—cannot answer either. Informal spot checks have three fundamental flaws: they are not reproducible (questions and decoding parameters are not fixed, so results cannot be rerun and verified), not comparable (different testers reach different conclusions that cannot be aligned horizontally or vertically), and not cumulative (results are discarded after testing, so no long-term capability curve can be built). A rigorous evaluation system must satisfy three conditions:

• Reproducible: Fix the evaluation set, decoding parameters (temperature, max output length, etc.), and scoring criteria, so that anyone rerunning the suite at any time gets approximately the same results;
• Comparable: Compare across models horizontally and across versions of the same model vertically, with conclusions anchored to a shared baseline;
• Cumulative: Persist evaluation results as data assets, tracking capability curves across releases rather than producing one-off snapshots.

Four Core Metrics: What to Track Beyond Accuracy

A complete evaluation should collect four classes of metrics at once. Together they answer a single question: is the model good to use, and affordable to use?

1. Overall score (accuracy): the capability baseline. Build on a question bank on the order of 10,000 items; with too few samples, swings of a few percentage points may be pure noise. Score objective questions programmatically and subjective questions with a judge model supplemented by manual spot checks, and state the scoring criteria in the report—otherwise the scores are not comparable.

2. Average response time: the direct driver of user experience. Record end-to-end latency on every call and, alongside the mean, watch the percentiles (P50/P95/P99). A good mean does not rule out a bad tail, and production experience is precisely what tails destroy. Also note that reasoning-mode models (those with an explicit chain-of-thought pattern) are inherently slower and must be measured separately from standard generation, or the comparison is unfair.

3. Average token consumption: a proxy for cost and reasoning depth. Token consumption directly determines cost and indirectly reflects how deeply the model “thinks.” The combination to watch for is surging consumption with flat accuracy—usually a sign of redundant reasoning chains or “overthinking”: paying more without gaining quality.

4. Cost per call: translating technical metrics into financial language. Aggregate under a “cost per 1,000 calls” convention (input tokens × input unit price + output tokens × output unit price, multiplied by call volume). This makes reporting to business stakeholders and management intuitive, and it underpins capacity and budget planning.

The key is to read the metrics together: they must be viewed in the same table. A high accuracy in isolation may be bought with longer thinking time; a speed advantage in isolation may sacrifice reasoning depth; a low cost in isolation may come from truncated outputs.

Designing the Evaluation Set: Stress-Testing to Expose the Capability Baseline

The quality of the questions in an evaluation set determines the ceiling of the evaluation. In practice, building a demanding test set around Chinese text and a logical core has proven effective time and again, for three reasons:

• Chinese is a naturally demanding exam. Real-world Chinese instructions often drop subjects, vary word order freely, and are dense with idioms and polysemous words, while mixing in format constraints and negated conditions. Whether a model can execute precisely under such fine-grained constraints directly tests instruction understanding and following—pressure that English benchmarks reflect less faithfully for Chinese-language business scenarios.
• A logical core generalizes across tasks. Whether the upper-layer application is writing code, building data reports, or reviewing contracts, the underlying support is the same reasoning chain. Scores on the logic and math dimensions are among the strongest predictors of complex-task performance.
• “Reduced-dimension” testing has special value for multimodal models. Most leading models now carry multimodal capabilities spanning images, audio, and video, yet in most enterprise workloads text and code still dominate. Testing a multimodal model purely on Chinese text and logic strips away the modality packaging and exposes the true baseline of its underlying reasoning and language ability—that foundation sets the ceiling for cross-modal capabilities. Multimodal throughput, however, deserves its own dedicated evaluation; the two cannot substitute for each other.

On the engineering side, three points are non-negotiable: scale—only at the ten-thousand-question level do you get statistical significance; stratification—sample in layers by task type and industry domain (language and instruction following, reasoning and math, code, education, finance, legal, healthcare, etc.) so that every sub-dimension has sufficient sample density; contamination control—regularly refresh and deduplicate the question bank to prevent questions that have leaked into training data from inflating scores.

Drilling Down from the Overall Score: Building a Capability Profile

The overall score is like the composite number on a physical exam report; what actually guides model selection is the breakdown. A competent evaluation report should include at least:

• Language and instruction following: multi-step instructions, format constraints, negated conditions, detail retrieval over long contexts—mapping to the most tangible everyday experience;
• Reasoning and math: mathematical solving, logical reasoning, numerical computation—the core predictor of complex-task capability;
• Vertical domains: education (problem explanations, knowledge Q&A), finance (data interpretation, compliance-sensitive phrasing), legal and public administration (statute comprehension, official document writing), healthcare and mental health (accuracy and rigor of professional Q&A), and so on. Each industry should re-weight the overall score according to its own business priorities.

The value of drilling down is identifying models that are equal in score but different in substance: two models with the same total score may owe it to language strength in one case and reasoning strength in the other, and their behavior on a concrete business task can differ dramatically. The same sobriety applies to external leaderboard rankings—question distribution and scoring criteria differ across leaderboards, so before celebrating a ranking jump, first examine what changed in the test set composition.

Version Comparison: Reading the Twin Signals of Speed and Depth

After a new version ships, the most valuable move is to rerun the old and new versions on the same evaluation set. Take a set of illustrative numbers (used only to demonstrate how to interpret them):

Metric	Old Version	New Version	Change
Overall score (accuracy)	72.5%	74.8%	+2.3 pp
Average latency	64s	53s	≈ −17%
Average token consumption	3,119	3,157	≈ +1%

Three signals deserve individual reading:

1. Is a 2.3-percentage-point accuracy gain significant? At the ten-thousand-question scale, that swing means hundreds of questions flipping from wrong to right or vice versa—far beyond random noise—so it can be judged a real capability gain. If the question count were only in the thousands, compute a confidence interval before concluding, so you don’t mistake noise for progress.

2. What does latency dropping against the trend mean? This is the signal worth the most attention. With accuracy up substantially, shorter latency indicates the gain did not come from “thinking longer” but from substantive optimization of the underlying architecture and inference efficiency (common industry techniques include more efficient attention implementations, speculative decoding, and tighter control of reasoning chains). “Faster and more accurate” together usually signals high upgrade value.

3. Token consumption is roughly flat—how does cost look? Consumption nearly unchanged while latency drops markedly means per-unit-time throughput efficiency has improved. Combine this with output pricing to compute the actual change in cost per 1,000 calls and judge whether the upgrade adds cost pressure—in this example it does not, which is the ideal case of “performance up, cost flat.”

Engineering the Evaluation Pipeline

Evaluation should not be a one-off exercise but a pipeline that runs continuously:

Evaluation set versioning → Batch call execution → Automated scoring → Metric aggregation and regression alerts

A minimal viable aggregation logic, for illustration:

@dataclass
class CallRecord:
    task_id: str
    correct: bool        # scoring result
    latency_s: float     # end-to-end latency
    prompt_tokens: int   # input token count
    output_tokens: int   # output token count
def aggregate(records: list[CallRecord],
              price_in: float, price_out: float) -> dict:
    """price_in / price_out: price per million tokens"""
    n = len(records)
    cost = sum(r.prompt_tokens / 1e6 * price_in
               + r.output_tokens / 1e6 * price_out
               for r in records)
    return {
        "accuracy": sum(r.correct for r in records) / n,
        "avg_latency_s": sum(r.latency_s for r in records) / n,
        "avg_tokens": sum(r.prompt_tokens + r.output_tokens
                          for r in records) / n,
        "cost_per_1k_calls": cost / n * 1000,
    }

Additional engineering advice: keep decoding parameters under version control, since parameter drift contaminates comparison conclusions; record percentiles, not just means, because tail latency hurts production experience more; establish a regression mechanism—rerun whenever the model version, prompts, or evaluation set changes, forming a time series of capability; and connect the data sources—if your enterprise consumes cloud-hosted LLM services (such as Baidu AI Cloud), you can build directly on the platform’s standard logging, monitoring and alerting, and billing capabilities to automatically capture per-call latency, token consumption, and cost, making evaluation data collection far more efficient.

Turning Evaluation Results into Decisions

The end point of evaluation is decision-making, and three typical scenarios call for different readings:

• Model selection: re-weight the sub-dimensions using your own business priorities rather than copying generic overall rankings;
• Upgrades: focus on the interplay of the accuracy–latency–cost triangle; versions that are faster, more accurate, and no more expensive deserve the highest priority;
• Budgeting: convert “cost per 1,000 calls” into monthly cost based on actual call volume, and assess value for money and capacity headroom.

Closing Thoughts

A model’s capability level will not freeze at any given version, and evaluation should not stop at any single report. Only by hardening the four core metrics, the layered evaluation set, and the version regression mechanism into a pipeline can developers face every “major release” with data instead of sentiment—both a mark of technical rigor and a basic discipline for controlling enterprise cost and risk.