LLM Evaluation in Practice: Core Metrics and Engineering Guidelines

1. Why Systematic Model Evaluation is Necessary

The iteration pace of large models has accelerated to a monthly cycle. Relying on a handful of ad-hoc Q&A trials can neither accurately gauge a model’s true capability nor support technical selection and version upgrade decisions. A seemingly routine version update often simultaneously impacts four variables: accuracy, response speed, token consumption, and API cost. These variables can constrain each other—improvements in capability frequently accompany longer reasoning chains, increased token usage, and higher latency.

Therefore, a qualified model evaluation must fundamentally answer three questions:

• Capability: How well does the model perform on questions matching the target business distribution?
• Efficiency: How much time and how many tokens are required to perform the same task correctly?
• Cost: What is the expense per single call and per unit of quality, translated into actual currency?

Consider a typical LLM version iteration (the data below is illustrative and used solely to demonstrate the analysis methodology, representing no specific product): On approximately 15,000 test questions, the new version increased its overall score from 72.5% to 74.8%. Meanwhile, the average single-call latency decreased from 64 seconds to 53 seconds, and the average token consumption slightly increased from 3,119 to 3,157. Behind these seemingly unremarkable numbers lie crucial details worth exploring in evaluation engineering.

2. Evaluation Framework Design: Test Sets and Dimensional Layering

2.1 Four Principles for Test Set Construction

• Balance Scale and Coverage: A test set of a few thousand questions only serves as a coarse filter. A scale of tens of thousands is required to give statistical significance to conclusions like a “2 percentage point improvement.” Coverage should span language understanding, instruction following, reasoning and math, and code generation, as well as vertical domains such as education, finance, law and administration, and healthcare and mental health.
• Difficulty Stratification: Easy questions verify baseline capabilities, while highly difficult questions differentiate top-tier models. If the test set is generally too easy, all models will cluster in the 90%+ score range, resulting in near-zero differentiation.
• Domain Stratification: The total score must be broken down into independently analyzable dimension scores. Otherwise, a flat “74.8%” cannot explain where the model excels and where it falls short.
• Anti-Contamination: Questions must be continuously updated and deduplicated to avoid mixing in public corpora that the model might have already “memorized.” Otherwise, you are testing memory rather than reasoning capability.

2.2 Why Use High-Difficulty Text and Logic Questions to Probe the Foundation

Current mainstream native multimodal models generally support the simultaneous ingestion of video, audio, images, and codebases, making cross-modal capabilities appear more impressive. However, language understanding and logical reasoning are the bedrock of all multimodal capabilities. If a model cannot stand firm on rigorous long texts, complex instructions, and multi-step logic questions, its multimodal capabilities are merely built on sand.

Therefore, applying “dimensionality reduction testing” to multimodal models using high-difficulty text and logic questions—even if it only covers a corner of their capability map—can effectively expose the true quality of their underlying reasoning. This is a highly cost-effective evaluation strategy: low question cost, objective grading, and transferable conclusions.

3. Definition and Measurement of Four Core Metrics

3.1 Accuracy: Don’t Just Look at the Total Score

The first trap in accuracy is the grading method: exact match, rule-based grading, and model-as-a-judge. Scores from these three approaches are not directly comparable. The second trap is distribution: the same 72.5% total score could mean “medium across all dimensions” or “extremely strong reasoning but extremely weak instruction following.” Therefore, the total score must be interpreted alongside dimension scores. When necessary, paired significance tests should be conducted to ensure the improvement is not just measurement noise.

3.2 Latency: Look Beyond the Average at the Distribution

An “average latency of 53 seconds” is necessary but far from sufficient. Engineering teams should focus more on:

• End-to-End Latency: From initiating the request to receiving the final token.
• Time to First Token (TTFT): A key metric affecting interactive user experience.
• Percentiles: P50/P95/P99. An average of 53 seconds might mean “half the requests take 30 seconds, while a few drag on for 3 minutes.” The latter causes significantly more business damage than the average suggests.

3.3 Token Consumption: The Intermediate Variable Between Capability and Cost

Token consumption bridges capability and cost. Reasoning-oriented models often generate longer thinking chains in exchange for higher accuracy, causing token usage to expand. Therefore, looking solely at the shift from “3,119 to 3,157” is meaningless. The right question is: For every additional token spent, how much accuracy improvement is gained? If a 2.3 percentage point improvement only costs about a 1% increase in tokens, it is a worthwhile trade-off.

3.4 API Cost: Calculating the “Cost Per Unit of Quality”

Cost calculation requires a clear pricing model (e.g., currency per million tokens, with input and output priced separately). Using the illustrative data: if the average single call consumes 3,157 tokens and the actual cost for 1,000 calls is 250.5 (in local currency), we can extrapolate a comprehensive unit price of approximately 79 per million tokens, averaging about 0.25 per call.

A more valuable derived metric is the Cost Per Unit of Quality: 250.5 ÷ 74.8 ≈ 3.35 per 1,000 calls per accuracy percentage point. Using this metric for horizontal comparison during model selection is much closer to business reality than simply comparing “who is cheaper” or “who is more accurate.”

Metric	Old Version	New Version	Change
Total Score (Accuracy)	72.5%	74.8%	+2.3pp
Average Latency	64s	53s	-11s
Average Tokens	3119	3157	+38
Cost per 1,000 Calls	—	250.5	—

4. Version Iteration Comparison: Signals Behind the Data

4.1 Interpreting Upgrade Quality by Dimension

Dimension	Old Version	New Version	Change
Language & Instruction Following	67.5%	72.4%	+4.9pp
Reasoning & Math	81.8%	85.1%	+3.3pp
Education	64.8%	68.6%	+3.8pp
Healthcare & Mental Health	High	88.7%	Remained Flat

This set of data reveals three typical iteration strategies:

• Strengthening Weaknesses: Language and instruction following saw the largest increase of 4.9 percentage points. This indicates the previous version had obvious shortcomings in complex instructions, and the new version prioritized fixing them. Every 1-point improvement in such foundational dimensions directly enhances the actual business experience.
• Breaking Through Ceilings: Reasoning and math reached 85.1%. In the high-score zone around 85%, the difficulty of gaining 1 percentage point is much higher than in lower-score zones. A 3.3 percentage point increase here is highly valuable.
• Maintaining Strengths: Healthcare and mental health maintained at 88.7%. In difficult vertical domains, “not regressing” is a proof of capability in itself.

4.2 The Efficiency Paradox: Higher Scores, Faster Speeds

Conventionally, an accuracy improvement implies a longer reasoning chain, which should increase latency. However, in the illustrative data, latency decreased from 64 seconds to 53 seconds, while token consumption remained almost flat. This is a highly valuable signal: it indicates the optimization focus was not on “thinking more,” but on underlying inference efficiency—such as attention computation optimization, inference framework scheduling improvements, and generation strategy acceleration. Capability and efficiency evolving simultaneously is the ideal form of version upgrade. It also highlights that evaluation must lock onto both quality and efficiency metrics, otherwise such critical progress will be missed.

5. Horizontal Comparison: The Right Way to Use Leaderboards

When comparing candidate models horizontally, it is recommended to fix three dimensions: Capability (accuracy), Efficiency (latency and tokens), and Cost (unit expenditure). Simultaneously, be aware of three traps:

• Incomparable Standards: Scores derived from different test sets, different graders, and different prompt templates cannot be directly summed and compared.
• Leaderboards ≠ Business: Public leaderboards reflect general capability levels, which may severely misalign with your business distribution. The safest approach is to build a custom evaluation set based on real business data and measure all candidate models with the same ruler.
• Averages Mask Shortcomings: Using a radar chart to view dimensional balance is often a better predictor of actual experience than a total score. There are no all-around champions, only varying degrees of match with your business needs.

6. Engineering Implementation of the Evaluation Pipeline

Evaluation should not be a one-off script, but a sustainable pipeline. It does not require any specialized infrastructure and can be built using general cloud capabilities:

• Object Storage: Version control for test sets, prompt templates, and historical results.
• Message Queue: Asynchronous distribution of evaluation tasks for peak shaving and valley filling.
• Container Platform: On-demand elastic scaling of evaluation workers to run tens of thousands of tasks in parallel.
• Logging Service: Retain the raw input and output of every request to support post-hoc auditing and error attribution.
• Monitoring & Alerting: Set alerts for latency spikes, abnormal error rates, and unexpected score fluctuations.

Using Baidu AI Cloud as an example, these general product categories can cover the entire workflow. The evaluation code is decoupled from the business code, making it reusable and portable. Below is a minimal evaluation skeleton:

import asyncio
import statistics
async def run_single(client, case: dict) -> dict:
    """Single test: call model -> grade -> collect metrics"""
    resp = await client.chat(
        model=case["model"],
        messages=case["messages"],
        timeout=120,
    )
    return {
        "score": judge(case["reference"], resp.text),   # Rule-based grading or model-as-a-judge
        "latency": resp.elapsed,                        # End-to-end latency
        "tokens": resp.prompt_tokens + resp.completion_tokens,
        "cost": calc_cost(resp.token_usage, price_table),
    }
async def evaluate(cases, concurrency=8):
    sem = asyncio.Semaphore(concurrency)
    async def guarded(c):
        async with sem:
            for _ in range(3):                          # Retry on failure
                try:
                    return await run_single(client, c)
                except TimeoutError:
                    continue
        return {"score": 0, "latency": 0, "tokens": 0, "cost": 0}
    results = await asyncio.gather(*(guarded(c) for c in cases))
    return {
        "accuracy":    statistics.mean(r["score"] for r in results),
        "p50_latency": statistics.median(r["latency"] for r in results),
        "avg_tokens":  statistics.mean(r["tokens"] for r in results),
        "cost_per_1k": sum(r["cost"] for r in results) / len(results) * 1000,
    }

7. Four Key Strategies for Cost Optimization

The end goal of evaluation is optimization. When data indicates high costs, you can sequentially try:

1. Prompt Simplification and Structuring: Reduce redundant context to directly compress input tokens.
2. Cache Reuse: Apply prefix caching or semantic caching to fixed prefixes and high-frequency similar questions.
3. Tiered Routing: Route simple requests to lightweight models and only escalate difficult requests to large models. This can reduce overall costs by an order of magnitude.
4. Output Length Control: Constrain the response structure and length within the prompt to prevent the reasoning chain from expanding indefinitely.

8. Five Common Pitfalls

• Looking Only at the Average Score: Ignoring distributions and percentiles will cause you to miss long-tail disasters.
• Ignoring Grader Reliability: If the judge model makes mistakes, all conclusions become distorted. Manual sampling and verification should be performed.
• Test Set Contamination: Outdated questions will inflate scores. The test set needs regular refreshing.
• Drawing Conclusions from a Single Run: Generation involves randomness. Critical conclusions should be sampled multiple times, with variance reported.
• Treating Leaderboards as Absolute Truth: Self-tested data aligned with your business is always more trustworthy than public leaderboards.

9. Conclusion

Model evaluation is not a one-time “exam,” but a continuous engineering practice. Once a systematic evaluation pipeline is established, every version upgrade and selection decision can yield quantified answers under the same ruler: how much capability improved, whether speed is faster or slower, and how much each point of accuracy costs. In an era of rapid model iteration, this reproducible, rational judgment capability is itself a team’s most vital technical asset.