LLM Evaluation in Practice: Balancing Accuracy, Latency, and Cost

1. Why LLM Evaluation Must Be Systematic

With the rapid iteration of the large language model market, “which model is stronger” has become an unavoidable question for every technical team. But making selection decisions based solely on vendor marketing or a single leaderboard score often leads to problems: some models rank high on benchmarks but underperform in real business scenarios; others respond extremely fast but consume tokens at a rate that pushes bills far beyond budget.

A rigorous evaluation must simultaneously answer three questions:

• Capability: Can the model get things right? This corresponds to accuracy.
• Efficiency: How fast can it produce results? This corresponds to average latency.
• Cost: How much does one run cost? This corresponds to token consumption and call expenses.

This article is based on a hands-on test focused on Chinese text and logical reasoning. The evaluation covered approximately 15,000 test questions, targeting a new version of a frontier large model. It is worth noting that even though the model features native multimodality—capable of simultaneously processing ultra-large-scale video, audio, images, and code repositories—text and logic remain its capability foundation. Stress-testing a multimodal model with demanding Chinese text and logic questions, while only revealing the tip of the iceberg of its cross-modal abilities, precisely approaches the true baseline of its underlying language capabilities.

2. How to Build an Evaluation Dataset

The credibility of evaluation conclusions depends first and foremost on dataset quality. Three points require attention during construction:

1. Dimension coverage must be layered. You cannot simply pile up general Q&A items. Instead, sample by capability dimension and vertical domain: language understanding and instruction following, reasoning and mathematical computation form the general foundation; vertical domains such as education, finance, law and public administration, and healthcare and mental wellness determine whether the model can handle industry scenarios. A scale of approximately 15,000 questions is sufficient to give each sub-dimension statistical significance.
2. Difficulty gradients must be spread out. The question set should include basic factual questions, multi-step reasoning questions, and complex instruction questions. Otherwise, scores will converge toward the range of “can handle all easy questions,” making it impossible to differentiate top-tier models.
3. Leak prevention must be routine. Avoid using publicly available question banks that circulate widely online. Regularly rotate questions and perform deduplication to prevent models from “memorizing answers” during pre-training, which would inflate scores.

3. Definitions and Interpretation of Four Core Metrics

In this hands-on test, the key data for the new version under evaluation was: approximately 15,000 test questions, overall score (accuracy) of 74.8%, average latency of 53s, average token consumption of 3,157, and a cost of approximately ¥250.5 per thousand calls. Each of the four metrics has its own interpretation considerations:

• Accuracy: This is the metric most easily misinterpreted. The judging method (exact match, rule-based judgment, or judge-model scoring) directly affects results. The evaluation report must specify the judging rules; otherwise, cross-evaluation comparisons are meaningless.
• Average latency: End-to-end latency consists of queuing time, first-token latency, and generation speed. A mean of 53s is on the long side for interactive scenarios, but acceptable for offline batch analysis and complex reasoning tasks. It must be assessed in the context of the business scenario.
• Token consumption: A mean of 3,157 reflects the model’s “verbosity level.” The longer the chain of thought, the more tokens are consumed, affecting both latency and cost. This is an often-overlooked implicit metric.
• Call cost: It can be estimated using the formula: cost per thousand calls ≈ average token consumption × unit price (¥ per million tokens) × 1000 ÷ 1,000,000. It converts capability into real money and is the hardest constraint in model selection.

4. Old vs. New Version: Judging Whether an Upgrade Is a “Real Improvement”

Compared with the previous generation, the model under evaluation showed the following data changes: overall score improved from 72.5% to 74.8% (+2.3 percentage points), and its ranking on a comprehensive leaderboard rose from 6th to 2nd. By sub-dimension, language and instruction following improved by 4.9 percentage points to 72.4%, reasoning and mathematics improved by 3.3 percentage points to 85.1%, the education domain rose from 64.8% to 68.6%, and healthcare and mental wellness maintained a high level of 88.7%.

More noteworthy are the efficiency-level changes: average latency dropped from 64s to 53s, while average token consumption only marginally increased from 3,119 to 3,157. This means the speed improvement was not achieved by “thinking less,” but rather through substantive optimization of underlying inference efficiency—compressing latency while token output remained essentially flat. This is key evidence for judging whether an iteration represents a “real improvement.”

When interpreting version comparisons, three principles are recommended: first, check whether the overall score crosses the threshold of statistical significance; second, examine each dimension for capability regressions where “one rises as another falls”; finally, look at long-tail questions rather than just averages, to avoid a few domains being masked by the mean.

5. Three Dimensions for Cross-Model Comparison

For multi-model horizontal comparison, it is recommended to focus on three fixed dimensions:

1. Capability dimension: Look at overall ranking and sub-domain strengths to determine whether the model’s “strong suits” match the core requirements of your business.
2. Efficiency dimension: Compare average latency and throughput to distinguish suitability for interactive versus batch-processing scenarios.
3. Cost dimension: Introduce the concept of “cost-effectiveness,” i.e., cost per unit of accuracy. If Model A has 2 percentage points higher accuracy but costs ¥80 more per thousand calls, you must weigh whether those 2 percentage points are worth the price difference for your business.

6. Evaluation Engineering: An Automated Pipeline

Whether the model under test is accessed through a model service platform such as Baidu AI Cloud or deployed in a private environment, the evaluation pipeline structure is the same: question management → concurrent calls → result judging → metric aggregation → report generation. Sample code:

import asyncio, statistics
async def eval_one(case, client, sem):
    async with sem:
        try:
            resp = await client.chat(
                model="model-under-test",
                messages=[{"role": "user", "content": case["question"]}],
                timeout=120,
            )
            return {
                "score": judge(case["answer"], resp.text),  # rule-based or judge model
                "latency": resp.elapsed,
                "tokens": resp.total_tokens,
            }
        except Exception:
            return {"score": 0, "latency": None, "tokens": 0, "error": True}
async def run(cases, client, concurrency=8):
    sem = asyncio.Semaphore(concurrency)
    results = await asyncio.gather(*(eval_one(c, client, sem) for c in cases))
    accuracy = sum(r["score"] for r in results) / len(results)
    avg_latency = statistics.mean(r["latency"] for r in results if r["latency"])
    avg_tokens = statistics.mean(r["tokens"] for r in results if not r["error"])
    cost_per_1k = avg_tokens / 1_000_000 * UNIT_PRICE * 1000  # UNIT_PRICE: ¥ per million tokens
    return accuracy, avg_latency, avg_tokens, cost_per_1k

Engineering considerations include: using semaphores to control concurrency and avoid triggering rate limits; implementing retries and flagging for timeouts and exceptions to prevent polluting statistics; caching requests for identical questions to reduce re-test costs; setting budget circuit breakers and persisting results after each dimension completes to avoid total loss from mid-run failures.

7. Five Common Misinterpretations of Results

• Treating leaderboard scores as business performance: Leaderboards are general capability references; business performance must be re-tested with your own data.
• Ignoring variance and fluctuation: The same model may produce different results across multiple calls. Confidence intervals should be reported rather than single-run averages.
• Overfitting to the evaluation set: Repeatedly tuning prompts or selecting models using the same question set will gradually erode the evaluation set’s discriminative power.
• Looking only at averages, not the long tail: Errors in certain high-value scenarios (such as law and healthcare) carry costs far exceeding what the mean can represent.
• Underestimating multimodal dimensions: Text evaluation only covers part of a model’s capabilities. Throughput for video, audio, images, and code repositories requires separately designed specialized evaluations; otherwise, you may form an overly conservative judgment of the model’s true boundaries.

8. Conclusion

LLM evaluation is not a one-time action but an ongoing process that accompanies model selection, version iteration, and business evolution. Build a metric system covering capability, efficiency, and cost; use engineering pipelines to ensure reproducible results; then cross-validate through old-vs-new version comparisons and horizontal comparisons. The value of this methodology lies not in “scoring” a particular model, but in ensuring that every technical decision is backed by data.