Evaluating Large Language Models with 40,000+ Test Cases: Cost, Metrics, and Decision-Making

    A reliable LLM evaluation should produce decision-ready evidence—not just a leaderboard score. This article presents a practical framework for evaluating task coverage, quality, robustness, safety, latency, and total cost across more than 40,000 test cases, with sampling, confidence analysis, cost control, and continuous regression testing for production adoption.

    1. From “Benchmark Scores” to Decision Evidence

    Large language model evaluations often fall into two traps. The first is drawing conclusions from a few impressive demos. The second is focusing exclusively on an aggregate leaderboard score. The former uses too little data, while the latter can hide major differences in task composition.

    A useful evaluation should answer four questions:

    • Which tasks can each candidate handle reliably?
    • Where are its failure boundaries?
    • How much does it cost to reach business-acceptable quality?
    • Does quality regress after a version upgrade?

    In the original evaluation, more than 40,000 test cases were run within a budget in the tens of thousands of yuan, and the conclusions were made public. This scale shows that model evaluation is moving from individual experimentation toward engineering-grade evidence production.

    However, whether the results can be reused depends less on the leaderboard headline than on the methodology: Are test cases properly segmented? Are scores explainable? Are costs included? Are model versions traceable?

    When the difference between aggregate scores is less than 0.2 points but invocation costs differ by nearly an order of magnitude, model selection should focus on statistical significance and total cost of ownership rather than mechanically choosing the top-ranked model.

    2. Define the Evaluation Boundary First

    Before evaluation begins, “model capability” must be broken down into an executable set of tasks. At a minimum, the following dimensions should be covered:

    • Task types: General question answering, information extraction, structured output, long-document understanding, code generation and repair, multi-turn dialogue, and multimodal understanding and generation.
    • Input distribution: Short text, long context, mixed Chinese and English, tables, images, noisy inputs, and adversarial inputs.
    • Difficulty levels: Basic examples, high-frequency business examples, long-tail edge cases, and common failure cases.
    • Output constraints: Free-form text, fixed formats, field completeness, traceable citations, and refusal boundaries.
    • Business constraints: Latency, concurrency, cost limits, security and compliance, and the proportion of cases requiring human fallback.

    More than 40,000 test cases should not be distributed evenly across all categories. A better approach is variance-based allocation: sample stable, high-frequency tasks less heavily, and allocate more samples to long-tail tasks with greater variability.

    High-risk scenarios should have a golden set. Version upgrades should use a regression set, while attacks and privilege-escalation prompts should be covered by a challenge set. Data contamination must also be addressed. Time-based splits and private holdout sets can help prevent candidates from overfitting to public examples.

    A practical test-case schema can be represented as follows:

    1. case_id: unique identifier
    2. task_type: task category
    3. difficulty: basic/business/long-tail/adversarial
    4. language: Chinese/English/mixed
    5. input: prompt, context, and attachments
    6. expected: reference answer or scoring rule
    7. weight: business weight
    8. tags: safety, formatting, factuality, and other labels

    3. Metrics: The Aggregate Score Is Only the Starting Point

    An aggregate score is useful for initial screening, but it should not determine the final decision. A robust evaluation should include at least five categories of metrics:

    1. Quality metrics: Task success rate, exact-match rate, structured-field consistency, factual consistency, format compliance, and code execution rate.
    2. Robustness metrics: Pass rates and degradation under paraphrasing, misspellings, noise, long context, and privilege-escalation prompts.
    3. Safety and compliance metrics: Sensitive-content blocking, resistance to prompt injection, data-leakage risk, and the appropriateness of refusal boundaries.
    4. Performance metrics: P50, P95, and P99 latency, throughput, timeout rate, and failed-retry rate.
    5. Cost metrics: Cost per invocation, cost per 1,000 tasks, retry overhead, cache-hit benefits, and human-review costs.

    Cost should not be calculated using input and output prices alone. A more complete estimate is:

    1. Per-call cost ≈ input token count × input price + output token count × output price
    2. Total cost = invocation count × per-call cost
    3. + retry cost
    4. + human fallback cost
    5. + evaluation and operations cost

    A low-cost model may generate unnecessarily long responses, produce unstable formats, or require frequent retries. Its total cost may therefore be higher than that of a more expensive model.

    Teams should plot a quality-cost Pareto frontier. The horizontal axis represents total cost after meeting the required SLA, while the vertical axis represents business-weighted quality. The best choice is usually neither the highest-scoring and most expensive model nor the cheapest option. It is often the balance point that meets the quality thresholds for critical tasks at an acceptable total cost.

    4. Cover More Samples with a Limited Budget

    Covering more than 40,000 cases with a budget in the tens of thousands of yuan is not primarily about reducing unit prices. The key is to increase the information value of each test case. The following strategies can help:

    • Two-stage evaluation: Use low-cost rules or a smaller model for preliminary screening, then perform full evaluation on shortlisted candidates. Screening rules must be fixed and their biases documented.
    • Stratified sampling: Divide cases by business volume, risk level, and difficulty to reduce redundant examples.
    • Caching and reuse: Keep prompt templates, judges, and random seeds fixed, and cache reusable intermediate results.
    • Early stopping: If a candidate falls significantly below a threshold in critical segments, stop its evaluation early to save budget.
    • Human spot checks: Blindly review a proportion of automatically scored results to calibrate evaluator bias.

    An abstract evaluation workflow looks like this:

    1. function run_evaluation(candidates, cases, weights):
    2. records = []
    3. for case in cases:
    4. for candidate in candidates:
    5. output = candidate.generate(case.prompt, temperature=0)
    6. score = judge(case, output)
    7. records.append({case_id, candidate_id, score, cost, latency})
    8. report = aggregate(records, weights)
    9. confidence = bootstrap_ci(records)
    10. return report, confidence

    In an engineering implementation, test cases, model outputs, and scoring records should be stored in object storage. Tasks can be orchestrated through a container platform, while logging and monitoring services collect latency, failure, and cost metrics.

    General-purpose object storage, container platforms, and logging services provided by Baidu AI Cloud can support this type of pipeline. The essential requirements are reproducible environments, least-privilege access, and complete records of model versions, parameters, prompt templates, and judge versions.

    5. How to Use Leaderboards and Cross-Model Comparisons Correctly

    The primary value of a leaderboard is not the ranking itself, but the evidence behind it. A credible leaderboard should disclose at least:

    • Test-case distribution
    • Scoring rules
    • Version date
    • Runtime parameters
    • Confidence intervals
    • Failure examples

    Without this information, even a high aggregate score is difficult to reproduce.

    Cross-model comparisons should use fixed task conditions and show the actual outputs from multiple candidates for the same input. For example, a multimodal generation test could specify a scene containing a neon sign with mixed Chinese and English text, a rain-soaked street, glass reflections, and a glow effect.

    Evaluation criteria could include:

    • Text readability and spelling accuracy
    • Stability of the main composition and layout
    • Style consistency and richness of detail
    • Physical plausibility, including the relationships among reflections, water marks, and glow
    • Consistency and controllability across multiple generations

    Automatic metrics are suitable for large-scale initial screening, but text rendering, composition, and style consistency often require blind human evaluation. Reviewers should use tiered scoring rubrics and cross-review by multiple evaluators. Failure-reason labels should also be retained.

    When aggregate scores differ by only a fraction of a point, the report must include confidence intervals. If prices differ by nearly 20 times, it should present the quality floor, stability, and total cost alongside the average score rather than showing the average alone.

    6. Interpreting Results and Applying Them in the Enterprise

    When interpreting evaluation results, start with three steps:

    1. Review weaknesses by task segment.
    2. Recalculate business weights based on cost.
    3. Examine failure cases according to their risk.

    For high-accuracy, low-tolerance scenarios, prioritize solutions that meet the required quality floor and provide stable version behavior. For large-scale, low-risk tasks with human fallback available, choose a balanced point on the Pareto frontier and implement gradual rollout and fallback strategies.

    Enterprise adoption should also establish a continuous evaluation loop:

    • Use a fixed golden set for regression testing.
    • Use a dynamic challenge set to probe failure boundaries.
    • Monitor production drift through sampled online traffic.
    • Record model versions, parameters, and score changes for every upgrade.
    • Preserve switchable architectures for critical business functions to avoid single-vendor or single-model lock-in.

    Public evaluation results can reduce research and comparison costs, but they cannot replace validation against proprietary business data. A genuinely actionable evaluation should tell readers what to choose, why to choose it, and under what conditions not to choose it—not merely leave them with a rank on a leaderboard.