Beyond Benchmark Scores: An Engineering Methodology for Evaluating Large Language Models

    This article explains why public leaderboard scores alone cannot determine which large language model fits a business need. It shows how to break requirements into capability dimensions, build an in-house evaluation set with fixed judging criteria and repeated runs, and understand how Mixture of Experts architectures trade inference cost against memory. It also covers deployment factors, including expert-parallel communication, throughput under concurrency, cross-hardware adaptation, and a final selection checklist.

    1. Why Leaderboard Scores Cannot Decide Model Selection

    Public leaderboards compare models under uniform conditions. They are designed to measure general capability, however, not to predict performance on a specific business task. Leaderboard questions are usually curated, and they may differ from real workloads in input distribution, context length, format requirements, and error tolerance.

    Leaderboards are therefore better at answering “What general level does this model roughly reach?” than “Can this model replace a specific step in an existing workflow?” Before choosing a model, break the problem down. Identify the capabilities the business needs, map each capability to a metric, and measure how the model performs on your own data.

    2. Breaking Down Capability Dimensions

    General evaluations typically cover the following categories. Each one has a different focus.

    1. Knowledge Q&A
    Focus on factual accuracy and knowledge coverage. For scenarios that depend on recent information or specialized domain knowledge, also check whether the model fabricates answers, and whether it declines to answer when it is uncertain.

    2. Reasoning and Math
    Focus on the accuracy of multi-step derivations. Distinguish a correct result from an explainable process. In scenarios that require auditing, the soundness of intermediate steps matters just as much as the final answer.

    3. Code and Engineering Tasks
    Focus on whether generated code runs, how well the model understands an existing codebase, and whether it can make minimal changes when editing existing files. Unit test pass rates are more reliable than subjective ratings.

    4. Long Context and Stability
    Focus on the model’s ability to locate key information in long inputs, and on how consistent its results are across repeated runs of the same task. Long-document summarization and log analysis are sensitive to both.

    5. Cost and Latency
    Focus on time to first token (TTFT), total time per request, and actual cost per 1,000 tokens. These metrics directly shape the response experience and the unit economics of a product.

    3. Building an Evaluation Set That Fits Your Business

    A proprietary evaluation set is one of the most valuable assets in model selection. You can build one by following these steps.

    1. Sample real requests: Draw representative inputs from production logs or manual support tickets. Cover both high-frequency scenarios and edge cases.
    2. Define judging criteria: For each sample, define a standard that can be checked automatically, such as whether all required fields are present, whether values match the database, or whether code passes its tests. Samples that cannot be judged automatically should be scored by reviewers who follow fixed rules.
    3. Fix evaluation conditions: Standardize the temperature, system prompt, and maximum output length. This prevents configuration differences from distorting the results.
    4. Run multiple times and use statistics: Run each sample several times, and record the pass rate and variance.

    The following is a general evaluation loop. The generate and judge functions are implemented for each specific business:

    1. from dataclasses import dataclass
    2. from typing import Callable, List
    3. @dataclass
    4. class EvalCase:
    5. prompt: str
    6. expected: str
    7. def run_eval(
    8. cases: List[EvalCase],
    9. generate: Callable[[str], str],
    10. judge: Callable[[str, str], bool],
    11. ) -> float:
    12. """Returns the pass rate, ranging from 0 to 1."""
    13. if not cases:
    14. raise ValueError("Evaluation set is empty")
    15. passed = 0
    16. for case in cases:
    17. output = generate(case.prompt)
    18. if judge(output, case.expected):
    19. passed += 1
    20. return passed / len(cases)

    This structure has one practical benefit. When you switch models, you only replace the generate function. The evaluation set and judging logic stay the same, so the results remain comparable.

    4. What Sparse Expert Architectures Mean

    In recent years, many large models have adopted the Mixture of Experts (MoE) architecture. These models contain a large number of parameters, but each forward pass activates only a subset of the experts.

    Two concepts help clarify this:

    • Total parameters: The full size of the model weights. This determines storage requirements.
    • Active parameters: The parameters that take part in a single inference pass. These are closely tied to the amount of computation and to inference speed.

    A sparse model with a large total parameter count can therefore have a lower per-inference cost than a dense model with the same total parameter count. Storage and memory (VRAM) usage, however, still depend on the total parameter count. Deployment must satisfy both constraints.

    5. Inference Deployment: Throughput, Parallelism, and Hardware Adaptation

    1. Communication Overhead in Expert Parallelism

    In MoE models, experts are usually distributed across multiple accelerator cards. Each token must be routed to the device that hosts its assigned expert, and the results are gathered back after computation. This process involves all-to-all communication, so network bandwidth and latency can become bottlenecks.

    In practice, pay attention to the following:

    • Whether routing is balanced, so that a few experts do not become overloaded;
    • Whether communication can overlap with computation;
    • How to set batch size to balance throughput and latency.

    2. Throughput and Serving Capacity

    Serving throughput is limited by available compute and the concurrency strategy. When compute is tight, a single request may still run at normal speed, but overall queuing time rises. When evaluating serving capacity, measure time to first token and tail latency under concurrency, rather than only the speed of individual requests.

    3. Cross-Hardware Adaptation

    The same model can perform very differently on different accelerator hardware. Cross-platform deployment usually requires:

    • An operator abstraction layer that separates model logic from low-level implementations;
    • A separate performance benchmark for each hardware type;
    • Regression testing for numerical precision differences, to confirm that output quality has not noticeably degraded.

    This decoupling reduces dependence on a single hardware ecosystem. It also makes it easier to adjust the deployment plan when resources change.

    4. Platform Selection

    Choosing a deployment platform requires weighing compute specifications, network topology, billing models, and operations support. You can run small-scale load tests on platforms such as Baidu AI Cloud. Use the measured data instead of estimates before deciding whether to scale up.

    6. Recommendations and Checklist

    Before finalizing a model selection, confirm the following:

    1. Every core capability the business requires has a corresponding evaluation metric;
    2. The evaluation set comes from real requests and covers edge cases;
    3. The cost model includes the inference unit price, memory usage, and operations overhead;
    4. Latency is measured under concurrent load;
    5. An automated regression process is in place for model upgrades and platform migrations;
    6. Output quality has a manual sampling process, especially for high-risk scenarios.

    Leaderboard scores can serve as a first-pass filter, but final decisions should rest on your own data. Making the evaluation process engineered and repeatable helps a team keep its judgments stable as models change quickly.