LLM Version Selection: From Public Leaderboards to Custom Evaluations

    Public leaderboards provide a useful starting point, but they rarely answer which model will perform best for a specific business workload. This article explains how to move from benchmark rankings to task-specific evaluation: defining task, engineering, cost, and compliance constraints; building a custom evaluation set with automated checks; weighting results by business traffic; tracking tail latency and error types; comparing quality, cost, and latency together; assessing multimodal capabilities only when needed; and using canary releases, rollback rules, and decision records to turn model selection into a repeatable engineering process.

    LLM Version Selection: From Public Leaderboards to Custom Evaluations

    1. Why Leaderboard Conclusions Are Hard to Apply Directly

    Public leaderboards use fixed question sets and uniform evaluation settings. The distribution of those questions is usually different from that of real business traffic. Customer service Q&A, code completion, contract information extraction, and long-document summarization each have very different requirements for reasoning depth, format stability, and context length. A composite score therefore cannot reliably represent performance on a specific workload.

    Improvements between model versions are also uneven. A new version may make clear progress on mathematical reasoning while showing no change in instruction following or conversational Chinese. In addition, prompt templates, sampling temperature, maximum output length, and other parameters all affect results. Under different settings, the ranking of the same evaluation can be completely reversed.

    For this reason, public leaderboards should be treated as an initial screening signal, not as the final basis for selection. The real question is: under my task, my data, and my constraints, which model delivers the best quality, the lowest cost, and the most controllable risk?

    2. Define Three Types of Constraints First

    Before selection, it helps to break requirements into three categories—task, engineering, and cost/compliance—and write each one down clearly.

    Task constraints

    • Input type: plain text, or multimodal content such as images, audio, and tables.
    • Output requirements: free-form text, or output that must follow a fixed JSON structure or field format.
    • Quality baseline: which errors are unacceptable, such as incorrect financial figures, code syntax errors, or leakage of sensitive information.

    Engineering constraints

    • Context length: average and peak input length per request.
    • Concurrency and throughput: requests per second during peak periods.
    • Latency: acceptable time to first token and total generation time for users.

    Cost and compliance constraints

    • Cost per request: calculated separately for input and output tokens.
    • Data location and retention: whether data may be sent to external services and whether private deployment is required.
    • Availability requirements: whether the system can depend on a single vendor and whether multiple backup paths are needed.

    Once these three sets of constraints are defined, the range of candidate models narrows considerably, and subsequent evaluation becomes more targeted.

    3. Build a Custom Evaluation Set

    A custom evaluation set is the most valuable asset in model selection. The following principles can guide its construction.

    First, use data from real business operations. Sample from online logs and cover both high-frequency and long-tail requests. High-frequency requests determine user experience, while long-tail requests determine risk.

    Second, prioritize questions that can be judged automatically. Whether extracted fields are correct, whether classification labels are consistent, and whether code passes unit tests can all be written as deterministic checks. For subjective questions, such as tone of response or appropriateness of expression, use multiple raters and calculate rating consistency.

    Third, set sample sizes by category. For each task category, prepare at least several hundred samples so that proportion-based metrics have reference value. With too few samples, a few errors can cause results to fluctuate sharply.

    The following schematic code shows an evaluation framework for calculating pass rates by task category:

    1. class EvalCase:
    2. prompt # input content
    3. expected # reference answer or pass condition
    4. category # task category
    5. def run_eval(model_call, cases, checker):
    6. # model_call: function that calls the model under test
    7. # checker: function that determines whether output meets expectations
    8. results = {}
    9. for case in cases:
    10. output = model_call(case.prompt)
    11. passed = checker(output, case.expected)
    12. results.setdefault(case.category, []).append(passed)
    13. return {
    14. category: sum(flags) / len(flags)
    15. for category, flags in results.items()
    16. }

    In real projects, model_call should uniformly wrap temperature, maximum output length, and system prompts so that different models are compared under the same conditions. The evaluation scripts themselves should also be placed under version control for reproducibility.

    4. Avoid Being Misled by a Single Metric

    A composite score can hide differences between categories. A model may have a higher total score simply because it performs well in a category that accounts for a small share of business traffic. It is better to weight each task category by business traffic, calculate a weighted pass rate, and retain independent data for each category.

    Averages can also hide tail problems. Suppose a model has a very short average response time but times out a few times per hundred requests. For an online service, those timeouts may deserve more attention than the average. Latency should therefore be examined at percentiles such as P50, P95, and P99.

    In addition, error types should be counted separately. Refusals, format errors, factual errors, and hallucinated citations have different business impacts. They should be recorded separately and assigned severity levels.

    5. Put Cost and Latency in the Same Table

    Quality improvements usually come with higher cost or slower responses. During selection, quality, cost, and latency should be compared in the same table.

    The following metrics can be used:

    • Unit effective cost: cost per thousand requests divided by pass rate, yielding the cost per correct result.
    • Time to first token: the response speed perceived by users.
    • Total generation latency: affects throughput for batch tasks.
    • Quality-cost curve: plot cost on the horizontal axis and quality on the vertical axis to identify the point of best cost-effectiveness.

    In many scenarios, a smaller model with better prompts or retrieval augmentation can achieve the effect of a larger model at a fraction of the cost. Such options should also be included as candidates during evaluation, rather than comparing only the models themselves.

    6. How to Judge Multimodal Capabilities

    Whether multimodal capability matters depends on whether the business actually needs to process non-text input. If the business handles only text, differences in image understanding should not be included in the selection score.

    If the business does have multimodal requirements, test against specific scenarios, for example:

    • Chart data reading: whether the model can accurately read axis values and legend meanings.
    • Receipt and form recognition: text recognition accuracy and field alignment capability.
    • Screenshot understanding: interface element recognition and inference of operation steps.

    Use real samples in testing, and distinguish between “can see” and “can see accurately.” A model being able to describe an image does not mean it can accurately extract numerical values.

    7. Canary Release and Rollback Mechanisms

    No matter how good the evaluation results are, they should first be validated with a small amount of traffic. A recommended approach is as follows:

    1. Connect the new model to the routing layer and initially send only a small share of requests to it, for example, five percent.
    2. Run an A/B comparison between outputs from the old and new versions, recording the quality complaint rate, refusal rate, format error rate, and average latency.
    3. Set clear stop conditions. For example, if the format error rate exceeds a threshold, or the complaint rate rises for two consecutive periods, automatically switch back to the old version.
    4. Keep the old version’s invocation path until the new version has run stably for a complete business cycle.

    Logs and evaluation data can be stored in object storage, and monitoring metrics can be configured with thresholds in a monitoring and alerting system. These are general engineering capabilities that are independent of any specific model and should be prepared before model selection.

    8. Decision Records and Continuous Evaluation

    The behavior of model services may change with version updates, and the same API may return different results several months later. Therefore, each selection decision should record the following information: evaluation set version, evaluation script version, model version number, invocation parameters, evaluation date, results for each metric, and the rationale for the final decision.

    This record helps the team review decisions and also makes it possible to quickly reuse the evaluation process when a new version is released. Model selection then changes from a one-time judgment into an engineering process that can run continuously.