LLM Selection Beyond Leaderboards: A Practical Guide to Evaluation and Cost

    Choose large language models using task-specific evidence, not rankings alone. This guide explains how to define measurable requirements, build a private evaluation set, compare total cost per task, and validate self-hosted deployments, with additional checks for Chinese-language and multimodal workloads.

    What Leaderboards Reveal—and What They Miss

    Leaderboards are useful for initial screening. They show which models are among the leading contenders and how evaluation priorities are changing. In recent years, many mainstream evaluation frameworks have placed greater emphasis on agent capabilities, including task decomposition, tool use, and error correction. The focus is shifting from “How well does the model answer?” to “Can it complete the task?”

    However, leaderboard testing conditions often differ from enterprise requirements. Many benchmarks rely primarily on English-language material and text-only questions, with testing conducted through public APIs. They measure a model’s baseline capabilities under those conditions.

    Businesses also need to consider Chinese-language performance, mixed text-and-image inputs, deployment environments, and cost per request. English-heavy benchmarks do not necessarily predict performance on Chinese-language workloads; those capabilities require separate validation.

    Before citing a specific score or weighting, review the publisher’s documentation. Confirm the evaluation organization, publication date, sample composition, and scoring methodology. Numbers without this context should not directly drive a selection decision.

    1. Turn Requirements into Measurable Criteria

    Before selecting models, answer three questions:

    1. What is the task? Customer support, contract summarization, code generation, and report analysis place very different demands on a model.
    2. What does the input contain? Text, scanned PDFs, tables, screenshots, and audio determine which capabilities are required.
    3. What constraints apply? Consider whether data can leave the controlled environment, acceptable latency, peak concurrency, and the budget per request.

    Document the answers in a requirements table that can guide each subsequent step:

    Requirement Example Purpose
    Primary task Extract contract clauses Determines the evaluation set’s composition
    Input format Scanned PDFs with tables spanning multiple pages Determines whether multimodal capabilities are needed
    Language Primarily Chinese, with English technical terms Determines the emphasis on Chinese-language evaluation
    Deployment Self-hosted; data must remain within the controlled environment Limits the candidate pool
    Response target Time to first token no greater than 2 seconds Guides model size and inference configuration
    Cost ceiling No more than CNY 0.50 per contract processed Defines the affordable model range

    With clear requirements, leaderboards have a specific role: narrowing the candidate pool. The final decision must come from testing on your own data.

    2. A Three-Step Model Selection Process

    Step 1: Use Leaderboards to Build a Shortlist

    Choose three to six models whose evaluated capabilities align with your task. Focus on relevant subscores rather than overall rankings.

    For coding assistance, examine coding results. For Chinese-language customer support, examine Chinese comprehension results. Models separated by only one or two places in an overall ranking may show no meaningful difference in a particular business workflow.

    Step 2: Validate Candidates with a Private Evaluation Set

    This is the decisive step. Build an evaluation set containing at least 200 samples drawn from real business workflows. Each sample should include:

    • Input content: original text, files, or images
    • A reference answer or scoring criteria
    • Task category, such as extraction, rewriting, reasoning, or generation
    • Difficulty level: simple, routine, or complex
    • Flags indicating long text, tables, or mixed text-and-image content

    Test all candidates with the same prompts and parameter settings. Record:

    • Task pass rate: The proportion of outputs that satisfy the scoring criteria.
    • Instruction-following rate: Whether outputs meet requirements for format, fields, and length.
    • Hallucination rate: The proportion of outputs containing fabricated facts or data.
    • Average response time and time to first token.

    Business stakeholders should define acceptance thresholds before testing begins. For example, contract extraction might require a pass rate of at least 95%, while customer support might require at least 90%. In both cases, the hallucination rate must remain below a predefined ceiling.

    Changing thresholds after seeing the results undermines the evaluation.

    Step 3: Calculate Total Cost per Task

    Comparing request prices alone is not enough. A cheaper model with a higher failure rate may require more manual correction and cost more overall.

    Use the following estimate:

    Total cost per task = Model-call cost per task + Failure rate × Manual correction cost per failed output

    Include retry calls in the model-call cost. Account for cache hits, batch processing, and amortized fine-tuning costs where they apply to the planned implementation.

    Consider this illustrative example, with all amounts in Chinese yuan (CNY):

    • Model A costs CNY 0.10 per call and has a 5% failure rate.
    • Model B costs CNY 0.02 per call and has a 20% failure rate.
    • Correcting one failed output costs CNY 2.00.

    The estimated costs are:

    • Model A: 0.10 + 0.05 × 2.00 = CNY 0.20 per task
    • Model B: 0.02 + 0.20 × 2.00 = CNY 0.42 per task

    Although Model A has the higher call price, its total cost per task is roughly half that of Model B. Replace these illustrative figures with your own prices, observed failure rates, and labor costs.

    3. A Self-Hosted Deployment Checklist

    Cloud-based leaderboard results do not directly predict self-hosted performance. Quantization, hardware differences, and GPU memory limits can all affect results.

    Before deployment, validate the following:

    Check Question to answer Validation method
    Quantization-related quality loss How much does the pass rate decline after quantization? Compare the original-precision and quantized versions on the same private evaluation set
    Hardware compatibility Is a usable inference framework available for the target compute platform? Deploy on the actual hardware and verify against vendor documentation
    GPU memory usage How much GPU memory is needed at peak concurrency? Estimate requirements, then measure under the expected load
    Parallelism strategy Can tensor or pipeline parallelism be configured for different GPU counts? Record throughput with different numbers of GPUs
    Serving performance Do latency and error rates meet targets during peak demand? Run load tests at the expected peak
    Operations How will upgrades, rollbacks, and log retention work? Complete an upgrade and rollback rehearsal

    Define acceptance criteria for every item and document the results before production launch. Deployment conclusions without supporting records are difficult to reuse in later reviews.

    4. Testing Chinese-Language and Multimodal Workloads

    Chinese-language test cases should cover:

    • Understanding idioms, proverbs, and two-part allegorical sayings
    • Ambiguity caused by homophones and words with multiple meanings
    • Industry abbreviations and specialized terminology
    • Long documents exceeding 10,000 Chinese characters, checking whether summaries omit important clauses

    For multimodal workloads, focus on variations in input quality. Scans may be skewed, blurry, or overlaid with stamps, while tables may span multiple pages.

    Track these two metrics separately:

    • Field extraction accuracy: Whether key fields are extracted correctly.
    • Numeric consistency rate: Whether numbers extracted from tables and charts match the source image.

    Text-only leaderboards cannot capture these issues. Both metrics therefore need separate scoring within the private evaluation set.

    5. Common Selection Mistakes

    1. Looking only at overall rankings instead of task-relevant subscores.
    2. Assuming public API results predict self-hosted performance.
    3. Testing only simple samples while excluding long documents and unusual inputs.
    4. Comparing request prices while ignoring retries, failure correction, and manual review.
    5. Repeatedly adjusting thresholds after testing until a preferred model wins.

    6. Put Leaderboards Back into the Decision Process

    Leaderboards support initial screening. Private evaluation sets validate task performance. Total cost per task determines which option is economically viable. Each has a distinct role, and skipping any one of them can distort the decision.

    A practical schedule is to spend one week defining requirements and building the evaluation set, followed by two weeks testing candidates and calculating costs.

    The resulting decision is grounded in your organization’s data and operating conditions—a more reliable basis for model selection than any single leaderboard ranking.