A Practical Framework for Evaluating New Large Language Models

    Systematic evaluation is essential for making informed decisions when adopting large language models, as public benchmarks alone cannot answer whether a model will perform well in your specific business scenarios. This article provides a structured framework for assessing LLMs across seven key dimensions, including reasoning, coding ability, long-context utilization, multimodal understanding, cost, and stability. It covers practical methods for building custom evaluation sets, conducting blind tests, automating evaluation with code, and calculating true cost per completed task. A five-step model selection process and common pitfalls are also discussed to help engineering teams establish a sustainable evaluation mechanism that supports deployment decisions with reliable data.

    Why Systematic Evaluation Matters

    When a new model is released, the available information typically centers on a few headline metrics: parameter count, context window size, and leaderboard rankings. While these figures offer some reference value, they do not answer three practical questions: How well will the model perform in my specific business scenario? What is the cost per task? And what is the risk if it produces errors?

    The goal of systematic evaluation is to convert these questions into measurable metrics. Evaluation results should come from your own data and workflows—public information serves only as a preliminary filter.

    Breaking Down the Evaluation Dimensions

    Before testing begins, define the dimensions you need to assess. The following categories are commonly used.

    1. Basic Reasoning and Instruction Following

    This dimension examines whether a model can understand complex instructions, produce output in the required format, and complete multi-step reasoning. Test samples should cover mathematical calculations, logical inference, structured extraction, and tasks with conditional constraints.

    2. Engineering and Coding Capability

    This includes function-level code generation, bug identification and repair, repository-level understanding across multiple files, unit test writing, and tool invocation. These tasks can typically be judged by automated tests, making the assessment relatively objective.

    3. Long-Context Utilization

    A large context window only indicates how much input a model can accept—it does not guarantee the model will effectively use all of that input. In practice, information located in the middle of a long document is often overlooked. Evaluation should include “needle-in-a-haystack” retrieval tests and cross-paragraph reasoning tasks, while tracking how accuracy changes as input length increases.

    4. Multimodal Understanding

    For image or video inputs, assess the model’s ability to recognize objects, describe content, interpret charts, and reason about temporal sequences. Note the output format. If the output is text, focus on description accuracy and information completeness. If the task requires generating a UI or image, separate evaluation criteria must be defined.

    5. Long-Horizon Task Execution

    Many real-world tasks require dozens of sequential steps—for example, building a small project from a requirements document. Key evaluation points include how well the model maintains intermediate state, recovers from errors, and whether the final deliverable meets acceptance criteria.

    6. Cost and Latency

    This covers the billed amount per task, time to first token, overall generation speed, and concurrent throughput. These metrics determine whether a model can be deployed in a production environment.

    7. Stability and Safety

    This includes hallucination rate, rate of unnecessary refusals, resistance to malicious instructions, and the consistency of outputs when the same input is run multiple times.

    Evaluation Methodology

    Using Public Leaderboards Correctly

    Public leaderboards are suitable for narrowing down candidate models, but not for making final procurement decisions. There are three main limitations. First, the distribution of test questions may differ significantly from your business context. Second, some test items may already be part of the training data, inflating scores. Third, model versions change frequently, and leaderboard data may be outdated.

    When using leaderboards, record the board name, version, and query date, and check whether the question types align with your business needs.

    Building a Custom Evaluation Set

    A custom evaluation set is the core basis for decision-making. Follow these steps:

    1. Sample from real business scenarios, covering high-frequency cases, long-tail cases, and edge cases.
    2. Stratify by difficulty level, ensuring a sufficient number of samples per category—commonly tens to hundreds per type.
    3. Define annotation guidelines that clearly specify what counts as a pass. Have at least two annotators label independently, calculate inter-annotator agreement, and review any disagreeing samples.
    4. Desensitize sensitive data before using it in evaluation.

    Blind Testing and Pairwise Comparison

    When output quality is difficult to judge by fixed rules, blind testing is an effective approach. Hide model identities, randomly order multiple responses to the same question, and have reviewers score them on a scale or simply choose the better one. Randomization reduces bias from brand recognition or position effects. You can calculate win rates for each model and use scoring methods like Elo to produce relative rankings.

    Automating Evaluation with Code

    The following code demonstrates a minimal evaluation framework that runs test cases in batches and calculates the pass rate. The model invocation function needs to be replaced with your actual API call.

    1. from dataclasses import dataclass
    2. from typing import Callable
    3. @dataclass
    4. class Case:
    5. prompt: str
    6. checker: Callable[[str], bool] # Takes model output, returns whether it passed
    7. def call_model(model_name: str, prompt: str) -> str:
    8. """Wrap the evaluation model's API call. Must be implemented for your service."""
    9. raise NotImplementedError
    10. def run_eval(model_name: str, cases: list[Case], repeat: int = 3) -> float:
    11. """Run each case multiple times and return overall pass rate to observe stability."""
    12. passed, total = 0, 0
    13. for case in cases:
    14. for _ in range(repeat):
    15. output = call_model(model_name, case.prompt)
    16. passed += int(case.checker(output))
    17. total += 1
    18. return passed / total if total else 0.0

    Running each case repeatedly serves a dual purpose: measuring both accuracy and variance. If results for the same case vary significantly across runs, the model is unstable for that type of task, and your deployment plan should include additional validation or human review.

    Statistical Significance

    A difference in scores between two models can sometimes be attributed to random fluctuation. Before drawing conclusions, calculate confidence intervals—bootstrapping (resampling) is a common approach. If the intervals overlap significantly, the evidence is insufficient; increase the sample size before making a judgment.

    Converting Cost

    LLM billing is typically tied to input and output token counts, and some services apply separate pricing for cached input tokens. The cost of a single task can be estimated with this formula:

    Cost per task = cached input tokens × cached input price + non-cached input tokens × non-cached input price + output tokens × output price

    Example: A task requires 20,000 input tokens in total, of which 15,000 are cache hits, and produces 2,000 output tokens. If the three unit prices are a, b, and c (in currency per million tokens), the task cost is (15,000 × a + 5,000 × b + 2,000 × c) ÷ 1,000,000.

    Note that actual costs should also include the overhead of failed retries. If a model has a lower unit price but a significantly lower first-attempt success rate, its overall cost may end up higher. It is therefore recommended to compare models based on the average cost of completing one qualified task.

    Model Selection Process

    Follow these five steps.

    1. Define scenarios and acceptance criteria: Specify task types, input formats, quality baselines, and latency requirements.
    2. Shortlist candidate models: Based on public information, compliance requirements, and deployment options, select two or three candidates.
    3. Run a small-scale pilot: Test candidates with your custom evaluation set for one to two weeks, recording results and anomaly samples.
    4. Score with weighted metrics: Assign weights according to business priorities, as shown in the table below.
    5. Gradual rollout with continuous monitoring: Start with a portion of traffic and track quality metrics on an ongoing basis.
    Evaluation Dimension Example Weight Description
    Task Quality 40% Pass rate on custom evaluation set and blind test win rate
    Cost 25% Average cost per qualified task
    Latency 15% Time to first token and total processing time
    Stability & Safety 20% Hallucination rate, refusal rate, output consistency

    Weights should be determined jointly by business and technical teams. The values in the table are examples only.

    Common Pitfalls

    • Relying on a single ranking: Topping a leaderboard does not mean the model is the best fit for your specific business.
    • Equating context window size with capability: A long context window must be paired with effective utilization tests.
    • Ignoring version changes: Performance can change after the same model is updated. Evaluation reports must record the model version and test date.
    • Treating a demo as representative capability: Curated demo examples do not reflect average performance; rely on large-sample statistics.
    • Overlooking data compliance: Sensitive data used in evaluation and production must be desensitized and handled in accordance with internal data governance policies.

    Building a Continuous Evaluation Mechanism

    Model capabilities change with version iterations, and business requirements evolve as well. It is recommended to keep the evaluation set under version control, rerun it whenever the model is upgraded or business needs shift, and retain historical results for comparison. This way, model selection decisions remain data-driven as conditions change over time.


    This article was adapted for an international audience. All product names, technical terms, and code examples have been preserved from the original content.