How to Evaluate and Select a New LLM After a Model Update

Why you should re-evaluate after a version update

Model versions move quickly. A new release on the same product line often changes context length, output limits, unit price, latency, and safety policy at the same time. Each change may look small; together they can shift cost structure and behavioral boundaries.

Do not swap the production model just because the new version is “stronger.” Use a consistent method on your own business data. Marketing describes general capability; production systems care about specific prompts, data formats, and compliance constraints.

Evaluation dimensions

List the dimensions you care about before you start, so a single metric does not dominate the decision. Cover at least the following five.

1. Context window and output limit

The context window determines how much input a request can hold—multiple source files, long contracts, or multi-turn history. The output limit caps how much can be generated in one call.

Measure the real length distribution of production requests, not only advertised numbers. If 95% of requests fit in 8K tokens, paying extra for a larger window is often not worth it. If long-document traffic is high, window size can make or break the design.

2. Price and cache billing

Most hosted models bill input and output tokens separately; some offer cheaper rates for cached repeated prefixes. A lower unit price does not always mean lower total cost. A stronger model may need fewer retries but may also produce longer outputs—account for both.

Measure cost as the full cost of completing a task. See the calculation in a later section.

3. Latency

Latency includes time to first token and end-to-end generation time. Interactive UX is more sensitive to first-token time. Batch jobs care more about throughput and concurrency quotas.

Vendor speed claims usually assume a specific load. Benchmark with your own request mix and report percentiles such as P50 and P95, not only the mean.

4. False refusal rate from safety policy

Safety filters block risky output, but they can also flag legitimate requests—for example medical or biology coursework or security-research questions. False refusals hurt UX and conversion.

Split refusals into correct refusals and false refusals. The metric that matters is the false-refusal rate: the share of compliant requests that are blocked.

5. Readability and style

Whether output is structured, matches the user’s context, and avoids opaque jargon affects human review and downstream processing. These qualities are hard to score fully automatically; use human sampling.

Build your own evaluation set

The eval set is the foundation. Stratified-sample by scenario, keep 50–200 items per layer, and cover:

• Code generation and bugfix, with real errors and log snippets
• Document understanding and summarization, including long text and nested headings
• Structured extraction, such as fields from tickets into JSON
• Math and logical reasoning with a verifiable final answer
• Compliant but sensitive edge cases, to measure false refusals
• Clearly violating samples, to confirm blocking has not degraded

Each item should include the input, expected points or a gold answer, and a scenario tag. Desensitize samples; do not include real user PII. Refresh the set regularly. Reusing the same items for too long can lead to implicit overfitting and inflated scores.

Example evaluation script

The following is a vendor-agnostic skeleton. client adapts any model API to a common generate interface that returns text, token usage, and a refusal flag. judge can be rule matching, another model, or human labels.

import time
from dataclasses import dataclass
@dataclass
class Result:
    sample_id: str
    passed: bool
    latency_s: float
    input_tokens: int
    output_tokens: int
    refused: bool
def run_eval(client, model_name, samples, judge):
    results = []
    for s in samples:
        start = time.perf_counter()
        resp = client.generate(model=model_name, prompt=s["prompt"])
        latency = time.perf_counter() - start
        results.append(Result(
            sample_id=s["id"],
            passed=judge(s, resp.text),
            latency_s=latency,
            input_tokens=resp.usage.input_tokens,
            output_tokens=resp.usage.output_tokens,
            refused=resp.refused,
        ))
    return results
def summarize(results, samples):
    total = len(results)
    should_refuse = {s["id"]: s["should_refuse"] for s in samples}
    safe_total = sum(1 for s in samples if not s["should_refuse"])
    false_refusals = sum(
        1 for r in results if r.refused and not should_refuse[r.sample_id]
    )
    return {
        "pass_rate": sum(r.passed for r in results) / total,
        "mean_latency_s": sum(r.latency_s for r in results) / total,
        "avg_input_tokens": sum(r.input_tokens for r in results) / total,
        "avg_output_tokens": sum(r.output_tokens for r in results) / total,
        "false_refusal_rate": false_refusals / safe_total,
    }

Notes: the denominator for false-refusal rate must be the count of compliant samples only; including violating samples dilutes the metric. Measure latency under the same network and concurrency. Run each sample 3–5 times and look at the distribution of pass rates to reduce noise from generation randomness.

Cost per successful task

Comparing unit prices alone is not enough. A better metric is average cost per passed sample:

Cost per call = input tokens × input price + output tokens × output price + cached tokens × cache price

Cost per passed sample = cost per call × average calls per sample ÷ pass rate

Illustrative numbers: model A costs 0.010 per call with 70% pass rate; model B costs 0.012 with 92% pass rate. Cost per passed sample is about 0.0143 for A and 0.0130 for B. B is more expensive per call but cheaper per success.

If caching is available, measure hit rate from logs. Long, frequently reused system prompts can cut input cost substantially. Do not assume a theoretical hit rate.

Measuring and trading off false refusals

Two requirements: score compliant and violating samples separately, and use the same set when comparing models.

Prompting with business context can reduce false refusals, but you must also check that violating samples are still blocked. A drop in false refusals with a rise in missed violations is not acceptable.

Report both false-refusal rate and miss (leak) rate so policy changes can be judged honestly.

Blinded human review of output quality

Automatic metrics miss style. A common complement:

1. Strip model identifiers and normalize format
2. Shuffle answers from different models
3. Score accuracy, completeness, readability, and redundancy on a 1–5 scale
4. Have at least two raters score independently and compute agreement (for example Cohen’s kappa). If agreement is low, align the rubric before continuing

Cross-check with automatic metrics. High auto pass rate with poor human readability often means the eval set does not match real use.

Canary rollout and rollback

After eval, ship in stages:

• Shadow traffic: the new model scores live requests without returning results to users
• Small canary: send 5–20% of traffic to the new model; watch error rate, latency, complaints, and human-review volume
• Full cutover: expand only after canary metrics are stable

Write rollback rules before launch—for example false-refusal rate more than 2 percentage points above baseline, or mean latency above a threshold. Thresholds should match business tolerance; those numbers are examples only.

Keep model parameters and version IDs in configuration so switch and rollback do not require a code release.