How to Evaluate LLMs: A Model Selection and Regression Testing Guide

    This guide explains how to evaluate large language models (LLMs) for production use, emphasizing stability and reproducibility over public leaderboard scores and viral demos. It shows how to build self-authored, contamination-resistant test sets and evaluate four key dimensions—multi-step reasoning with traps, strict format constraints, long-context "needle in a haystack" retrieval, and dirty-data handling—each with concrete examples and automated judging scripts. It then explains how to consolidate results into a reusable scorecard, turn that scorecard into a weighted model-selection decision with veto items, and wire evaluation into CI and production observability so regressions are caught before release.

    1. Start with the conclusion: LLM evaluation measures “stable usability,” not “smartness”

    Teams evaluating models often ask: How do you evaluate an LLM? How should you test a model before launch? What are the common mistakes in LLM evaluation? These questions all point to the same conclusion: don’t judge a model only by its public leaderboard scores, and don’t be swayed by impressive demo cases. Instead, use a reproducible test set and automated judging criteria to verify whether it performs reliably on your business tasks.

    Here is a working definition: LLM evaluation means using reproducible test sets, explicit judging criteria, and a regression mechanism to quantify how stable a candidate model is on specific business tasks. It is not about whether a model “looks smart,” but whether it can continuously meet requirements for formatting, reasoning, long context, dirty-data handling, and cost/latency in production.

    So a more reliable question is not “how capable is this model,” but “is it stable, parseable, and reproducible on my type of tasks, and does it meet my cost and compliance thresholds?”

    2. Why benchmark scores and viral demos can’t be launch criteria

    When teams evaluate LLM capabilities, select models, or assess production readiness, the first things they usually see are not business results but two types of content: high scores on public benchmark leaderboards, and demo cases shared on social platforms. For developers who need to integrate a model into a real system, these are worth referencing, but they should not be used directly as launch criteria.

    The problem with benchmark scores is that the questions are public. As soon as a model gains attention, vendors typically train or tune specifically on public datasets. A high score therefore only shows “it performs well on questions it has seen,” not that it is equally stable in business scenarios you have never shown it and that could not be optimized in advance.

    The problem with viral demos is sample selection. A demo may look great because the prompts are simple, the edge cases are favorable, or the cases themselves were cherry-picked. Once you put it into a real workflow facing complex inputs, malformed inputs, and high-concurrency calls, the behavior can be completely different.

    So the question that actually matters before launch is: on my own tasks, can it consistently pass automated judging?

    3. The first principle: write your own questions, don’t test memorized ones

    The core principle of LLM evaluation can be summed up in one sentence: don’t just test questions a model has memorized—design questions it has not seen but that have clear judging criteria.

    Focus on three directions:

    1. Avoid common memorization zones: common-sense Q&A, classic algorithm problems, and public sample questions have likely already appeared in training corpora and cannot reflect real reasoning ability.
    2. Introduce state and traps: make the model maintain state throughout a multi-step task rather than guessing answers by pattern matching.
    3. Use automatically judgeable criteria: don’t just write “is the answer good”—use objective criteria such as “is each step legal,” “does the JSON parse,” and “is the final state correct.”

    Also watch out for contamination when building your own test set:

    • Don’t upload the evaluation set to public platforms, open-source communities, or public code repositories.
    • Before adding new questions, run n-gram or semantic-overlap deduplication against public benchmarks to avoid questions the model has already seen.
    • Rotate questions periodically, especially for frequently used evaluation sets, so they don’t get covered by training data over time.
    • Desensitize or abstract sensitive questions to keep business secrets and personal information out of the evaluation pipeline.

    4. Dimension 1: Multi-step reasoning with traps

    The key to multi-step reasoning ability is that the question must involve state changes and require the model to stay consistent across multiple steps.

    Example question: You have three jugs with capacities of 3 liters, 5 liters, and 8 liters. The 8-liter jug is now full of water. How do you pour water to end up with two 4-liter amounts? List the process step by step.

    To avoid the difficulty of parsing free text, require the model to output in this format:

    1. Output one operation per line, in the format src->dst.
    2. 0 means the 8L jug, 1 means the 5L jug, 2 means the 3L jug.
    3. If it can't be done, output only FAIL.
    4. Do not output explanations.

    Reference solution. Let the three jugs be A=8, B=5, C=3, with initial state (8,0,0):

    1. (8,0,0)
    2. A→B: (3,5,0)
    3. B→C: (3,2,3)
    4. C→A: (6,2,0)
    5. B→C: (6,0,2)
    6. A→B: (1,5,2)
    7. B→C: (1,4,3)
    8. C→A: (4,4,0)

    The corresponding model output can be:

    1. 0->1
    2. 1->2
    3. 2->0
    4. 1->2
    5. 0->1
    6. 1->2
    7. 2->0

    When evaluating, don’t just look at the final answer—also check whether the process is legal. Use a regex to first parse the model output into an operation sequence, then hand it to a judging script:

    1. import re
    2. def parse_ops(text):
    3. if "FAIL" in text:
    4. return []
    5. pairs = re.findall(r"\b([0-2])->([0-2])\b", text)
    6. return [tuple(map(int, p)) for p in pairs]
    7. def judge(ops):
    8. # 0: 8L jug, 1: 5L jug, 2: 3L jug
    9. state = [8, 0, 0]
    10. caps = [8, 5, 3]
    11. if not ops:
    12. return False
    13. for src, dst in ops:
    14. if src == dst:
    15. return False
    16. amount = min(state[src], caps[dst] - state[dst])
    17. if amount <= 0:
    18. return False
    19. state[src] -= amount
    20. state[dst] += amount
    21. return state[0] == 4 and state[1] == 4

    This does not lock in a unique first move by default. As long as every pour is legal and the final state satisfies (4,4,0), it should pass. If a business question explicitly specifies the first move, add a separate check before calling judge, rather than hard-coding it into the general script.

    Common failure types fall into three categories:

    • State drift: the water amounts suddenly stop adding up after some step.
    • Step skipping: skipping a key step and jumping straight to the result.
    • Mid-way abandonment: the model claims it cannot compute or gives a vague explanation.

    These questions are only one observation point for reasoning ability. Don’t draw conclusions from a single question—prepare multiple state-based questions and test repeatedly. Whether a model can reliably walk through this kind of logic chain is a discriminating metric.

    5. Dimension 2: Strict format constraints—the litmus test for production readiness

    Format-constraint ability determines whether a model can enter plugins, APIs, automation pipelines, or post-processing programs. Many teams think a model is good during demos, only to find after integration that an extra sentence, an extra code block, or an extra field causes parsing to fail.

    News text to summarize:

    1. A city's metro operator announces its 2026 holiday transport plan: extended operating hours from September 30 to October 7, plus additional temporary shuttle buses; holiday passenger volume is expected to rise about 20% over a normal period.

    Model output requirements:

    • Output must be a single JSON object.
    • Field names are fixed as EVENT, TIME, IMPACT_SCORE.
    • All field names must be in uppercase English.
    • EVENT is an event summary, TIME is a date, IMPACT_SCORE is a rating from 0 to 10 with one decimal place.
    • Output must be JSON only, with no preamble, no explanatory text, and no Markdown code-block markers.

    To keep scoring from becoming subjective, add a 0–10 banding rule:

    • Impact scope, 0–6 points: single point 1.0; single line 2.0; regional or multiple lines 4.0; city-wide with multi-modal coordination such as temporary shuttles 4.5–5.0; cross-city or large-scale disruption 6.0.
    • Duration, 0–2 points: same day 0.5; 1–3 days 1.0; 4–8 days 1.5; 9 days or more 2.0.
    • Passenger/social impact, 0–2 points: below 5% scores 0.5; 5%–10% scores 1.0; 10%–20% scores 1.5; above 20%, or obvious congestion and emergency-pressure situations, scores 2.0.
    • All bands use 0.5 as the step.

    Under this rule, this example can be derived as follows: the event is a city-wide metro operator announcing holiday transport arrangements with additional shuttle buses, so the impact scope scores 4.5; the duration is September 30 to October 7, which falls in the 4–8 day band, scoring 1.5; passenger volume is expected to rise about 20%, which falls in the 10%–20% band, scoring 1.5. Total: 7.5.

    Expected output example:

    1. {
    2. "EVENT": "A city's metro operator announces its 2026 holiday transport plan",
    3. "TIME": "2026-09-30",
    4. "IMPACT_SCORE": 7.5
    5. }

    Judging criteria:

    • Any extra text such as “Sure, here’s what you prepared…” counts as a failure.
    • Output must be directly parseable by json.loads.
    • No trailing commas, comments, or unescaped characters.
    • Field names, field count, and field types must exactly match the requirements.
    • Numeric values must not be written as strings unless the task explicitly requires strings.
    • IMPACT_SCORE must be between 0 and 10, with one decimal place.
    • If the team only tests structural validity, validate just field types and value ranges, not the specific score; if score reproducibility is needed, use the banding rules above and check that scores follow the band logic.

    These tests are best validated automatically by a script. In production, model output is usually consumed directly by downstream programs, so unstable formatting means the pipeline breaks. A model that can reliably honor format constraints is better suited for automation chains.

    6. Dimension 3: Long-context “needle in a haystack”

    Don’t trust the 128K or 1M context lengths advertised on marketing pages. A stated length is not the same as effective recall ability. A more reliable method is the “needle in a haystack” test.

    How to do it: prepare a long document—say a contract, report, novel, or log. Insert a unique piece of information at different positions, such as:

    1. Mia's cat is named Shadow.

    Then ask:

    1. What is Mia's cat's name?

    If the model answers accurately, it can recall target information buried in a long text. If the answer is wrong, vague, or fabricated, then even with a long stated context, real usability is limited.

    To improve reproducibility, I recommend:

    • Keep the probe sentence short and distinctive (roughly 15–30 characters).
    • Insert probes at the 10%, 30%, 50%, 70%, and 90% positions of the document.
    • Pick several random instances at each position to avoid the model accidentally memorizing a template.
    • Repeat each position 3 times to observe stability.
    • Use documents with different length gradients, e.g., 8K, 32K, 128K, 256K.
    • If a long document contains personal information, contract secrets, or internal data, it must be desensitized before entering the evaluation environment.

    I recommend splitting document length into three tiers:

    • Short documents: 8K, 32K.
    • Long documents: 128K.
    • Very long documents: 256K.

    Actionable thresholds:

    • For 8K and 32K documents, probe recall at every position should be no lower than 90%.
    • For 128K documents, recall at every position should be no lower than 80%, and at middle positions no lower than 70%.
    • For 256K documents, recall at every position should be no lower than 60%, and at middle positions no lower than 50%.

    This reveals the model’s real ability across different lengths and positions, and gives the “very long document” row in the scorecard a clear definition.

    7. Dimension 4: Dirty-data handling

    Real-world input is usually messy. Typos, repeated content, missing fields, contradictory information, and garbled ads all affect output quality. This dimension distinguishes “toy models” from “production-ready models.”

    Dirty-data example:

    1. John Smith Phone: (415) 555-0138? Tel: (415) 555-0139; Address: 88 Maple St, Springfield; Email: unknown; Note: added: John Smith's phone may be (415) 555-0137 (see the contract for the final one).

    Ask the model to extract these fields:

    1. name, phone, email, address, conflict

    Handling rules:

    • If a field exists and is unique, output it directly.
    • If a field is missing, output null—don’t fabricate.
    • If a field has conflicts, set the main field to null and list the candidate values and conflict reason in conflict.
    • If an address can be normalized, complete it (for example, add city/state); if it cannot be confirmed, keep the original text or output null.
    • The address field can accept either “keep the original” or “normalize to city/state”; both count as passing during evaluation.

    Expected output example:

    1. {
    2. "name": "John Smith",
    3. "phone": null,
    4. "email": null,
    5. "address": "88 Maple St, Springfield",
    6. "conflict": "Multiple phone candidates: (415) 555-0138, (415) 555-0139, (415) 555-0137"
    7. }

    The focus is not whether the model “looks smart,” but whether it follows “don’t fabricate, be traceable, be explainable.” A model that stays cautious on dirty data is better suited for real business use.

    8. Consolidate evaluation into a reusable scorecard

    Running scattered tests a few times is not enough. I recommend fixing the evaluation into a scorecard and putting it under version control so you can run regression checks after model upgrades.

    First, record the evaluation metadata:

    1. model_version: candidate-2026-01
    2. eval_date: 2026-09-21
    3. temperature: 0.0
    4. top_p: 1.0
    5. max_output_tokens: 2048
    6. repetitions: 5
    7. random_seed: 42

    You should also document the statistical basis and reproducibility notes:

    • With a sample of 20 questions and an 80% pass threshold, 16/20 already passes and 19/20 also passes. If you want to distinguish subtle differences between models, expand to 50–100 questions and observe confidence intervals.
    • Random seeds and temperature 0 are only reproducible in some self-hosted inference frameworks; most commercial APIs do not guarantee full determinism. For cross-version regression, run several rounds with the same configuration and observe the fluctuation range rather than relying on a single run.

    Design the scorecard like this:

    Dimension Test method Sample size Pass threshold Judging script basis
    Multi-step reasoning Water-pouring, state-transition questions 20 questions Legal process and correct final state ≥ 80% Parse the operation sequence; verify each step is legal and the final state matches
    Format constraints JSON, fixed fields, enum-value output 50 cases Strict-parse success rate ≥ 95% Validate with json.loads, field sets, types, and value ranges
    Long context Multi-position probes 5 positions per document, 3 times each 8K/32K ≥ 90% at every position; 128K ≥ 80% at every position and ≥ 70% at middle; 256K ≥ 60% at every position and ≥ 50% at middle Exact match on probe answers or whitelist answers
    Dirty-data handling Noise, missing, and conflicting field extraction 30 cases No fabrication and conflict-labeling accuracy ≥ 90% Validate fields, missing-field labeling, conflict lists, and acceptable address answers

    The point is not to make the numbers look good, but to make the evaluation repeatable. After every model swap, prompt change, parameter tweak, or inference-framework upgrade, run the regression to see whether ability has actually changed.

    9. Model selection: turn the scorecard into a decision

    A scorecard is just raw material; what it ultimately serves is model selection. When selecting, first narrow the candidate set to verifiable models, then use the scorecard to compute a weighted result.

    9.1 Candidate screening

    The candidate set can include:

    • The model currently in production.
    • The last 2–3 new model versions.
    • Cheaper small or lightweight models.
    • Models that can be deployed privately or meet data-residency requirements.

    For initial screening, focus on:

    • Whether it meets compliance, licensing, and data-residency requirements.
    • Whether it supports the context length your business needs.
    • Whether it supports stable JSON output or function calling.
    • Whether it has usable APIs, rate limiting, monitoring, and version management.
    • Whether per-call cost, P95 latency, and concurrency fit your budget.

    9.2 Dimension weight table

    Weights should differ by business goal. Here is a reference table:

    Business goal Multi-step reasoning Format constraints Long context Dirty-data handling Cost/latency/concurrency
    Automation pipeline 15% 40% 10% 25% 10%
    Long-document Q&A 25% 10% 40% 15% 10%
    Customer-service ticket handling 20% 30% 15% 25% 10%

    9.3 Weighted decision formula and veto items

    I recommend ruling out veto items first, then computing the weighted total.

    Weighted formula:

    1. Candidate total score = Σ (dimension weight × dimension score)

    Dimension scores can be converted to 0–100 from the pass rates in the scorecard.

    Veto items include:

    • Compliance does not allow it, or it cannot meet data-residency requirements.
    • Required desensitization cannot be done, or there is an unacceptable data-retention risk.
    • Format-compliance rate is below 95%.
    • Long-context recall is below the business minimum.
    • Dirty-data mis-extraction rate is above 5%.
    • P95 latency exceeds the business SLA.
    • Per-call cost exceeds the budget cap.
    • Concurrency is insufficient to handle peak traffic.

    9.4 A selection comparison between two candidates

    Suppose there are two candidates:

    Metric Candidate A Candidate B
    Multi-step reasoning accuracy 85% 70%
    Format-constraint success rate 96% 92%
    Long-context performance 128K 82%, 256K 58% 32K 90%, 128K 75%, no 256K support
    Dirty-data handling accuracy 92% 88%
    Per-call cost Higher Lower
    P95 latency 1.8 s 0.9 s
    Concurrency 500 QPS 800 QPS

    If the business goal is an automation pipeline handling mainly short documents, Candidate B may be more cost-effective. If the business involves long contract documents, complex state reasoning, and requires 256K context, then Candidate B triggers a veto for not supporting 256K, making Candidate A the better fit.

    After selection, fold cost, latency, and concurrency into regression monitoring. After each model-version, prompt, parameter, or inference-framework change, besides running the scorecard, also record:

    • Average per-call cost.
    • P95/P99 latency.
    • Error rate under peak concurrency.
    • Parse-failure rate.
    • Timeout rate.
    • Total cost of ownership per unit of business volume.

    10. Practical advice: wire evaluation into CI and production observability

    For developers, evaluation ultimately has to land in the production system. I recommend connecting your self-built evaluation set to the CI pipeline and running it automatically after each model-config change. Evaluation output should not be just for humans to read—hand it to downstream programs for automatic validation.

    Here is a minimal CI sketch to start with:

    1. name: llm-eval
    2. on:
    3. push:
    4. paths:
    5. - model.yaml
    6. - prompts/**
    7. jobs:
    8. eval:
    9. runs-on: ubuntu-latest
    10. steps:
    11. - uses: actions/checkout@v4
    12. - run: python eval/run_eval.py --scorecard scorecard.yaml
    13. - run: python eval/compare_threshold.py --candidate candidate.json --baseline prod.json --block-on-fail
    14. - run: echo "monitor parse_fail, recall, format_pass, dirty_error, p95_latency, cost"

    Triggers should include at least changes to the model version, prompts, or key parameters. Evaluation results need to be compared against a baseline or threshold, and a failure should block the release so you don’t push a model with regressed capabilities to production.

    You can archive evaluation datasets, historical outputs, and failure samples in object storage; use a log service to record each call’s input, output, and error information; and use monitoring and alerting to track changes in parse-failure rate, recall rate, format-compliance rate, and dirty-data mis-extraction rate.

    One compliance note is essential here: evaluation sets, logs, and failure samples may contain personal information and contract secrets. Desensitize them first, or record only what is minimally necessary and apply access control; object-storage archives should restrict access and set retention periods. This is consistent with the “must desensitize before entering the evaluation environment” requirement in the long-context test.

    General cloud infrastructure—object storage, log services, monitoring, and alerting—can help a team close the loop of “evaluate → deploy → observe.”

    Back to the original question: whether a model is truly capable has no one-size-fits-all answer. The most reliable evaluation always hides in that batch of “questions nobody has seen” that you design for your own business.