Scenario-Based Evaluation: How to Make LLM Selection Decisions That Hold Up in Production

    This article explains how to build a repeatable, scenario-driven evaluation process for choosing large language models (LLMs) in real business contexts. Instead of relying on public leaderboards or one-off benchmarks, it outlines how to define decision boundaries, design test cases that cover both capabilities and failure modes, establish auditable scoring criteria, and track cost, latency, and quality in a single view. The goal is to turn evaluation from a snapshot activity into a continuous regression system that supports model switching decisions with evidence.

    Define the Decision First, Then Design the Test Cases

    Evaluation is not about producing a single aggregate leaderboard. It is about answering a set of actionable questions: whether a scenario can go live, how to split traffic between a primary model and a fallback model, how much quality degradation is acceptable, and whether the cost per successful call stays under control. Before writing any test cases, clarify the decision boundaries.

    At minimum, four types of information should be fixed. First, the scenario objective—for example, customer service response, document summarization, structured extraction, multi-turn rewriting, image-text generation, or code completion. Keep only the success criteria that the business actually cares about. Second, constraints: latency limits, context length, output format, compliance restrictions, and acceptable failure types. Third, cost measurement: unify the basis for input, output, retries, and human review instead of only looking at sticker prices. Fourth, switch conditions: for example, only change models when key metrics fall below thresholds for consecutive runs, when a certain error category spikes, or when unit cost exceeds budget.

    Without these boundaries, even a large set of test cases is just sample display. With them, a few hundred highly discriminative cases are often more useful than tens of thousands of homogeneous questions.

    Test Cases Should Cover Capabilities and Failures

    Scenario-based test cases can be organized along the dimensions of capability, difficulty, and risk—not by model vendor.

    Capability dimensions include instruction following, factual consistency, long-context retrieval, format stability, multi-turn memory, refusal and safety boundaries, and result verification before and after tool calls. If the business involves images or rich media, also add separate dimensions for composition constraints, text rendering, style consistency, and fidelity to reference descriptions. Avoid replacing objective checkable items with subjective judgments like “looks good.”

    Difficulty should be tiered. A regular set is used for daily regression. A boundary set intentionally amplifies rare but high-loss situations, such as mixed Chinese-English text, negative constraints, tables and units, time and numbers, role conflicts, or local modifications within very long materials. A stress set does not aim for high scores; it is only used to observe the ceiling and the mode of failure: hallucination, missed constraints, broken formatting, or a sharp rise in cost due to retries.

    Each test case should record the input, expected output, automatically checkable checkpoints, and points that require human judgment. For example, “output must be a valid structure and all required fields must be present” can be automated. “Tone must match brand voice” or “the image must actually show the specified material” requires sampled human review. Do not mix human and automated scores into a single unexplainable total.

    Scoring Criteria Must Be Auditable

    A composite score tends to hide failures. A more robust approach is to keep dimension-level scores and define weights and veto items in advance. Format errors, factual errors in key fields, or policy-violating outputs can be one-vote vetoes—they should not be pulled back by strong performance in other dimensions. Weights should come from business impact, not from making a result look better.

    Scoring can be layered. A rule-based layer checks format, length, required fields, forbidden words, and numeric ranges. A model-assisted layer is used only for initial screening—for example, determining whether a summary covers key points—and must itself be calibrated with human sampling. The model must not become an unquestionable authority. A human layer focuses on high-risk samples and cases where the model-assisted layer conflicts with the rule-based layer.

    Any horizontal comparison across models must lock down the prompt, decoding parameters, retry strategy, timeout, and post-processing. Otherwise you are seeing differences in the entire pipeline and mistakenly attributing them to model capability. For generation tasks, also save the original image or text, the scoring rationale, and the reviewer’s name, so the results can be replayed later—not just a single number.

    Put Cost, Latency, and Quality on the Same Table

    When quality is close, the decision often depends on the total cost of one successful outcome. Break down cost into four components: direct input and output fees, failed retries, human sampling and rework, and the extra concurrency or caching needed to meet latency targets. Compare with “cost per valid result” instead of “cost per call.” A cheap model that frequently requires rework can be more expensive than one that succeeds on the first try.

    Latency should also be split into time to first token, total completion time, and queuing latency. Interactive scenarios are more sensitive to the former; batch processing cares more about throughput and the impact of retries on the overall window. Put all results in one view: dimension scores, veto rate, P95 latency, cost per valid result, and representative failure examples. If the price difference is several times or more, as long as the key veto items do not degrade, there is a good reason to keep the expensive model for a few high-risk steps and route routine traffic to a model that meets the thresholds.

    Do not invent prices or scores for any model. Real numbers must come from your own bills, logs, and the same test set, and the observation window must be clearly stated.

    Make Evaluation a Regression Process, Not an Event

    A single offline run only captures a snapshot. Models, prompts, retrieval sources, and business rules change over time. A sustainable approach is to put the test set under version control, distinguishing a golden set, a newly added failure-case set, and outdated questions that are about to be removed. Any time a candidate model, prompt, or post-processing step changes, run the golden set first, then add targeted cases for new failure clusters.

    The pipeline can be simple: load test cases, batch call the model, run rule-based scoring, generate a diff report, put edge cases in a human review queue, and archive conclusions. Object storage works for storing inputs, outputs, and attachments; message queues work for peak shaving and retries; a container platform works for isolating different call environments; logs and monitoring alerts track failure rates, timeouts, and cost spikes. These are generic capabilities and do not need to be tied to any specific vendor’s proprietary interface. If your team already runs workloads on Baidu AI Cloud, treat evaluation jobs as ordinary offline or near-real-time tasks governed by the same rules: permissions, quotas, audit, and data masking take priority over “running more times.”

    Reports should address two audiences. For decision makers, keep only scenario conclusions, risks, and costs. For engineers, keep failure examples, reproduction parameters, and prompts that need fixing. Avoid replacing a layered conclusion with a single aggregate score.

    A Minimal Viable Loop You Can Start With

    A single test case can be expressed in the following structure. The field names are illustrative, not tied to a specific platform.

    1. case_id: extract-invoice-017
    2. scenario: structured extraction
    3. input: desensitized invoice text
    4. must_pass:
    5. - output is parseable
    6. - amount matches currency
    7. - missing fields are explicitly null, not fabricated
    8. judge:
    9. auto: schema + numeric validation
    10. human: only when automatic score conflicts with sample check
    11. budget:
    12. latency_p95_ms: set by business
    13. max_retries: 1

    The minimal loop can run weekly: freeze 50 to 200 golden cases; fix prompts and parameters; compare at least the current solution with one candidate; output veto rate, dimension scores, and cost per valid result; turn new failures into regression cases for the following week. You do not need tens of thousands of samples at the start. First ensure task representativeness, explainable scoring, and traceable cost. Once the scale grows, expand by scenario and periodically remove old questions that the business has abandoned, to prevent the evaluation set from overfitting.

    How to Use the Conclusion

    A model selection conclusion should be written as a conditional statement: under scenario X, constraints Y, and cost ceiling Z, model A can be the primary path, and model B can serve as fallback or for high-risk review. When conditions change, rerun the evaluation. Do not treat a historical ranking as a long-term fact.

    The value of scenario-based evaluation is to turn “feels better” into “fewer errors of a specific type, cheaper per success, and a documented basis for switching.” When the task boundaries are clear, scoring is auditable, cost and quality are shown on the same screen, and regression is continuously updated, your team no longer needs to answer “which model should we use” based on marketing claims.