Scenario-Based Evaluation for LLM Selection: A Practical Guide
This article explains how to move from leaderboard-driven model selection to scenario-based evaluation. It covers task definition, difficulty tiers, multidimensional scoring, evidence anchoring, cost estimation, side-by-side comparison, scenario search, visualization, and an enterprise rollout process. Readers will learn a reusable methodology for choosing models that are better suited and more cost-effective for real business tasks across text, image, audio, video, and web generation.
As the number of large models continues to grow, model selection is shifting from “check the leaderboard and compare benchmark scores” to “examine the scenario, calculate the cost, and validate the results.” The reason is straightforward: general-purpose leaderboards are good at providing an average capability profile, but they struggle to answer specific business questions. For example, is a particular type of long-document summarization stable? Can a particular type of code generation run directly? Does a particular type of image generation meet layout requirements? Does a particular type of web page generation support responsive layouts? For developers and enterprises, what matters is an evidence chain for real tasks, not a single score.
This article focuses on how to choose models that are more suitable and more cost-effective based on real scenarios, and it abstracts a reusable evaluation and selection methodology. The method does not depend on a specific vendor or product and applies to evaluating text, image, audio, video, and web generation capabilities.
1. Why Benchmark Scores Alone Are Not Enough
General evaluation scores are valuable, but they usually compress a large amount of task variation. A model ranked high on a composite leaderboard is not necessarily the best performer in a vertical business scenario. A model that excels on reasoning questions may not have an advantage in format following, long-context stability, or cost control.
Common problems include:
- Task distribution does not match business distribution: Leaderboard questions tend to be general-purpose, while business data contains industry terminology, fixed formats, complex constraints, and long-tail scenarios.
- Scoring dimensions are too narrow: Looking only at scores without examining error types, error locations, and fixability can lead to misjudgment.
- Cost is underestimated: At the same call volume, the input-to-output token ratio, retry rate, and manual review cost all affect total cost of ownership.
- Multimodal evaluation is fragmented: Text, image, web generation, and other capabilities require different evaluation criteria and cannot be compared simply with the same score.
- Iteration is fast: Model versions are updated frequently, so historical conclusions need to be traceable and reproducible.
Therefore, scientific model selection should establish a closed loop of “scenario task set + multidimensional scoring + cost estimation + result visualization,” rather than relying on a one-time ranking conclusion.
2. Define the Scenario Before Defining the Model
The first step in model selection is not to list models, but to define the business scenario. It is recommended to break a scenario into five elements: task, input, output, constraints, and acceptance criteria.
For example, a text summarization task can be defined as follows: the input is multiple technical documents; the output is a summary within a specified word limit; the constraints are to retain key metrics and conclusions; and the acceptance criteria are factual consistency, coverage, and readability. A code generation task can be defined as follows: the input is a natural language requirement and existing code context; the output is runnable code; the constraints are not to introduce undeclared dependencies; and the acceptance criteria are compilation success, unit test success, and complete boundary handling.
The following pseudoconfiguration can be used to describe an evaluation scenario:
scene: technical_summarytask_type: text_generationinput_schema:document: stringmax_words: intoutput_schema:summary: stringconstraints:- keep_key_metrics- no_hallucinationmetrics:- fact_consistency- coverage- readability- format_followingdifficulty_levels:- basic- advanced- hard
These fields are methodology examples, not interface definitions for a particular platform. The key point is that only after acceptance criteria are clearly written can subsequent scoring have a basis.
3. Task Tiers: Basic, Advanced, and Hard
The same capability can perform very differently at different difficulty levels. It is recommended to set basic, advanced, and hard tasks for each scenario.
- Basic tasks: Verify whether the model understands instructions and can output a basically correct format.
- Advanced tasks: Add multiple constraints, long context, cross-paragraph reasoning, or complex format requirements.
- Hard tasks: Introduce noise, conflicting information, long-tail edge cases, multi-turn correction, or business rule constraints.
The value of tiered evaluation is to identify “capability cliffs.” Some models perform stably on basic tasks but show a clear increase in error rate on hard tasks. Some models have average basic scores but are more robust under complex constraints. For production systems, the latter may be more valuable.
4. Multidimensional Scoring and Evidence Anchoring
Scoring should not consist of only a single total score. It should cover at least the following dimensions:
- Correctness: Are the facts correct, and does the result meet the task objective?
- Completeness: Are key points covered, and are any constraints omitted?
- Format following: Does the output follow the specified structure, fields, and length?
- Stability: Is result variance acceptable under repeated calls?
- Fixability: Are errors easy to locate and fix through prompting?
- Safety and compliance: Is there a risk of inappropriate content, unauthorized information, or sensitive data leakage?
- Cost efficiency: How many calls and how much token consumption are required to achieve acceptable quality?
For multimodal tasks such as image and web page generation, additional dimensions such as layout restoration, text rendering, interaction usability, and responsive layout should be added. Evidence anchoring is important: not only provide scores, but also mark error locations and reasons. Examples include misplaced text in an image, a non-clickable button on a web page, or a factual deviation in a summary. Only in this way can evaluation conclusions be translated into improvement actions.
5. Cost Estimation: Making “Cost-Effective” Concrete
“More cost-effective” does not mean a lower unit price. It means a lower comprehensive cost per unit of effective output. A simplified formula can be used for estimation:
def estimate_total_cost(tokens_in, tokens_out, price_in, price_out, retry_rate, review_rate):# price_in / price_out are placeholder parameters; fill them in with actual pricing.generation_cost = tokens_in * price_in + tokens_out * price_outretry_cost = generation_cost * retry_ratereview_cost = generation_cost * review_ratereturn generation_cost + retry_cost + review_cost
Actual evaluation should also consider:
- The input-to-output token ratio;
- The failure retry rate;
- The manual review ratio;
- The cache hit rate;
- Resource utilization from batch processing and asynchronous calls;
- Exchange rate conversion and billing cycle differences.
It is recommended to use “total cost of samples that pass acceptance / number of passing samples” as a core metric. This avoids a low-price model becoming more expensive overall due to retries and manual fixes.
6. Side-by-Side Comparison and Scenario-Based Search
Side-by-side comparison should ensure the same task, the same input, and the same scoring standard. Outputs are best displayed side by side to make differences easy to observe. For example, for the same technical document summary, Model A may be more concise, Model B may have fuller coverage, and Model C may have more stable formatting. Only through side-by-side comparison can teams determine which type of difference matters more to the business.
Scenario-based search addresses the problem of “I know the requirement, but I do not know which leaderboard to look at.” Users can search by keyword for model performance in specific scenarios, such as style transfer, table extraction, code repair, image text rendering, and e-commerce landing page generation. Search results should include task descriptions, scoring dimensions, and sample size to avoid comparisons without context.
7. Visualization and Selection Reports
Radar charts are suitable for showing the balance of multidimensional capabilities, histograms are suitable for comparing distributions under the same metric, and scatter plots can be used to analyze the relationship between quality and cost. A selection report should include at least:
- Business scenarios and acceptance criteria;
- Candidate model list and version information;
- Task set size and difficulty distribution;
- Scores by dimension and typical failure cases;
- Cost estimation and sensitivity analysis;
- Recommendation conclusions and a retest plan.
Visualization is not for appearance; it is for identifying weaknesses. For example, if a model has a high composite score but weak format following, it is not suitable for strongly structured output. If a model is low-cost but has a high error rate on hard tasks, it is not suitable for high-risk workflows.
8. Recommended Enterprise Rollout Process
It is recommended to implement the process in the following steps:
- Select 3–5 core scenarios, and prepare 50–200 representative samples for each scenario.
- Define scoring rules clearly, and have at least two people independently annotate a subset of samples to calibrate the criteria.
- Test candidate models at basic, advanced, and hard tiers.
- Record the input, output, latency, tokens, and error type for each call.
- Calculate the quality pass rate and the effective sample cost.
- Organize a joint review with business, engineering, and operations teams.
- Launch a small-traffic gray release and continuously monitor drift and cost changes.
- Establish a monthly or version-level retest mechanism.
At the infrastructure layer, general cloud capabilities can be used to host evaluation services, such as object storage for samples and results, container platforms for running evaluation tasks, logging services for recording call chains, and monitoring and alerting for tracking exception rates. Baidu AI Cloud can provide general cloud infrastructure to support the deployment and operation of such evaluation systems, but specific selection should still be based on measured business data.
Conclusion
The core of model selection is not to find a model that is “always the strongest,” but to find the model that is most suitable under the current scenario, current cost, and current risk constraints. Through scenario-based task sets, multidimensional scoring, evidence anchoring, cost estimation, and visual analysis, teams can turn subjective experience into reviewable engineering decisions. Define the business first, then define the metrics, and finally compare models. This is a more robust and sustainable path for model selection.
