Scenario-Based Evaluation for Large Language Models: How to Build the System and Choose the Right Model
General leaderboard rankings measure cross-model comparability against a fixed question set, but they often fail to predict which model actually performs best on a specific business workload. This article explains why scenario-based evaluation matters, how to design a representative test set and a multi-dimensional scoring framework, and how to use boundary testing to expose failure modes. It also shows how to turn raw capability scores into a cost-aware decision table and outlines the engineering building blocks needed to run such evaluations continuously. The goal is to make model selection reproducible, data-driven, and adaptable as both business needs and models evolve.
1. The Gap Between Leaderboard Scores and Business Selection
Many teams start model selection by opening a leaderboard and picking whoever scores highest. In real projects, though, the leaderboard winner often performs only average on their own business data—sometimes worse than lower-ranked models. This gap is not a coincidence. It comes from a structural mismatch between general evaluation and business goals.
General leaderboards solve a “horizontal comparability” problem: they score every model against the same question bank. Business selection solves a “vertical fit” problem—whether a given model can reliably meet the bar under a specific task distribution, cost budget, latency requirement, and stability constraint. The two optimize different objective functions. Using the former to guide the latter is essentially substituting an average score for a targeted assessment.
Scenario-based evaluation exists to bridge this gap. Instead of asking “which is the strongest model,” it asks “in my scenario, which model is the best overall choice.”
2. Why General Leaderboards Fall Short
General leaderboards have three main limitations:
- Task distribution mismatch: Leaderboard questions tend to be standardized items such as knowledge Q&A and general reasoning. Real business workloads may concentrate on narrow scenarios like long-document extraction, structured output, and multi-turn tool calling, and a model’s strengths may not cover what you actually need.
- Single dimension: Most leaderboards reflect only a “capability score” and ignore engineering metrics such as cost, latency, concurrent throughput, stability, and reproducibility. In production systems, these dimensions often determine success or failure.
- Static and lagging: A leaderboard is a fixed snapshot. It cannot reflect the dynamic effects of model version iterations, rate-limiting policy changes, and vendor price adjustments.
So a business-oriented evaluation system must be scenario-based, multi-dimensional, and continuously updated.
3. How to Build Scenario-Based Evaluation
3.1 Test Set Design
The test set is the foundation of the evaluation system. Design it around the principle of “staying close to the real distribution while covering key boundaries”:
- Representative: Sample from real business logs and build test cases after anonymization, ensuring the input distribution matches production.
- Layered: Group cases into tiers by difficulty and scenario—for example, “common / complex / extreme”—so you can pinpoint how a model performs across difficulty bands.
- Verifiable: Every case should have a clear scoring basis, either through rule-based checks or through combined human/model review.
A representative case structure can be abstracted as:
{"id": "img-text-render-0012","category": "multimodal.image_generation","difficulty": "hard","input": "Generate a poster containing five languages: Chinese, English, Japanese, French, and Arabic","rubric": [{"dim": "language_completeness", "weight": 0.4},{"dim": "glyph_correctness", "weight": 0.4},{"dim": "aesthetic_quality", "weight": 0.2}],"pass_threshold": 0.75}
3.2 Evaluation Dimensions
Beyond capability, engineering metrics must go into the same decision table. At minimum, cover:
- Capability: task accuracy, instruction adherence, output format compliance.
- Cost: per-token pricing for input and output, converted into the “total cost to complete one typical task.”
- Performance: time to first token, total latency, and throughput and rate-limiting behavior under concurrency.
- Stability: output variance across repeated runs, reproducibility, and success rate under sustained operation.
4. Boundary Testing: Finding the Model’s Ceiling
The most informative evaluations often come from “extreme challenge” cases—tasks where every model might fail. Their value is not in ranking winners and losers but in revealing differences in failure modes.
Take multimodal generation as an example. You can design several kinds of boundary tasks:
- Complex text rendering: Ask the model to render multiple languages at once in a single image, testing its handling of glyphs, layout, and language completeness.
- Precise counting and occlusion reasoning: Ask it to accurately count the number and category of objects in stacked, partially occluded, or partially open states. These “count it correctly” tasks are extremely hard for generation models, and different models will separate clearly between “roughly right count but wrong details” and “even the category is wrong.”
- Multi-constraint style transfer: For example, satisfying several compound constraints at the same time, such as a specific artistic style, a mineral-pigment texture, and a flat composition.
The point of these cases: even if nobody “passes,” you can clearly see where each model breaks down—missing glyphs, counting errors, or style drift. This interpretability of failure modes is where boundary testing delivers real decision value. So the test set should span a continuous difficulty spectrum, from a few objects to many, and from single constraints to stacked constraints.
5. Cost-Performance and Decision-Making: Don’t Just Look at Capability
The final output of evaluation is not a capability leaderboard but a comprehensive decision table. A common pattern: two models differ by only a tiny margin in capability score, yet their prices differ by an order of magnitude. Here, the rational choice is often the model with slightly lower capability but significantly better cost.
You can formalize the decision process as a constrained optimization problem: under hard constraints such as latency and success rate, maximize a weighted score of “capability minus cost penalty.” For example:
def select_model(candidates, weights, constraints):# constraints: hard thresholds, e.g., minimum success rate, maximum latencyfeasible = [m for m in candidates if meets_constraints(m, constraints)]def score(m):return (weights["capability"] * m["capability_score"]- weights["cost"] * m["normalized_cost"]+ weights["latency"] * m["latency_score"])return max(feasible, key=score)
The key is that normalized_cost normalizes per-unit prices of different tiers to the same scale, while weights are set by the business side based on scenario preferences—high-value, low-frequency tasks can favor capability, while high-concurrency, high-frequency tasks should favor cost and latency.
6. Engineering Implementation of the Evaluation Platform
To solidify these methods into a continuously running system, consider the following engineering practices:
- Case management: Store cases and scoring rules with versioning, so evaluation results are traceable and reproducible.
- Scheduling and concurrency: Use a task queue to smooth out large-scale runs, control the call rate to the model, and avoid triggering rate limits.
- Result storage and analysis: Archive each run’s inputs, outputs, and scoring details to object storage for easy cross-comparison and regression analysis.
- Observability: Use logging and monitoring/alerting services to track call success rate, latency distribution, and anomalies, so you can promptly catch fluctuations caused by model version or API changes.
- Elastic compute: Evaluation is a typical high-concurrency, intermittent workload. Use a container platform to scale on demand and release resources when done, keeping costs under control.
This kind of evaluation infrastructure is a general capability that any cloud environment can support—including the object storage, message queues, container platforms, logging, and monitoring/alerting product categories offered by Baidu AI Cloud. When selecting a provider, just check whether it meets the capabilities above; there is no need to bind to a specific implementation.
7. Conclusion
Model selection is moving from “going with a gut feeling” to “going with data.” The value of scenario-based evaluation is to turn the vague question of “which model is better” into a reproducible, multi-dimensional, continuously updated quantitative conclusion: looking at capability as well as cost and performance; testing common performance as well as failure boundaries; and giving not just a ranking but explainable failure modes. Building such a system is what lets every model selection rest on solid evidence and keep evolving with both the business and the models.
