LLM Model Selection: Scenario-Based Evaluation Methodology and Engineering Practice
Teams choosing an LLM for production often start with a familiar question: which model is the best? This article explains why that question has no universal answer. Generic leaderboard scores hide weaknesses on business-critical tasks and say nothing about cost. It then outlines a scenario-based evaluation approach, grounded in real business request distributions, with task set design, multi-dimensional scoring, a cost-effectiveness metric, stress testing, and practical advice for building an automated evaluation pipeline.
1. Why “Which Model Is the Best?” Is the Wrong Question
Almost every team that starts evaluating large language models (LLMs) for real projects asks the same question: “Which model is the strongest right now?”
It sounds reasonable, but there is no single answer that fits everyone. The reason is simple: model capability is not a single scalar. It is how a model performs across specific tasks, use cases, and constraints. A model that excels at long-text reasoning may be mediocre at multilingual layout, image detail reproduction, or instruction following. A model that ranks near the top of general leaderboards may become unusable in a specific business because of response latency or inference cost.
I once ran large-scale tests across dozens of mainstream models, with tens of thousands of test calls in total. The tests covered text understanding, image generation, multilingual mixing, and other areas. The biggest takeaway was this: there is a gap, widely overlooked, between leaderboard scores and business-driven selection decisions. Until that gap is closed, model selection tends to degrade into gut feeling or copying rankings.
2. Three Limitations of Generic Leaderboards
General-purpose leaderboards are not without value. They are an important reference for understanding a model’s baseline capabilities. However, using them directly to make business decisions leads to three common traps:
1. The evaluation set does not match the business distribution.
Public leaderboards typically use general, balanced, and carefully curated test cases. Inputs in real business scenarios, by contrast, are often highly skewed. Your users might ask about the same type of question 80% of the time, and that question type may carry very little weight on a general leaderboard.
2. A single overall score hides capability gaps.
A model with an overall score of 95 may perform well on most tasks yet happen to be relatively weak on the type of task that matters most to you. The higher the overall score, the easier it is to overlook structural weaknesses.
3. Cost constraints are completely ignored.
This is the most critical issue. I have seen a typical pattern: two models differed by only a fraction of a point in their overall scores and could be considered equivalent. Yet one cost nearly 20 times more per call than the other. If you look only at scores, you would pick the higher-scoring model without hesitation. Once you multiply that choice by your actual business call volume, however, the cost gap quickly grows into a number that is hard to ignore.
3. The Core Idea: Scenario-Based Evaluation
The goal of scenario-based evaluation is to measure how well each model actually fits your business, using your own real task distribution. It rests on three key elements:
- Scenario alignment: The task set must come from real business use cases, not general public benchmarks.
- Multi-dimensional scoring: Don’t judge models by a single overall score. Present a capability profile broken down by dimension.
- Cost built in: Treat price, latency, and stability as first-class evaluation criteria, not an afterthought.
4. How to Build the Task Set
The task set is the foundation of the entire evaluation system. A high-quality task set should follow these principles:
Task set design principles├── Realism: Prioritize de-identified real online requests rather than hand-written cases├── Layering: Classify by business importance (core / important / long tail) with different weights├── Diversity: Cover different difficulty levels and input types (plain text / mixed language / image input, etc.)├── Reproducibility: Fixed inputs and a fixed scoring rubric, with support for regression testing└── Extensibility: New business scenarios can be added at low cost
Take image generation as an example. A practical task set might include categories such as:
- Text layout and rendering, especially mixed multilingual text
- Transferring a specific artistic style
- Reproducing complex lighting and materials
- Controlling composition
Each category would then hold several concrete test cases, forming a structured tree of cases.
5. Multi-Dimensional Scoring
The most common scoring mistake is collapsing every dimension into a single overall score. A better approach is to score each dimension independently, then apply weights as needed. Common dimensions for text evaluation include:
- Instruction following: Whether the model strictly honors constraints such as word count, format, and role.
- Factual accuracy: Whether the output is correct and free of hallucinations.
- Language fluency: Whether the expression is natural and professional.
- Reasoning depth: Whether the chain of reasoning stays complete on multi-step tasks.
Image generation relies more on subjective quality assessment, typically covering:
- Text rendering accuracy
- Style fidelity
- Realism of materials and lighting
- Composition quality
These dimensions are hard to fully automate. They require human reviewers or cross-review by multiple models, with a unified scoring rubric.
Scoring is especially demanding for high-difficulty cases such as multilingual layout. I once came across a typical extreme case: render text correctly in five different writing systems on a single poster. These were Chinese, English, Japanese, French, and a right-to-left script from the Arabic family. None of the models I tested passed completely, and the highest score barely exceeded a passing mark. The value of such cases lies not in ranking the models, but in revealing the limits of current technology. They make it clear that product requirements involving complex multilingual layout should not be promised blindly at this stage.
6. Cost-Performance Tradeoffs: The Overlooked Dimension
Professional model selection should always include a cost-performance perspective. A simple cost-effectiveness metric can support the decision:
Cost-effectiveness index = Scenario fit score / Cost per callCost per call = Input token price × average input length+ Output token price × average output length
The point of this formula is to put “score” and “money” in the same frame of reference. For example (the numbers are illustrative only), suppose Model A scores 95 but costs nearly 20 times as much per unit as Model B, while Model B’s score is only a fraction of a point lower. In that case, Model B is the rational choice for most cost-sensitive businesses. Conversely, if a business has zero tolerance for quality issues, such as high-stakes scenarios involving compliance or security, paying a premium for marginal quality gains can also be justified.
The key point is that there is no absolute best model, only the best choice under a given set of constraints. Selection becomes objective only when constraints, such as cost ceilings, latency requirements, and availability SLAs, are stated explicitly.
7. Stress Testing: Finding the Capability Ceiling
Beyond routine test cases, a mature evaluation system should include a stress-testing layer that uses extreme inputs to push models to their limits:
- Extra-long context: Feed in content close to or beyond the model’s stated context window limit, and check information recall and consistency.
- Conflicting instructions: Insert contradictory constraints into a normal task to test how robustly the model follows instructions.
- Mixed multilingual and multimodal input: Cases such as multilingual layout and reasoning over combined text and images.
- Adversarial input: Vague, ambiguous inputs that need clarification, to see whether the model asks reasonable follow-up questions instead of answering carelessly.
The purpose of stress testing is risk management. It helps you uncover hidden problems such as models that seem usable in normal conditions but break down in extreme cases. That way, you can avoid failures on long-tail scenarios after launch.
8. Engineering Best Practices
If you want to build your own scenario-based evaluation system, you can follow this implementation path:
- Start with data: Collect logs of real business requests first (with sensitive information removed). These are the raw material for your task set.
- Build an evaluation pipeline: Connect batch model calls, result collection, scoring, and reporting into an automated workflow that supports one-click regression runs.
- Unify the scoring rubric: Whether evaluation is done by humans or automatically, fix the scoring criteria and recalibrate them regularly.
- Create a dashboard: Replace a single ranking with dimension-by-dimension comparison views. This lets the team see at a glance how each model performs on different tasks.
- Keep it updated: Models iterate quickly, so the evaluation system must run continuously rather than as a one-off effort.
- Record decisions: Document the scenario, constraints, and conclusion behind each selection decision. Over time, these become a traceable record of past choices.
When implementing this system in the cloud, you can rely on general-purpose cloud capabilities:
- Object storage to archive test cases and results
- A container platform to run batch evaluation jobs
- A message queue for task scheduling
- Logging services and monitoring alerts to track pipeline health
Baidu AI Cloud, for example, offers general-purpose cloud infrastructure that can serve as the foundation for such an evaluation pipeline. This lets teams focus on the evaluation methodology itself rather than operational overhead.
9. Conclusion
Back to the original question: “Which model is the best?” A better question is: “Under my business scenario, cost, and latency constraints, which model is the best fit?”
Leaderboard scores are a starting point, not the finish line. Real selection expertise comes from translating business requirements into a scenario-based evaluation system that is measurable, reproducible, and sustainable. When your decisions are backed by per-dimension data, a built-in cost perspective, and stress tests that probe the limits, “choosing a model by gut feeling” becomes a thing of the past. That is what technology selection should look like in the age of AI.
