LLM Capability Comparison and Scenario-Based Model Selection

1. “Which model is the strongest?” is the wrong question

Almost every model selection discussion eventually drifts toward “which model is the strongest.” In an engineering context, that question has no useful answer for three reasons.

Capability is a distribution, not a scalar. A model may perform well on structured extraction but struggle with long-document reasoning. It may follow instructions strictly but be overly conservative in refusals. A single score hides differences that are directly relevant to a specific workload.

Public benchmarks do not match business traffic. Public evaluation sets often differ from real traffic in question style, domain distribution, and preferred answer length. Training-data overlap with benchmarks can also make leaderboard scores systematically higher than online performance. A common pattern is that two models with close leaderboard scores can differ significantly on a company’s own samples, and the direction of the gap may be opposite to what the leaderboard suggests.

The evaluation target is not only “answer quality.” Time-to-first-token, cost per request, concurrent throughput, context length, structured-output stability, data residency, and compliance requirements are all hard constraints. In production, a model with slightly lower quality but one-fifth the cost, one-third the latency, and compliance with data-residency requirements is often the better choice.

The actionable question is therefore not “which model is the strongest,” but: For my task distribution, which combination of capabilities can complete the work at acceptable quality, latency, and cost?

2. Break “what it is good at” into six measurable dimensions

To compare models seriously, vague notions of “smartness” must be broken into observable and measurable dimensions. The following six dimensions cover most business selection decisions.

2.1 Instruction following and context understanding

This dimension measures execution precision under constraints, not raw knowledge. Evaluate whether the model can follow word limits, output formats, tone requirements, banned-word lists, and role settings, and whether it maintains those constraints across multiple turns.

Do not ask only “Is this well written?” Use hard constraints that can be judged automatically. For example, require output to be a fixed key-value structure, require exactly three points, or require a specific opening sentence pattern. These checks are binary and are well suited for automated pipelines.

For Chinese-language or mixed-language scenarios, pay extra attention to two issues: first, intent recovery when input contains omitted subjects, inverted order, or colloquial references; second, understanding of domain-specific abbreviations or industry jargon. Results measured with polished prompts are usually better than results measured with real-user input, so evaluation sets should include some “messy input.”

2.2 Long context and information utilization

Context-window size is a nominal value. The real thing to test is long-context utilization: if key information is placed at token 60,000 of an 80,000-token document, can the model recall it accurately?

Common failure modes include loss of information in the middle of a document, incorrect source selection when multiple documents conflict, and distraction by irrelevant instructions inside the document.

A practical evaluation method is a variant of the “needle in a haystack” test: insert 5 to 10 scattered facts into a long document, place some near the beginning, some in the middle, and some near the end, and ask the model to answer each one and provide its location. Add distractors by inserting passages that appear relevant but are actually invalid, and check whether the model fabricates information. This dimension is decisive for applications such as contract review, log troubleshooting, research-report summarization, and code-base question answering.

2.3 Code and tool calling

Code capability should be evaluated as two separate items, because they are often confused:

• Code generation and repair: given a defect description and context, produce a runnable patch. Evaluation should be based on execution results, not text similarity. Put the generated code into a sandbox, run unit tests, and measure pass rate.
• Tool calling and parameter construction: determine whether the model can choose the right tool based on tool descriptions, fill in parameters correctly, chain multiple steps properly, and self-correct after a tool error. Evaluation should include trap scenarios: missing parameters, empty tool results, and dependencies such as “call A before B.”

For many engineering systems, the second item affects user experience more than the first. It determines the reliability of agent workflows, and the quality of tool descriptions and parameter schemas can matter even more than the model itself.

2.4 Multimodal capability

If the business involves images, scanned documents, audio, or video, evaluate each modality separately: document OCR and layout restoration, such as whether table structure is preserved; chart reading, especially numerical accuracy; factual consistency of image descriptions, especially whether the model invents nonexistent elements; and speech recognition performance under specific accents and noise conditions.

The key practical point is: multimodal failure modes differ from pure-text failure modes and are harder for users to notice. For example, chart-reading errors often produce plausible but incorrect values. Quantitative validation with labeled samples is necessary.

2.5 Engineering cost and latency

Cost should be calculated end-to-end, not only by unit price:

Single request cost = input token count * input price
                     + output token count * output price
                     + retrieval/tool-call overhead
                     + expected retry overhead

Output tokens are usually the main cost driver because generation cannot be parallelized in the same way as input processing. Therefore, “make the model write shorter” often saves more money than switching models.

Common cost-reduction methods include prompt-prefix caching, asynchronous batch processing channels, using a small model for initial filtering before handing work to a larger model, and result caching or semantic deduplication.

Latency should be separated into time-to-first-token, or TTFT, and total completion time. Conversational products are sensitive to TTFT, while offline batch processing is sensitive to total duration. Their optimization directions are different: the former benefits from streaming output and shorter input, while the latter benefits from concurrency and batch discounts.

2.6 Compliance, deployment model, and stability

This includes whether data is used for model training, log-retention policies, support for private or dedicated deployment, service-level agreements, and version-change policies. This dimension is often a veto item. In finance, healthcare, and government scenarios, a solution that does not meet data-residency requirements cannot be used regardless of quality.

Also pay attention to model-version update cadence. Silent upgrades can cause stable prompts to drift in behavior. Contracts should clarify version locking and change-notification mechanisms.

3. Three capability profiles and their applicable boundaries

Without focusing on specific brands, mainstream general-purpose LLMs can be roughly grouped into three capability profiles. This is an engineering abstraction for organizing decisions, not a ranking of quality.

Profile A: Balanced general-purpose. Strengths include no obvious weak area, stable instruction following, natural expression, relatively mature tool and plugin ecosystems, and low learning cost. Suitable boundaries include consumer-facing question answering and writing, internal knowledge-base assistants, customer service, general orchestration of multi-step workflows, and exploratory projects that need rapid validation. The risk is that it may be weaker than more specialized models on deeply professional tasks.

Profile B: Long-chain reasoning and rigor-oriented. Strengths include stronger long-document processing, multi-step logical derivation, code refactoring, and consistency in structured output. This profile usually comes with longer reasoning time and higher per-request cost. Suitable boundaries include contract and financial-report review, repository-level code changes, complex data analysis, and scenarios that require traceable reasoning. The risks are cost and latency, as well as reduced cost-effectiveness on small tasks due to “overthinking.”

Profile C: Lightweight and high-concurrency-oriented. Strengths include fast response and low unit cost, making it suitable for large-volume, low-value-per-task workloads. Suitable boundaries include content classification and labeling, extraction and formatting, batch translation and summarization, and acting as the first filter in a routing architecture. The risks are insufficient complex reasoning and long-context capability, which require strict task boundaries and fallback mechanisms.

Most mature systems do not use a single profile. They combine all three.

4. Mapping scenarios to capabilities

Business scenario	Primary capability	Secondary capability	Suggested strategy
Intelligent customer service	Instruction following, tone control	Latency, cost	Lightweight model + retrieval augmentation + fallback to human agents for sensitive issues
Code assistant	Code execution pass rate	Tool calling, context length	Lightweight model for completion; reasoning model for refactoring and troubleshooting
Contract/research-report review	Long-context utilization	Traceable citations	Reasoning model + forced output of location information
Data extraction and labeling	Structured-output stability	Throughput, unit price	Lightweight model for batch processing + validation layer
Marketing and content production	Chinese expression, style transfer	Creative diversity	General-purpose model + human review
Chart/document recognition	Multimodal accuracy	Layout restoration	Prefer dedicated vision models; use general multimodal models as fallback
Data query and analysis	Tool calling, SQL-generation correctness	Result validation	Enforce read-only permissions + validate result row counts

The value of this table is that it downgrades “model selection” to “capability-combination selection,” and capability combinations can usually be assembled from existing models first.

5. Replace leaderboards with your own evaluation set

The only trustworthy comparison is the comparison made on your own data. A practical implementation has the following steps.

5.1 Build a golden set

Sample 100 to 300 cases from real traffic, covering high-frequency scenarios at about 60%, edge scenarios at about 30%, and adversarial scenarios at about 10%. Each sample should include the input, a criterion for judging the expected output, and the type of automated checker.

Prefer programmable validation: regular expressions, key-value structure comparison, numerical tolerance, and unit-test execution.

Adversarial samples are especially important. Examples include questions beyond the model’s knowledge range, to test whether it fabricates answers; instructions containing contradictory constraints, to test whether it asks for clarification; and prompt-injection attempts, to test whether it leaks system prompts.

5.2 Use a unified abstraction layer

Do not couple business code directly to one vendor’s API. Isolate differences with a thin abstraction layer so that switching models becomes a configuration-level change:

class LLMProvider:
    """Unified abstraction for LLM calls; hides protocol differences across providers."""
    name: str
    def chat(self, messages, *, temperature=0.2,
             max_tokens=1024, timeout=30) -> ChatResult:
        raise NotImplementedError
    def supports(self, capability: str) -> bool:
        """Declare capability flags for the routing layer."""
        raise NotImplementedError
class ChatResult:
    def __init__(self, text, usage, latency_ms, finish_reason):
        self.text = text
        self.usage = usage          # Input/output token counts
        self.latency_ms = latency_ms
        self.finish_reason = finish_reason  # stop / length / error

The abstraction layer should unify three things that are easy to overlook: token accounting, since different providers use different tokenizers and usage must be recorded separately rather than estimated; error semantics, since rate limits, timeouts, and content blocking require completely different retry strategies; and structured output, which should go through a unified schema-validation process rather than relying on vendor-specific syntax features.

5.3 Run layered evaluation and regression

def run_eval(provider, cases):
    passed, total_cost, p95 = 0, 0.0, []
    for case in cases:
        r = provider.chat(case["messages"])
        total_cost += estimate_cost(provider, r.usage)
        p95.append(r.latency_ms)
        ok = True
        for checker in case["checkers"]:
            if not checker(r.text):
                ok = False
                break
        passed += int(ok)
    return {
        "provider": provider.name,
        "pass_rate": passed / len(cases),
        "cost_per_case": total_cost / len(cases),
        "p95_latency_ms": sorted(p95)[int(len(p95) * 0.95)],
    }

A key practice is to integrate the evaluation set into continuous integration. Run full regression whenever prompts change, model versions are upgraded, or retrieval strategies are adjusted. In practice, it is not uncommon for a seemingly harmless prompt optimization to reduce pass rate on a certain class of samples by more than ten percentage points. Manual testing often fails to detect this kind of localized degradation.

6. Put quality, latency, and cost into the same objective function

The three metrics cannot be optimized separately. A practical approach is to first determine the business tolerance thresholds, then work backward to acceptable options.

def is_acceptable(result, sla):
    return (result["pass_rate"] >= sla["min_pass_rate"]
            and result["p95_latency_ms"] <= sla["max_latency_ms"]
            and result["cost_per_case"] <= sla["max_cost"])
# First filter out unacceptable options, then choose the lowest cost among the rest.
candidates = [r for r in results if is_acceptable(r, sla)]
chosen = min(candidates, key=lambda r: r["cost_per_case"]) if candidates else None

The order of “filter first, then minimize cost” matters. If the process is reversed, sorting by cost first and checking quality later, the result is often an option that fails quality requirements, because advantages in the cost dimension can be over-weighted.

Two other levers are often underestimated: prompt-prefix caching can significantly reduce input cost and TTFT in scenarios with long system prompts; and semantic caching often has a high hit rate in high-frequency repeated scenarios such as customer service and FAQ, making it one of the highest-return optimizations.

7. Multi-model routing: more realistic than choosing one model

A common production pattern is layered routing: first handle the task with a lightweight model, then decide whether to escalate based on confidence or validation results.

def route(task, budget):
    # 1. Deterministic tasks with clear formats and automatic checks go to the lightweight model.
    if task["kind"] in {"classify", "extract", "translate"}:
        return "lightweight"
    # 2. Complex reasoning tasks go directly to the reasoning model to avoid ineffective downgrade retries.
    if task["kind"] in {"root_cause", "contract_review", "refactor"}:
        return "reasoning"
    # 3. Fallback: upgrade and retry when the lightweight model output fails validation.
    return ("lightweight", "reasoning")
def with_fallback(task, providers, validator):
    chain = route(task, budget=None)
    chain = chain if isinstance(chain, tuple) else (chain,)
    last_err = None
    for name in chain:
        try:
            out = providers[name].chat(task["messages"])
            if validator(out.text):
                return out, name
        except Exception as e:      # Decide retry policy after distinguishing rate limits/timeouts.
            last_err = e
    raise RuntimeError(f"all providers failed: {last_err}")

The benefit of routing is not only cost savings. It removes single-point dependency: if one provider experiences rate limiting or a regional outage, traffic can automatically switch to a backup. It also makes capability evolution gradual. When a new model is introduced, it can first take 5% of traffic for shadow comparison, and traffic can be increased after metrics confirm no regression.

Pay attention to observability cost. Record which path each request takes, why it was escalated, and how much quality improved after escalation. Otherwise, after a few weeks, no one will be able to tell whether routing rules are still effective.

8. Five common pitfalls

Pitfall 1: treating leaderboard scores as business metrics. Leaderboards measure average performance on a specific distribution. Businesses care about worst-case performance on their own distribution. A severe hallucination can wipe out the benefit of thousands of correct answers.

Pitfall 2: assuming longer context is always better. Long context increases cost and latency linearly or even superlinearly, and middle-context utilization often decreases. In most scenarios, the better approach is retrieval augmentation: send only relevant fragments into the context and reserve the window for content that truly needs it.

Pitfall 3: ignoring the cost impact of output length. Output tokens are a major cost item, and generation is hard to parallelize. Explicitly constraining length in prompts often saves more than switching models.

Pitfall 4: assuming model versions are stable. Silent upgrades can change the output distribution. Lock versions, subscribe to change notifications, and make regression evaluation a prerequisite for upgrades.

Pitfall 5: treating prompts as one-off assets. Prompts require version management, change review, and A/B validation. Their rigor should be similar to code. Production prompts scattered across hard-coded locations are a major source of long-term maintenance cost.

9. Compliance and security as upfront constraints

Before running evaluations, confirm hard constraints to avoid discovering after selection that a solution cannot be used. These include whether data will be used for model training, log-retention periods and access permissions, whether data can remain within a specific region, content-safety filtering policies, and ownership of generated content.

Two engineering practices are also required. First, prompt-injection protection: strictly isolate external content, such as web pages, documents, and user-uploaded files, from system instructions; follow least privilege for tool calls; and require human confirmation for write operations. Second, an output-validation layer: validate structured output against schemas, apply rule-based checks to results involving numbers, amounts, and dates, and keep human review for high-risk scenarios. Generative model output is probabilistic and should not be the sole basis for decisions.

10. Conclusion

The answer to “what is a model good at?” will keep changing as versions iterate, but the evaluation method does not. The reusable process can be summarized in four steps:

1. Break requirements into capability dimensions. Identify hard constraints that are veto items, such as compliance, data residency, and context length, and soft indicators that can be traded off, such as expression style and creativity.
2. Build your own evaluation set. Replace subjective judgment with programmable validation, and cover high-frequency, edge, and adversarial samples.
3. Use a unified abstraction layer. Make models replaceable configuration items rather than code-level dependencies.
4. Adopt routing and regression. Use layered scheduling to balance quality, cost, and latency, and use continuous evaluation to prevent silent degradation.

For teams that need a unified platform to complete these validation workflows, Baidu AI Cloud’s large language model service platform can support unified access to multiple models, evaluation, and gray release. It can also be combined with common cloud capabilities such as object storage, message queues, logging, monitoring, and alerting to build a complete evaluation and observability pipeline.

Model selection is essentially a continuously running data-driven process, not a one-time decision. Once the evaluation pipeline is in place, the question “which model should we use?” will answer itself and will keep updating as the business changes.