LLM Selection for Applications: Scenario-Driven Evaluation Methods and Practices

    This article explains how to choose an LLM for an application through scenario-based evaluation rather than public leaderboard scores. It covers task decomposition, internal evaluation sets, context-length validation, latency and stability, cost modeling, security and compliance, and integration design. It also provides a reusable evaluation workflow, common pitfalls, and an implementation checklist.

    1. Why Selection Deserves a Separate Process

    Public LLM leaderboards update quickly, but the relationship between leaderboard scores and business outcomes is not stable. A model that performs well in general Q&A may fail noticeably in contract-clause extraction or internal knowledge-base Q&A. Selection should therefore start from the business task and be validated with your own data, rather than relying only on rankings.

    Selection also has to balance three types of constraints: quality requirements, cost budget, and compliance boundaries. These often pull against one another. The highest-quality model is not necessarily the cheapest per unit, and the fastest-responding model is not necessarily the best fit for long-document processing.

    2. Break the Scenario into Tasks

    Once requirements are broken into concrete tasks, each task type needs different evaluation criteria. Common categories include:

    • Open-ended Q&A: focus on factual accuracy and answer completeness.
    • Text generation and rewriting: focus on language quality, style consistency, and instruction following.
    • Code generation and repair: focus on whether the generated code passes unit tests, not whether it merely looks reasonable.
    • Long-document understanding: focus on information localization, cross-paragraph reasoning, and citation accuracy.
    • Voice or multi-turn interaction: focus on response latency, interruption handling, and context retention.

    After decomposition, each task type should define clear inputs, expected outputs, and a scoring method. An evaluation without scoring criteria produces impressions, not conclusions.

    3. Core Evaluation Dimensions

    3.1 Task Quality: Build an Internal Evaluation Set

    Public benchmarks can only reflect general capabilities. It is advisable to sample 200 to 500 examples from real business scenarios, covering high-frequency cases and edge cases, and to annotate each example with a reference answer or scoring rule. Scoring methods fall into two categories: automatic scoring, such as regex matching or unit-test pass rate; and human scoring, such as a five-point scale. For highly subjective tasks, at least two reviewers should score independently, and inter-rater agreement should be calculated to rule out bias caused by inconsistent standards.

    3.2 Context Length: Distinguish Nominal from Effective Utilization

    The context window is the maximum input length a model can receive in a single call, usually measured in tokens. A large nominal window does not mean that information at every position in a long text can be used accurately. A common phenomenon in practice is that information at the beginning and end of the text is used better, while information in the middle is easily missed. A validation method is to randomly embed key facts at different positions in a long text and measure the model’s recall at each position, thereby obtaining the true effective length.

    Chinese and English have different character-to-token ratios; Chinese documents usually consume more tokens. When estimating capacity and cost, calculate directly from the tokenizer output of the target model rather than using a rough character-count conversion.

    3.3 Interaction Experience: Latency and Stability

    The response speed perceived by users is mainly determined by two metrics: time to first token, i.e., the time from sending a request to receiving the first output token; and generation speed, i.e., the number of output tokens per second. Streaming output can significantly improve perceived speed even when total latency remains unchanged.

    Stability is equally important. Timeout rate, rate-limit error rate, and server error rate should be recorded, and behavior during peak hours should be observed. For online products, it is advisable to set timeout thresholds and fallback strategies, such as switching to a backup model when the primary model call fails, or returning cached results within an acceptable range.

    3.4 Cost Model: Estimate by Call Structure

    Most usage-based services price input tokens and output tokens separately, and output tokens usually cost more. At the same unit price, total cost can differ by several times between long-output and short-output scenarios.

    The example below uses hypothetical numbers and does not represent any real pricing:

    Assume each request averages 2,000 input tokens and 500 output tokens, with 100,000 requests per day. The input price is CNY 2 per million tokens, and the output price is CNY 8 per million tokens. Then:

    • Input cost: 2,000 × 100,000 ÷ 1,000,000 × 2 = CNY 400/day
    • Output cost: 500 × 100,000 ÷ 1,000,000 × 8 = CNY 400/day
    • Total: CNY 800/day

    A realistic estimate should also include retries, tokens consumed by system prompts, and discounts from cache hits.

    3.5 Data Security and Compliance

    Before selection, three questions need to be clarified: whether input data will be used for model training, in which region data is stored, and whether industry regulatory requirements are met. When personal information, trade secrets, or regulated data are involved, priority should be given to private deployment or to services with explicit data isolation clauses. Relevant terms should be based on the provider’s formal agreement, not on oral statements or marketing materials.

    3.6 Integration and Replaceability

    A sound architecture encapsulates model calls behind a unified interface layer. Business code depends only on internal interfaces and is not directly coupled to a specific provider’s SDK. In this way, switching costs remain controllable when model quality changes, prices adjust, or a service is interrupted.

    4. Build a Reusable Evaluation Workflow

    The code below shows an evaluation framework that is independent of any specific provider. The core idea is to abstract model calls as functions and compare multiple candidate models using the same sample set and the same scoring logic.

    1. import time
    2. from dataclasses import dataclass
    3. from statistics import mean
    4. from typing import Callable
    5. @dataclass
    6. class Sample:
    7. prompt: str
    8. expected: str
    9. def exact_match(output: str, expected: str) -> bool:
    10. return output.strip() == expected.strip()
    11. def evaluate(model_fn: Callable[[str], str], samples: list[Sample]) -> dict:
    12. correct, latencies = 0, []
    13. for s in samples:
    14. start = time.perf_counter()
    15. output = model_fn(s.prompt)
    16. latencies.append(time.perf_counter() - start)
    17. if exact_match(output, s.expected):
    18. correct += 1
    19. return {
    20. "accuracy": correct / len(samples),
    21. "avg_latency_s": mean(latencies),
    22. }

    To use it, write an adapter function for each candidate model, wrap it as Callable[[str], str], and run evaluate on the same sample set. For open-ended tasks, replace exact_match with a human scoring function or a rule-based scoring function. It is recommended to record the sample version, model version, and inference parameters together so results can be reproduced.

    5. Common Pitfalls

    1. Looking only at public leaderboard rankings. Leaderboards have limited correlation with business data and may be subject to leaderboard-specific optimization.
    2. Drawing conclusions from a small number of samples. Differences among a few dozen samples may be random fluctuation; use statistical tests or a sufficient sample size.
    3. Ignoring output length. Longer outputs increase cost and latency but do not necessarily produce better results.
    4. Testing only normal inputs. Add edge-case samples such as ambiguous instructions, malformed formats, and sensitive content.
    5. Treating personal trial impressions as overall conclusions. Trials are useful for discovering issues but cannot replace systematic evaluation.

    6. Implementation Checklist

    • Business tasks have been decomposed, and each task type has clear scoring criteria.
    • The internal evaluation set covers high-frequency scenarios and edge cases.
    • Long-context performance has been validated at different positions in the text.
    • Cost has been estimated from the real token distribution and includes retries and caching.
    • Data storage region, training-use terms, and compliance requirements have been confirmed.
    • Model calls are encapsulated behind a unified interface and support switching.
    • Timeout, rate-limit, and fallback strategies have been configured.

    Selection is not a one-time decision. It is advisable to rerun the same evaluation set every quarter or whenever model versions are updated, and to adjust the approach based on the results.