LLM Selection in Practice: From Use-Case Matching to Evaluation and Rollout

1. Define the problem first: choose the best fit, not the strongest model

Almost every team building an AI application starts with the same question: which large language model is the strongest right now?

From an engineering perspective, this question is hard to act on. Public leaderboards reflect average performance on general-purpose corpora, while your business only cares about performance on your own small data distribution. A model that scores well on general writing may not reliably output JSON that matches your template; a model with strong reasoning may be impractical for customer service if its unit price and latency are several times higher than another model’s.

A more reliable approach is to turn model selection into an executable pipeline:

task profile -> capability mapping -> candidate model set -> custom evaluation -> cost and stability constraints -> phased rollout

The most important step is the task profile. Before choosing a model, quantify your task:

• What does the input look like? A single-turn short text, dozens of PDF pages, hundreds of conversation turns, or a ticket with attachments?
• What is the output requirement? Natural-language text, or structured data that must strictly conform to a schema?
• What latency is required? Interactive chat usually requires first-byte response within one second, while offline batch processing can tolerate tens of seconds.
• What is the volume? Daily call volume determines cost sensitivity: if the per-request cost differs by 2x, the total bill can differ by an order of magnitude.
• Can the data leave the internal network? This requirement often determines the candidate set.

Answers to these five questions will eliminate most candidates.

2. Seven capability dimensions that affect production results

1. Language and domain coverage

Ability in Chinese, English, and minor languages is not symmetric. Differences in Chinese tone, idioms, industry jargon, and dialect transcription can be much larger than score gaps on general leaderboards. If your business uses specialized terminology, such as in healthcare, legal, finance, or industrial protocols, test with real texts from your domain through a blind evaluation instead of relying on general benchmark scores.

2. Context window and effective recall

Context length is often emphasized in marketing, but engineering teams care more about effective recall. As context grows, models usually pay more attention to the beginning and end of the input than to the middle. If a key clause is buried in the middle of 80,000 tokens, the model may see it but fail to use it. For long-document scenarios, the correct approach is usually retrieval first: split documents, retrieve only relevant passages, and place only those passages in the context instead of stuffing an entire knowledge base into the prompt.

3. Reasoning depth vs. latency

Current model forms can be roughly divided into two categories: conversational models optimized for direct responses, and deep-reasoning models that perform multiple internal reasoning steps before answering. The latter are often more stable for mathematics, multi-constraint logical reasoning, and complex planning, but they usually have higher first-token latency and higher token consumption. Do not simply choose the “strongest” model; tier by task: use lightweight models for simple classification, and upgrade to deep-reasoning modes only for complex reasoning.

4. Instruction following and reliable structured output

This is often underestimated but can be fatal in production. Business systems need output that can be parsed, not text that merely looks like JSON. When evaluating, measure the format-compliance rate: the proportion of responses that can directly pass schema validation. Also classify failures: extra explanation text, missing fields, invalid enum values, or wrong numeric types.

5. Tool calling and agent capabilities

Whether a model can choose the right tool, fill in parameters correctly, and maintain state across multiple turns determines the ceiling of agent applications. To evaluate this, construct a tool set with distractors, such as 15 functionally similar APIs, and observe whether the model selects the wrong tool, calls the same tool repeatedly, or hallucinates parameters.

6. Multimodal understanding

Actual capabilities vary widely across image question answering, table screenshot parsing, OCR-to-structured output for scanned documents, and summarization after speech transcription. Test with your own business samples rather than public sample images, and focus on hard cases such as handwriting, low-resolution images, and complex tables.

7. Engineering and compliance attributes

This category determines whether a solution can go live: whether the service supports private cloud or on-premises deployment, whether data is used for training, whether audit and content-safety capabilities are available, what quota and rate-limiting policies exist, whether the SDK is compatible with common protocols, and whether related capabilities such as fine-tuning and knowledge-base retrieval are provided. Some teams prefer to consolidate model serving, fine-tuning, knowledge-base retrieval, logging, and monitoring in one cloud platform, such as the large-model service platform from Baidu AI Cloud. The benefit is fewer separate authentication systems and billing setups.

3. Matching typical scenarios with model forms

Scenario	Core need	Recommended form	Key metrics
Marketing copy / creative writing	Language quality, controllable style	General conversational model, long output	Human rating, adoption rate
Contract / report review Q&A	Effective recall in long documents, accurate citations	Long context + retrieval augmentation	Answer accuracy, citation hit rate
Code assistant	Completion quality, cross-file understanding	Strong coding ability + long context	Compile pass rate, adoption rate
Smart customer service / ticketing	Stability, low latency, low unit price	Lightweight models first	P95 latency, resolution rate, cost per ticket
Data analysis / query generation	Structured output, SQL correctness	Strong instruction following + execution validation	Execution success rate, result error rate
Agents and workflow orchestration	Tool calling, multi-step planning	Deep-reasoning model	Task completion rate, average steps
Multimodal moderation / understanding	Image and table parsing	Native multimodal model	Field extraction F1, misjudgment rate
Offline batch processing (labeling, translation, summarization)	Throughput, unit price	Lightweight model + batch API	Cost per item, total runtime

A common mistake is assuming that one model can serve all scenarios. A more economical design is two or even three tiers: high-frequency simple requests go to lightweight models, complex requests are upgraded to stronger models, and long-document scenarios use retrieval augmentation consistently.

4. Three engineering techniques that work across models

4.1 Write prompts as executable specifications

A prompt is not a wish; it is an interface contract. A stable structure contains six parts: role, business context, input data, task requirements, constraints, and output format/examples.

SCHEMA = {
    "type": "object",
    "properties": {
        "risk_level": {"enum": ["low", "medium", "high"]},
        "clauses": {"type": "array", "items": {"type": "string"}},
        "reason": {"type": "string"}
    },
    "required": ["risk_level", "clauses", "reason"],
    "additionalProperties": False
}
PROMPT = """You are a contract review assistant.
Business context: {biz_context}
Document to review: <text>{document}</text>
Task: Determine the risk level, list excerpts of risky clauses, and provide the reason.
Constraints:
1. clauses must be exact excerpts from the original text and must not be rewritten;
2. If you cannot determine the answer, state "insufficient information" in reason and do not guess;
3. Output only JSON that conforms to the given schema; do not output any explanatory text.
"""
def review(client, document, biz_context):
    resp = client.chat(
        model="primary-model",
        messages=[{"role": "user", "content": PROMPT.format(
            biz_context=biz_context, document=document)}],
        response_schema=SCHEMA,   # Use the platform's structured-output constraint
        temperature=0.0           # Disable randomness for extraction tasks
    )
    return validate(resp, SCHEMA)  # Local validation; retry on failure

There are three key points: wrap untrusted input in tags to prevent instruction injection; set sampling temperature to 0 for extraction tasks to make results reproducible; validate the response against the schema locally after the model returns, and retry or degrade automatically if validation fails. Do not rely entirely on the model for correctness.

4.2 Use retrieval to add knowledge instead of stuffing knowledge into prompts

For knowledge that changes, is large in volume, or requires traceability, such as policies, prices, ticket history, and private documents, do not rely on the model’s parametric memory. The standard process is: split documents semantically rather than by fixed character counts; combine vector retrieval with keyword retrieval; rerank and select the top-K passages; concatenate citations into the prompt; and instruct the model to answer only based on the provided passages and explicitly say “I don’t know” when evidence is missing. This can significantly reduce both hallucination and token cost.

4.3 Layered routing: assign models by task difficulty

def route(task):
    if task.est_tokens > 20000 or task.need_reasoning:
        return "deep-reasoning-model"     # Complex reasoning and long documents
    if task.category in ("chitchat", "faq"):
        return "light-model"              # High-frequency simple requests
    return "general-model"
def call_with_fallback(task, max_retry=2):
    for model in [route(task), "backup-model"]:
        for attempt in range(max_retry + 1):
            try:
                return invoke(model, task)
            except (Timeout, RateLimit) as e:
                if attempt == max_retry:
                    break
                sleep(backoff(attempt))   # Exponential backoff with jitter
    return cached_or_degraded_answer(task)

Routing, fallback, and retry with backoff improve stability more directly than simply switching to a stronger model.

5. Build your own evaluation set: the real dividing line for long-term efficiency

Model selection without an evaluation set is essentially luck. Start building one from day one:

1. Sampling: extract 200 to 500 real requests from historical logs or human-written Q&A, covering high-frequency, long-tail, boundary, and adversarial cases.
2. Annotation: provide reference answers and scoring criteria, and distinguish “must-have” from “nice-to-have” requirements.
3. Scoring: combine three layers: rule-based validation (format, enum values, numeric ranges), automatic judging (use a strong model to score according to a rubric), and human spot checks (10% to 20%, especially for open-ended questions where automatic judging is unreliable).
4. Metrics: task success rate, format-compliance rate, hallucination rate, P50/P95 latency, token consumption per request, and cost.

def evaluate(cases, runner):
    report = defaultdict(list)
    for c in cases:
        out = runner(c.input)
        report["format"].append(schema_ok(out, c.schema))
        report["accuracy"].append(judge(out, c.golden, rubric=c.rubric))
        report["latency_ms"].append(out.latency_ms)
        report["cost"].append(out.token_in * P_IN + out.token_out * P_OUT)
    return {k: mean(v) for k, v in report.items()}

With this baseline, model upgrades, prompt changes, and provider switches become measurable changes rather than arbitrary decisions.

6. Treat the model as a replaceable component in the architecture

Different platforms have different SDKs, authentication methods, and field names. If business code directly depends on one provider’s interface, migration can be very costly. The correct approach is to add a unified gateway that provides stable semantics upward and adapts to multiple implementations downward:

class LLMGateway:
    """Hide differences between model services from the application; models can be swapped in configuration."""
    def chat(self, model_alias, messages, **kw):
        cfg = self.registry[model_alias]      # Alias -> actual model and endpoint
        payload = normalize(messages, kw)      # Unify message and parameter formats
        return self.adapters[cfg.provider].chat(cfg.endpoint, payload)

At the same time, add several production necessities: propagate a request-level traceId through logs, maintain token usage and cost dashboards, use canary traffic splitting to send a percentage of traffic to a new model and compare metrics, and implement content safety plus fallback for sensitive content.

7. Cost and stability governance checklist

• Semantic caching: cache hits for frequent repeated questions can eliminate most calls; cache keys should include model version and prompt version.
• Context compression: remove redundant history, summarize long conversations, and send only retrieved passages.
• Batch and async: offline tasks should use batch APIs, whose unit price is usually significantly lower than real-time APIs.
• Streaming output: improves user experience and reduces timeout risk for long outputs.
• Rate limiting and quotas: use a token bucket on the client side to avoid cascading failures after upstream rate limiting.
• Observability: track success rate, latency distribution, and cost by model dimension, and use these data for quarterly reviews.

8. Rollout rhythm: three phases, and do not do everything at once

• Phase 1 (about 2 weeks): POC. Use 50 to 100 real samples to validate feasibility and format stability for a single task, and determine whether it can be done.
• Phase 2 (1 to 2 months): pilot. Integrate the unified gateway, retrieval augmentation, evaluation set, and monitoring; run a closed loop in a small-traffic real scenario; and determine the cost model.
• Phase 3: scale. Implement multi-model routing, caching, and batch-processing optimization; build fine-tuning or knowledge-base capabilities; run canary and disaster-recovery drills; and set a quarterly review mechanism. Model capabilities iterate quickly, so conclusions from today may change in a few months.

9. Conclusion

The question “which model is the strongest” is not meaningful. The meaningful question is “which model performs best for my task, cost, and compliance constraints.” Break model selection into five steps: task profiling, capability dimensions, scenario matching, custom evaluation, and architectural decoupling. When a new model appears, you only need to run your own evaluation set to reach a conclusion within a day. That is the real moat of AI application engineering.