# LLM Selection Guide: How to Evaluate Scenario Fit, Performance, and Cost
Choosing a large language model is not about finding the single best model — it is about searching for the best option under constraints. This guide lays out a practical selection workflow: translating business requirements into measurable technical requirements, scoring candidates across task capability, agent engineering readiness, inference performance and service levels, deployment and data-compliance options, and total cost of ownership. It also covers building a model gateway and abstraction layer so models stay replaceable, running a two-week in-house evaluation loop, and avoiding six common selection mistakes. You will not get a product recommendation here, but you will get a process and a scoring method you can run yourself.
1. First, fix the question: “which model is best” is a non-question
Every time a new foundation model ships, engineering teams get interrupted by the same question: which one should we use?
The question is hard to answer not because information is scarce, but because the way it is asked does not hold up. “Best” has no reference frame. Leaderboard scores are measured on public test sets, while your business runs on a private distribution of tasks. Leaderboards measure single-turn answer accuracy, while a production system cares about end-to-end success rate, P99 latency, and cost per task.
The more useful framing: among today’s leading foundation models, the first tier splits into several families with different strengths, and none is optimal in an absolute sense. Selection is essentially constrained optimization — the constraints are whether data may leave your environment, your latency budget, concurrency scale, and budget ceiling; the objective function is your business metrics.
So this article recommends no specific product. Instead, it gives you a selection process and a scoring method you can run yourself.
2. Step one: translate business needs into measurable technical requirements
The most common starting point for a failed selection is comparing a pile of spec sheets against “I want a smart model.” The correct approach is to produce a requirements card that turns the scenario into measurable metrics.
| Scenario | Typical tasks | Key metrics |
|---|---|---|
| Content generation | Marketing copy, report writing | Human preference win rate, factual error rate |
| Understanding and extraction | Contract elements, invoices, support tickets | Field-level F1, JSON Schema compliance rate |
| Long-document analysis | Cross-analyzing research reports, comparing regulations | Mid-document recall, cross-document consistency |
| Code-related | Logic implementation, refactoring, review | Unit test pass rate, compile pass rate, precision of actionable alerts |
| Multimodal | Screenshot reconstruction, converting diagrams to structure | Layout fidelity, coordinate error, OCR replacement rate |
| Agent tasks | Multi-tool orchestration, long-horizon automation | Task completion rate, average number of steps, failure recovery rate |
The requirements card should also state hard constraints explicitly, for example:
- Data boundary: whether data is allowed to leave the corporate network
- Latency budget: ceiling on time to first token, ceiling on full response
- Throughput scale: peak QPS, daily call volume, total output tokens
- Budget ceiling: acceptable cost per task, monthly total
With this table in hand, the later comparisons have a target to aim at.
3. Five core evaluation dimensions
3.1 Task capability: evaluate against your distribution, not the leaderboard
Code and architecture capability. If your core need is implementing complex business logic, system design, and reviewing an entire codebase, the evaluation focus is not “can it write quicksort” but four things: repository-level context handling (can it follow cross-file dependencies and call chains), the compile-and-run rate of the code it generates, its ability to explain business semantics (can it articulate what the code intends, rather than inventing interfaces that do not exist), and its ability to interpret and close the loop on static-analysis warnings. A practical suggestion: pull 20–50 real issues from your own repository as an evaluation set, and track “first-pass rate” and “human rework hours.”
Long-context capability. A nominal window of hundreds of thousands or even millions of tokens is the entry ticket; it does not mean effective utilization. Engineering-wise, you must distinguish three concepts:
- Nominal window: the maximum input length the API accepts;
- Effective window: the range over which the model still answers detail questions reliably at that length;
- Lossless window: the degree to which information inside that range is not “diluted and forgotten.”
Validate with two test families: “multi-needle retrieval” and “aggregation reasoning.” The first plants unrelated facts at multiple positions in a long document and queries them one by one. The second asks the model to perform statistics, comparisons, and conflict detection across the full text — for example, finding clauses that contradict each other between two documents. Many models do well on single-needle tests and degrade noticeably on mid-document aggregation. If your scenario is cross-analysis of “dozens of research reports plus a full year of meeting notes,” you must specifically test mid-document aggregation, and for very long inputs use “hierarchical summarization plus targeted retrieval” as preprocessing rather than dumping the entire source text in at once.
Native multimodality. The criterion is chain length: if the model accepts screenshots, design files, PDFs, or video directly, you can drop the bolt-on sequence of “format conversion → OCR → layout reconstruction → feed into a language model.” The shorter the chain, the less error accumulates, and the lower the latency and cost. Tasks like converting design mockups into front-end code depend heavily on the model’s understanding of spatial layout and visual hierarchy; bolt-on OCR loses critical information.
3.2 Engineering readiness: what decides whether an agent can ship
Conversational ability is only the tip of the iceberg. What really determines success or failure is the stability of the engineering interfaces:
- Tool calling: adherence to parameter schemas, support for parallel calls, and the ability to self-correct after an error is returned;
- Structured output: whether constrained decoding is supported (for example, JSON Schema constraints), and whether compliance can exceed 99%;
- Long-chain reliability: in tasks longer than ten steps, does the model drift from the goal, retry endlessly, or call the same tool repeatedly;
- Context-management compatibility: whether instruction-following survives summarization and compression, retrieval injection, and history truncation.
A useful offline metric is average number of calls per task: stronger models often finish in fewer rounds, which directly saves real money in token spend. For each agent scenario, track the triple of “task completion rate / average rounds / average cost” rather than looking at completion rate alone.
3.3 Inference performance and service levels: the underrated dimension
On technical specs, distinguish at least four items:
- TTFT (time to first token): drives interaction feel; with streaming, users see content within 300ms;
- TPS (output tokens per second): determines how long it takes to “read a paragraph”;
- Concurrency capacity and rate-limiting policy: does traffic queue at peak, get throttled, and how does the service degrade;
- P99 and the long tail: does it slow down significantly or truncate on complex tasks.
One caution: figures like “this model can exceed 100 tokens per second” in public marketing are typically measured under specific input lengths and concurrency levels, and are not a commitment about your workload. The only trustworthy approach is to load-test with your own input and output length distributions. A sketch of a load-test script:
import time, statisticsfrom concurrent.futures import ThreadPoolExecutorasync def bench(model_client, prompts, concurrency=32, max_out=512):lat = []async def one(p):t0 = time.perf_counter()first = Noneasync for chunk in model_client.stream(p, max_tokens=max_out):if first is None:first = time.perf_counter()total = time.perf_counter() - t0lat.append({"ttft": (first - t0), "e2e": total})sem = asyncio.Semaphore(concurrency)async def guarded(p):async with sem:await one(p)await asyncio.gather(*(guarded(p) for p in prompts))ttft = sorted(x["ttft"] for x in lat)e2e = sorted(x["e2e"] for x in lat)return {"ttft_p50": ttft[len(ttft)//2], "ttft_p99": ttft[int(len(ttft)*0.99)],"e2e_p50": e2e[len(e2e)//2], "e2e_p99": e2e[int(len(e2e)*0.99)],"throughput_rps": len(lat) / max(e2e),}
Land the conclusion on business terms: for a consumer product, conversational latency maps directly to retention, while an internal tool can tolerate a longer response. The performance requirements these two impose on a model are completely different.
3.4 Deployment form and data compliance
Choose a deployment form according to data sensitivity and operational capacity:
| Form | Suitable for | Main cost |
|---|---|---|
| Hosted API | Fast validation, public-facing business | Data leaving your environment, vendor lock-in, unit-price sensitivity |
| Dedicated instance / reserved capacity | Steady throughput, performance isolation | Capacity planning, waste during idle periods |
| Private deployment (open weights) | Confidential material, internal systems, strict compliance | GPU procurement and operations, lagging model updates |
For a need like “organizing confidential internal material,” open-weight models combined with private deployment are almost the only option — data never leaves your environment, and the capability ceiling is set by your own hardware. But be clear about the cost: you take on deploying the inference service, scaling it, monitoring it, and upgrading it. If your team lacks that operational capability, a combination of hosted service plus data masking plus audit trails is usually more economical than forcing a self-built stack.
3.5 Total cost of ownership: compute TCO, not unit price
A cheap unit price does not mean a low cost. The real cost per task should be estimated like this:
cost per task = (input tokens × input price + output tokens × output price) × (1 + retry rate)+ retrieval / tool-calling cost+ human and operational cost allocated to that task
Three frequently overlooked levers:
- Prefix caching: system prompts, knowledge-base chunks, and long documents reused across many requests can substantially cut input-side cost;
- Tiered routing: intent recognition, format normalization, and simple classification go to a small or distilled model; only complex reasoning reaches the flagship model;
- Batching: non-real-time offline work (nightly report generation, bulk labeling) goes through an asynchronous batch queue, which usually comes with a price discount.
4. Combined architecture: don’t “pick one,” pick “a set”
Mature systems treat models as replaceable components rather than welding business logic to a single interface. The core is a model gateway plus an abstraction layer:
from dataclasses import dataclassfrom typing import Protocol@dataclass(frozen=True)class ModelProfile:name: strstrengths: frozenset[str] # {"code","long_ctx","agent","vision","high_qps"}eff_ctx_window: int # measured effective window, not the nominal onettft_p99_ms: inttps_p50: intcost_per_1k_tok: float # blended unit price weighted over input/outputdeploy_mode: str # "api" | "dedicated" | "on_prem"class ChatModel(Protocol):def chat(self, messages: list[dict], **kw) -> str: ...def stream(self, messages: list[dict], **kw): ...def route(task: dict, registry: dict[str, tuple[ModelProfile, ChatModel]]):"""Filter by capability + hard constraints, then optimize for the objective"""need = frozenset(task.get("capabilities", ()))hard = task.get("constraints", {})cands = [(p, c) for p, c in registry.values()if need <= p.strengthsand (hard.get("deploy_mode") in (None, p.deploy_mode))and task.get("in_tokens", 0) <= p.eff_ctx_window]if not cands:raise LookupError(f"no qualified model for {task['name']}")if task.get("latency_sensitive"):return min(cands, key=lambda pc: (pc[0].ttft_p99_ms, pc[0].cost_per_1k_tok))return min(cands, key=lambda pc: (pc[0].cost_per_1k_tok, -pc[0].tps_p50))
The supporting components can all come from standard cloud service categories: object storage for documents and datasets, a message queue for asynchronous batch jobs, a container platform for inference services and elastic scaling, logging with monitoring and alerting for call-chain tracing and dashboards, and an API gateway for unified authentication, rate limiting, and canary traffic splitting. If you build on a platform such as Baidu AI Cloud, these capability components can be combined as needed and paired with the model service to route between models and handle fallback.
This architecture yields two key benefits: canary switchability (split traffic proportionally between two models to compare them) and fallback degradation (when the primary model times out or is throttled, requests automatically switch to a backup model instead of failing).
5. A two-week, closed-loop evaluation you can run in-house
- Days 1–2: collect and mask data. Extract 200–500 real requests from production logs or requirement documents, covering high-frequency scenarios and hard samples (long inputs, ambiguous inputs, multi-constraint inputs), then mask them and freeze them as an evaluation set. Do not rely only on public examples.
- Days 3–4: establish ground truth. For extraction tasks, provide field-level gold labels; for generation tasks, write out scoring rubrics (factuality, completeness, style), and have two reviewers score independently to calibrate agreement.
- Days 5–7: automated batch scoring. Record accuracy, schema compliance rate, average number of rounds, token consumption, and latency distribution, and produce a unified scorecard.
- Days 8–9: load testing. Run concurrency tests against your real input/output length distribution to obtain P50/P99 and throttling thresholds.
- Days 10–12: cost conversion. Factor in retry rate and cache hit rate to compute cost per task for each scenario, and derive a total-cost comparison.
- Days 13–14: shadow traffic and canary rollout. First run shadow traffic in parallel (no user impact), observe for one to two weeks, then shift traffic by proportion.
A scorecard might look like this (values are illustrative):
| Criterion | Weight | Model A | Model B | Model C |
|---|---|---|---|---|
| Task accuracy | 30% | 88 | 84 | 79 |
| Structured output compliance | 15% | 99.1% | 97.4% | 93.0% |
| Agent task completion rate | 20% | 81 | 74 | 66 |
| P99 end-to-end latency | 15% | 2.1s | 1.4s | 1.6s |
| Cost per task | 20% | High | Medium | Low |
The value of this table is that it turns “which one should we pick” from an argument into arithmetic. Different weights reflect different business priorities: a real-time consumer business gives latency more weight, while offline document processing gives cost and long context more weight.
6. Six common mistakes
- Leaderboard absolutism: the correlation between public leaderboards and your private distribution is often lower than expected; leaderboards are only good for an initial shortlist.
- Using general benchmarks instead of business evaluation: general benchmarks test common knowledge, while your business tests private terminology, format conventions, and internal logic.
- Ignoring P99: averages hide the long tail. User churn usually happens on that 1% of requests that time out.
- Overlooking version drift: the default version of a hosted model can be silently upgraded, which may make previously tuned prompts regress. In production, pin the model version explicitly and run the regression evaluation set before upgrading.
- Over-privatizing: self-built inference requires GPU procurement, capacity planning, and on-call operations. When the team is undersized, TCO is often higher than a hosted option.
- Using one model for everything: sending intent recognition to a flagship model is pure waste. Tiered routing is usually the highest-ROI optimization available.
7. Conclusion
Back to the original question: which model should you pick?
The correct answer is a three-step process. First, break the scenario down into requirements cards that state hard constraints and metrics. Second, score candidate models on the same scorecard using your own evaluation set, focusing on task capability, agent engineering readiness, P99 performance, and total cost. Third, preserve replaceability and fallback paths in the architecture, so that selection becomes a decision you can redo at any time rather than a one-off bet.
One point deserves emphasis: the capability differences among today’s leading models are not uniform across tasks. Some are more mature on code and long-chain agent work, some excel at cross-analysis of very long documents, some have an advantage in high-throughput, low-cost concurrency, and some have a shorter chain for direct multimodal input. Your business will land in only one or two of those cells — and only a conclusion tested against your own data is trustworthy.
