How to Choose an LLM: A Four-Dimension Evaluation and Cost-Decision Framework
This article lays out a practical framework for selecting large language models for production use instead of ranking models on a single axis. It shows how to build a real cost model (per-token pricing, input/output price ratios, effective context, hidden costs), how to translate vague "capabilities" into six measurable engineering metrics such as structured-output success rate, how to run a repeatable five-step selection and rollout process with a weighted scorecard, and how to cut cost at the architecture layer through tiered routing, two-level caching, context engineering, output constraints, batch/async processing, and multi-vendor fallback. It closes with common pitfalls and a scenario-to-configuration reference table.
1. Fixing the question first: “which model is the strongest” is unanswerable
Most model-selection discussions are inefficient because they try to rank candidates on a single axis: who scores higher on some public leaderboard, or whose price list has the smaller number. A live system, however, faces a set of constraints — latency budget, per-call cost, structured-output success rate, context-length requirements, data-compliance boundaries — and the model is just one variable inside that constraint system.
The right question to ask is this: given a specific scenario and traffic distribution, which model minimizes the cost per unit of usable output while staying above the quality threshold? Two easily overlooked keywords sit inside that sentence:
- A given scenario — quality is scenario-dependent; there is no cross-scenario “total score” you can compare.
- Usable output — an answer that was generated but needs to be rewritten by a human does not cost one API call; it costs one API call + one round of human work + one round of rework.
The rest of this article follows four steps: do the math → decompose capabilities → build a process → fix the architecture.
2. Step 1: Get the cost model right
2.1 Unit price is not your bill
Billing for mainstream LLM APIs is generally per token, with separate rates for input and output, and some platforms additionally offer context-cache hit pricing, batch-job discounts, and tiered pricing. Looking only at “how much per million input tokens” is a classic cognitive trap, because the output price is often several times the input price.
Per-request cost can be written as:
C_req = L_in * P_in + L_out * P_outC_month = Q * (avg(L_in) * P_in + avg(L_out) * P_out) * (1 + r_retry)
Here L_in and L_out are input and output token counts, P_in and P_out the corresponding unit prices, Q the monthly request volume, and r_retry the extra-call ratio caused by retries and fallbacks (routinely underestimated in engineering practice; 10%–30% is not unusual).
Call P_out / P_in the price spread ratio. Its effect on the choice of model is decisive:
| Scenario type | Input/output ratio | Dominant cost term | What to prioritize |
|---|---|---|---|
| Long-document understanding, RAG Q&A | Input far exceeds output | Input price + context length | Long-context price/performance, cache hits |
| Content generation, rewriting, marketing copy | Input and output comparable, or output larger | Output price | Output unit price, fluency and naturalness |
| Intent recognition, classification, tagging | Small input, tiny output | Input price | A low-tier small model is enough |
| Agent multi-turn tool calls | Accumulating input across turns | Input price × number of turns | Instruction-following stability, structured output |
This table also explains why “the expensive model is not necessarily expensive”: a model with mid-tier unit pricing that answers correctly on the first attempt can have a far lower total cost than a cheap model that needs multiple rounds of rework.
2.2 Context length is capacity that can be wasted
A nominal context window — whether 128K, 200K, or 1M — is a ceiling, not usable capacity. What matters in practice is the effective context:
- Mid-context degradation: retrieved passages placed in the middle of a long prompt are typically used less reliably than passages placed at the beginning or end. This is a widely known phenomenon in long-context applications.
- Attention dilution and instruction loss: once the system prompt and user instructions are surrounded by large amounts of retrieved content, adherence to format constraints (for example, “output only JSON”) drops.
- Cost grows linearly with context: a call that stuffs in 80K tokens can cost several times more than one that first reranks for relevance and keeps only 6K tokens.
Conclusion: treat the context window as a budget, not a resource, and spend it deliberately through retrieval reranking, passage deduplication, and summary compression.
2.3 Hidden-cost checklist
At minimum, total cost of ownership should include: building evaluation sets, the labor of iterating on prompts, retries caused by structured-output parse failures, queuing and fallback after rate limits are hit, storage for logs and observability data, and the regression cost introduced by model version upgrades — the most expensive item in long-term operation.
3. Step 2: Decompose “capability” into measurable engineering metrics
Selection should not rely on marketing adjectives. It should translate capability into measurable metrics. A scorecard built from the following six dimensions is a reasonable starting point.
1. Language naturalness and style controllability
Measure this indirectly with negative indicators of “AI-flavored” text: sentence-pattern repetition rate, frequency of stock phrases, and uniformity of paragraph structure. For content-creation scenarios, human blind evaluation is essential — automated metrics such as perplexity correlate only weakly with how a human actually reads the text.
2. Instruction following and structured-output success rate
This is the hardest metric in engineering terms. Run stress tests with a fixed schema and measure the parse success rate:
import json, redef parse_json_strict(text: str):# Try a direct parse first; otherwise extract the first balanced JSON objecttry:return json.loads(text)except json.JSONDecodeError:m = re.search(r"\{.*\}", text, flags=re.S)return json.loads(m.group(0)) if m else Nonedef structured_pass_rate(client, cases, schema_hint: str, n=3):ok = 0for case in cases:for _ in range(n): # Sample multiple times to observe stability, not a one-off successout = client.generate(system="Output JSON only, with no explanatory text.",user=case.prompt + "\n" + schema_hint,temperature=0.0,)if parse_json_strict(out) is not None:ok += 1return ok / (len(cases) * n)
A model whose success rate at temperature=0 falls below 98% should not be placed in any pipeline whose output is consumed by a machine.
3. Multi-step reasoning and tool-call reliability
What you are evaluating is not “can it do math problems,” but accuracy of parameter extraction across multi-turn tool calls, whether it can self-correct after a failure, and whether it tends to hallucinate tool names that do not exist. Replay 20–50 real tool-chain traces and measure the end-to-end task completion rate.
4. Code ability
Split this into three separately measured sub-items: completion-level (local correctness), generation-level (does it pass unit tests on the first attempt), and maintenance-level (can it modify existing code inside a long context without breaking other parts). Judging only on generation-level seriously overestimates how usable a model will be in real engineering work.
5. Multimodal ability
Capabilities such as speech synthesis, image generation, and video generation are usually better handled by specialized models than by expecting a general-purpose chat model to cover them incidentally. The decision criterion is: does the modality have its own independent quality metrics (for example, naturalness and phoneme-duration accuracy for speech) and its own cost structure? If so, it should be a separate technical selection rather than being bound to the main model.
6. Engineering and compliance attributes
Throughput and concurrency quotas, rate-limiting policy, availability commitments, SDK and documentation maturity, support for batch or async invocation, logging and audit capabilities, whether data is used for training, and data residency region. Major cloud vendors generally expose multi-model access and operations through model-as-a-service platforms (for example, the model services offered by Baidu AI Cloud). A unified gateway of this kind has one practical engineering benefit: switching models becomes a configuration change instead of a code change, which makes continuous A/B testing and canary rollout possible.
4. Step 3: Build a reusable selection process
4.1 The five-step method
- Inventory the scenarios: list every LLM call site (summarization, classification, extraction, generation, code completion) and annotate each with monthly call volume, average input/output length, latency budget, and the quality red line.
- Build your own evaluation set: 50–150 cases per scenario, covering the real distribution plus edge cases (very long input, empty input, adversarial input). This has the highest return on investment of anything in the selection process, and it is reusable for every subsequent version upgrade.
- Blind dual evaluation: hide model identity, have humans score outputs (a 1–5 scale is recommended), and record automated metrics alongside (parse success rate, format-compliance rate, average output length).
- Cost stress test: plug the real traffic distribution from step 1 into the cost formula and compute cost per usable output = total cost / number of results that pass the quality threshold. This metric is far more meaningful than unit price.
- Canary rollout and continuous regression: roll out to a small traffic share → observe production metrics → go to full traffic. Fix the evaluation set as part of CI so that any model version change runs regression first.
4.2 Replace intuition with a weighted scorecard
CRITERIA = {"quality": 0.35, # Average blind-review score (normalized to 0-1)"structured": 0.20, # Structured-output success rate"latency": 0.15, # How well P95 latency meets the target"cost": 0.20, # Cost per usable output (lower cost = higher score)"ops": 0.10, # Rate limits, docs, observability, compliance}def score(candidate: dict) -> float:return sum(CRITERIA[k] * candidate[k] for k in CRITERIA)
Weights should be tuned per scenario: for content creation, raise quality above 0.5; for high-frequency classification, raise cost and latency above 0.6 combined. The value of the scorecard is not absolute precision in the number, but that it forces the team to turn disagreement from “I feel” into “which specific item is below threshold.”
5. Step 4: Cutting cost in the architecture often beats switching models
Picking a model with good price/performance is only the starting point. In practice, the room for a 50%–80% cost reduction at equal quality usually comes from architecture rather than from unit price.
1. Tiered routing
Use a tiny classification call (or a local lightweight classifier) to judge request complexity first, then route to either the low-cost tier or the flagship tier:
def route(task: dict) -> str:if task["need_reasoning"] or task["est_in_tokens"] > 32000:return "flagship"if task["type"] in ("classify", "extract", "tag"):return "lite"return "standard"
2. Two-level caching
An exact-match cache (hash of the request body) intercepts duplicate requests; a semantic cache (vector nearest neighbor plus a similarity threshold) intercepts near-duplicate requests. Hit rates in FAQ, customer support, and document Q&A scenarios are typically substantial. Note that the cache key must include the model version and the prompt version, or upgrades will produce behavior drift that is hard to locate.
3. Context engineering
Rerank retrieved results and keep only the top-K; summarize long documents in layers first; keep the system prompt lean and put hard constraints up front. This step improves quality, lowers cost, and shortens latency at the same time — a rare three-way win.
4. Output constraints
Set an explicit maximum generation length so the model does not expand indefinitely on open-ended tasks; for structured tasks, use constrained decoding or strict schema validation plus a limited number of repair retries (a retry cap of 1 is recommended; beyond that, take the fallback path).
5. Batch and async
Route offline tasks (batch summarization, data labeling, log classification) through async batch endpoints and message queues to smooth out peaks. This usually yields a clear price discount while avoiding competition with online traffic for quota.
6. Multi-vendor redundancy and fallback
Do not bind your system to a single model. Abstract a unified adapter layer and funnel model calls through one place:
class ModelClient:def generate(self, system: str, user: str, **kw) -> str: ...class FallbackRouter:def __init__(self, primary: ModelClient, secondary: ModelClient):self.chain = [primary, secondary]def call(self, system: str, user: str, **kw):last_err = Nonefor cli in self.chain:try:return cli.generate(system, user, **kw)except (TimeoutError, RateLimitError) as e:last_err = e # Log it, then try the next tierraise last_err
The value of this layer is not just fault tolerance: it turns “model selection” from a one-off technical decision into a knob that can be operated continuously.
6. Common pitfalls
- Leaderboard absolutism: the question distribution in public benchmarks differs from your business distribution; leaderboards are only good for filtering out obviously unqualified candidates.
- Price absolutism: a low unit price plus a high rework rate equals a high real cost.
- Context-length absolutism: paying a long-context premium on every request for the rare scenario that actually has long input.
- One model for everything: pushing classification, extraction, generation, and code all onto a single model — expensive and rarely optimal.
- Ignoring version drift: behavior changes after a vendor upgrades a model version, quietly invalidating your prompts. The countermeasures are pinning versions, keeping the regression evaluation set, and canarying before the change goes live.
- Treating compliance as an afterthought: for scenarios involving user data, questions about whether the data is used for training, how long it is retained, and which region it is stored in should be settled during architecture design, not assembled as paperwork right before launch.
7. Scenario-to-configuration reference
| Scenario | Recommended tier | Key metrics | Optimization focus |
|---|---|---|---|
| High-frequency classification/tagging | Low-tier small model | Accuracy, P95 latency | Caching, batching, few-shot prompts |
| Chinese content creation | Mid-tier workhorse | Human blind-review naturalness | Output-length limits, multi-round revision (cheap input makes iteration affordable) |
| Knowledge-base Q&A / RAG | Mid-tier workhorse | Citation accuracy, refusal rate | Retrieval reranking, context trimming, semantic caching |
| Complex reasoning / code | Flagship tier | First-pass rate, structured success rate | Explicit boundaries, decompose tasks into tool calls |
| Agent multi-turn | Flagship tier + low-tier routing | End-to-end task completion rate | Context compression, retry caps, fallback strategy |
| Speech/image/video | Specialized models | Independent per-modality quality metrics | Decouple from the main pipeline; separate billing and rate limiting |
8. Closing thoughts
The right answer to “which model should I choose” is to replace the question with an operable process: build evaluation sets from real scenarios, decide with cost per usable output instead of unit price, drive costs down with routing, caching, and context engineering, and keep risk in check with a unified adapter layer plus canary regression testing.
For most teams near the quality threshold, migrating to a different model yields far less benefit than doing tiered routing and context slimming once in the architecture. Build evaluation capability first, then argue about which model is better — otherwise any conclusion is just a one-off, unverifiable intuition.
