LLM Selection Guide: Performance, Cost, and Engineering Checklist
A practical guide to selecting large language models (LLMs) that looks beyond unit price and leaderboard rankings. It explains how token-based billing works and why task-level cost is what really matters, then lays out a four-dimensional evaluation framework covering general capability, scenario fit, engineering metrics, and safety. It also covers the open-source vs. closed-source tradeoff, a four-step process for building your own evaluation set, model routing and cost-optimization techniques (with example Python code and a worked cost comparison), a selection checklist, and an FAQ on common selection and cost questions.
TL;DR
- Don’t judge by unit price alone: Whether an LLM is “expensive” should be measured by the total cost of completing a unit of business work, not the price per million tokens.
- Don’t judge by leaderboards alone: General leaderboards are useful only for initial screening; what really drives selection is an evaluation set that reflects your business scenarios.
- Cost reduction comes from engineering: Prefix caching, model routing, batching, observability, and gradual rollout are the core cost-optimization techniques.
- Verify pricing and compliance: Vendors adjust pricing across versions, sometimes up and sometimes down, so rely on the official pricing pages; when data security is involved, complete a compliance assessment.
The prices in this article are illustrative. For actual figures, refer to each vendor’s official pricing page and record the access date. This article does not evaluate specific products and does not constitute a recommendation or endorsement of any vendor or product. It also distinguishes between official vendor statements and community rumors.
1. Why do LLMs get more expensive the more you use them?
Many teams complain that token bills keep rising, but the cause is usually not the unit price itself — it’s that call patterns aren’t managed carefully. To understand cost, first understand how LLM billing works.
Most commercial model services use token-based billing, typically pricing input tokens and output tokens separately, commonly in currency per million tokens. A few key points deserve attention.
1.1 Input and output are priced separately
Output tokens are usually priced significantly higher than input tokens. As a result, output-intensive scenarios such as long-form generation, code generation, and summary expansion should focus on output cost.
1.2 The context window sets the ceiling; conversation history sets actual consumption
The context window determines how many tokens a single request can hold, but actual consumption is determined by conversation history. Input tokens for a single request grow linearly with the number of turns; without caching, cumulative consumption across a multi-turn conversation grows roughly with the square of the number of turns; with prefix caching enabled, repeated prefixes can be billed at the cached rate.
This means a long-context model that looks cheap can produce a cumulative bill far higher than expected if it has no caching strategy.
1.3 Whether caching reduces cost depends on hit rate and billing rules
Prefix caching can reduce cost, but not necessarily. Whether it pays off depends on:
- The degree of prefix reuse, i.e., the cache hit rate;
- The gap between the cached-input unit price and the regular-input unit price;
- Whether there is an additional cache-write fee;
- Whether the cache lifetime (TTL) matches your request pattern.
1.4 Batching suits offline tasks, not real-time scenarios
Batch processing usually lowers unit cost, but response times typically range from a few hours to 24 hours — refer to the vendor’s pricing page for specifics. It is suitable only for offline tasks such as batch summarization, batch classification, offline labeling, and offline data cleaning, not online real-time Q&A.
1.5 Task-level cost is the real cost
To judge whether something is “expensive,” don’t look only at the per-call unit price — compute the total cost of completing a unit of business work. You can use the following illustrative formula:
C_task = (in × P_in × (1 - h) + in × h × P_cached + out × P_out) × (1 + r) × N_turn
Where:
C_task: total cost of completing one business task;in: average input tokens per turn;out: average output tokens per turn;P_in: regular input token unit price;P_cached: cached input token unit price;h: prefix cache hit rate;P_out: output token unit price;r: retry rate or failure-retry surcharge;N_turn: average number of turns needed to complete the task.
So a model with a higher unit price, if it completes the task in a single call, may have a lower total cost than a cheap model that needs multiple rounds of retries. When selecting a model, under a given quality threshold, you should estimate and approach a target per-task cost rather than simply chasing the lowest unit price.
2. How to evaluate LLM performance: a four-dimensional framework and pitfalls
Performance can’t be captured by a single leaderboard rank. A rigorous evaluation should cover at least the following four dimensions.
2.1 General capabilities
General capabilities include reasoning, code generation, long-context understanding, and multi-turn consistency. Public benchmarks can serve as an initial reference, but pay attention to the evaluation setup, date, and task distribution. Start with model cards, leaderboard homepages, and official documentation, and record the access date. For leaderboard results, check whether the methodology is public, whether there’s benchmark gaming, and whether it reflects real business needs.
2.2 Scenario-specific capabilities
Evaluation detached from real business data has limited reference value. You must sample from production environments to build an evaluation set and validate output quality with inputs close to real tasks. A model strong on general capabilities isn’t necessarily stable in your specific business scenario.
2.3 Engineering metrics
Engineering metrics directly affect user experience and capacity planning. At minimum, include:
- Streaming time to first token (TTFT);
- Output speed (tokens/s);
- End-to-end P95 latency;
- Throughput and concurrency stability;
- Rate-limiting and retry-backoff mechanisms;
- Timeout and fallback strategies;
- Structured-output reliability;
- Tool-calling or function-calling reliability.
2.4 Controllability and safety
In high-risk scenarios, controllability and safety can matter more than raw capability scores. When evaluating, check:
- Whether refusal policies are reasonable;
- Whether the hallucination rate is acceptable;
- Whether content-compliance capabilities exist;
- Whether there are prompt-injection and jailbreak defenses;
- Whether output content filtering is available;
- Whether sensitive-data masking is supported.
3. Open source or closed source: how to weigh the tradeoff?
New models are often marketed on being “open source, with massive parameter counts.” Parameter scale is indeed a rough proxy for capability, but not the whole story. Selection requires a comprehensive tradeoff.
3.1 Deployment cost
Self-hosting a very large model requires investment in compute, operations, monitoring, scaling, and security governance. Self-hosting is economical and compliant only when call volume is high enough, data is highly sensitive, and long-term costs are controllable.
3.2 Iteration speed
Closed-source APIs usually give faster access to the latest capabilities with no operations burden; open-source models offer greater autonomy and on-premises deployment. The team needs to decide whether the business relies more on fast iteration or on autonomy and control.
3.3 Data security
Before using a closed-source API, complete this data-security checklist:
- Will inputs and outputs be used for model training?
- Are data retention periods and deletion mechanisms clearly defined?
- Does the data residency region meet requirements?
- Are encryption in transit and at rest robust?
- Are there access audits and key management mechanisms?
When personal information or critical data is involved, complete impact assessments and compliance reviews in accordance with applicable data protection, privacy, and AI regulations.
The conclusion is simple: There is no strongest model — only the model that best fits the scenario.
4. How to build your own model evaluation set: 4 steps
Rather than being swayed by leaderboards and debates, build your own evaluation pipeline.
Step 1: Build the evaluation set
Sample from real business to build the evaluation set. To detect a 10-percentage-point difference in pass rate with roughly ±10% error, you typically need 300+ samples, sampled in a stratified fashion by scenario. Samples should cover high-frequency and edge scenarios and include expected outputs or acceptance criteria.
Maintain an evaluation registry containing at least:
| Field | Description |
|---|---|
| Task ID | Unique identifier |
| Scenario | Business scenario category |
| Input summary | Desensitized input content |
| Expected points | Business acceptance criteria |
| Weight | This task’s influence weight on overall selection |
| Model score | Each model’s score on this task |
| Input tokens | Measured input consumption |
| Output tokens | Measured output consumption |
| Cache hit | Whether the cache was hit and the hit ratio |
| Per-task cost | Computed via the total-cost formula |
Also record the evaluation-set version and update rules to avoid overfitting.
Step 2: Multi-dimensional scoring
Use a 1–5 scoring scale:
- 1: Unusable — severe errors or unable to complete basic tasks;
- 2: Minimally usable — requires substantial manual editing;
- 3: Basically usable — meets low-requirement scenarios;
- 4: Production-ready — directly usable for most business;
- 5: Excellent — stable, accurate, and matching the expected style.
Combine human review with automatic metrics, and spot-check inter-rater consistency, for example double-blind scoring and review of divergent samples.
Step 3: Compute cost-effectiveness
Don’t look only at model scores — also look at measured cost. Define the metric as:
Effective output per unit cost = Tasks passing business acceptance / Total cost
Total cost should include call fees, retry fees, manual-review cost, and failure-rollback cost.
Step 4: A/B gradual rollout and rollback
Don’t switch everything at once before launching a new model. First run a gradual rollout against a small traffic share and set guardrail metrics:
- Failure rate;
- P95 latency;
- Human-takeover rate;
- Content-safety block rate;
- User complaints or negative-feedback rate.
Set rollback thresholds in advance; once exceeded, automatically or manually switch back to the old model.
5. Engineering implementation: how to do model routing and cost optimization?
Even after evaluation passes, there is still significant room to cut costs in the implementation phase. Core techniques: cache reuse, model routing, batching, and cost observability.
The code below illustrates the idea of model routing and cost estimation.
from dataclasses import dataclass@dataclassclass ModelProfile:name: strinput_price_per_1m: float # price per 1M input tokensoutput_price_per_1m: float # price per 1M output tokenscached_input_price_per_1m: float # price per 1M cached input tokenscontext_window: int # context window sizeavg_latency_ms: int # average first-token latencydef estimate_cost(p: ModelProfile,in_tokens: int,out_tokens: int,cached_in_tokens: int = 0,) -> float:"""Estimate per-call cost, supporting cached input tokens."""cached = min(max(cached_in_tokens, 0), in_tokens)uncached = in_tokens - cachedin_cost = uncached / 1_000_000 * p.input_price_per_1mcached_cost = cached / 1_000_000 * p.cached_input_price_per_1mout_cost = out_tokens / 1_000_000 * p.output_price_per_1mreturn round(in_cost + cached_cost + out_cost, 6)def route_request(profiles: list[ModelProfile],in_tokens: int,out_tokens: int = 512,cached_in_tokens: int = 0,task_level: str = "simple",low_latency: bool = False,) -> ModelProfile:"""Route based on context, task difficulty, and latency requirements.Convention: profiles are pre-sorted by target priority; simple tasks defaultto the first item (the cheapest baseline). Complex tasks are scored by"cost + latency," and when scores are close, prefer the model with lessremaining context."""if task_level not in {"simple", "complex"}:raise ValueError("task_level must be 'simple' or 'complex'")candidates = [p for p in profiles if p.context_window >= in_tokens]if not candidates:raise ValueError("No model can fit this input length; truncate or shard")# Simple task without low-latency requirement: use the caller's cheap baselineif task_level == "simple" and not low_latency:return candidates[0]# Low latency first: compare first-token latency, then cost and context headroomif low_latency:return min(candidates,key=lambda p: (p.avg_latency_ms,estimate_cost(p, in_tokens, out_tokens, cached_in_tokens),p.context_window - in_tokens,),)# Complex task: normalize cost and latency, then weighted scorecosts = [estimate_cost(p, in_tokens, out_tokens, cached_in_tokens)for p in candidates]latencies = [p.avg_latency_ms for p in candidates]min_cost, max_cost = min(costs), max(costs)min_lat, max_lat = min(latencies), max(latencies)scored = []for p, cost, latency in zip(candidates, costs, latencies):cost_norm = ((cost - min_cost) / (max_cost - min_cost)if max_cost > min_costelse 0.0)lat_norm = ((latency - min_lat) / (max_lat - min_lat)if max_lat > min_latelse 0.0)score = 0.65 * cost_norm + 0.35 * lat_normscored.append((score, p.context_window - in_tokens, p))return min(scored, key=lambda x: (x[0], x[1]))[2]
This code embodies three engineering principles:
- Cost observability: Estimate cost with
estimate_costbefore calling, supporting cached-input pricing. - Task-tier routing: Simple tasks default to the cheap baseline; complex tasks are optimized across cost and latency.
- Long-context constraints: Candidate models must meet context-window requirements; for complex tasks, when scores are close, prefer the model with less context headroom to avoid unnecessary resource usage.
Combined with prefix caching, offline batching, and a logging, monitoring, and alerting system, enterprises can estimate and approach a target per-task cost under a given quality threshold.
Worked example: how a higher-unit-price model that passes on the first try comes out ahead
Suppose a task averages 1000 input tokens and 500 output tokens per turn, with a 60% cache hit rate and a 10% retry rate. The prices below are illustrative, not real vendor quotes.
- Model A: higher input and output unit prices, but passes on the first try, averaging
N_turn = 1. - Model B: lower unit price, but complex scenarios need multiple rounds of clarification and retries, averaging
N_turn = 9.
Applying the task-cost formula:
- Model A’s per-task cost is about
0.0180; - Model B’s per-task cost is about
0.0200.
In this example, Model A has a higher unit price but a lower total cost because it passes on the first try. That is the value of the task-level cost perspective.
6. Selection checklist
Before finalizing selection, confirm each item:
- Have you built an evaluation set close to the business?
- Did you sample in a stratified fashion by scenario and annotate acceptance criteria?
- Did you distinguish simple tasks from complex tasks?
- Did you evaluate latency, throughput, P95, and timeout/retry strategies?
- Did you evaluate safety, refusal, hallucination, injection defenses, and masking?
- Did you compute costs at the task level rather than looking only at unit price?
- Did you estimate cache hit rate, cache price, batch time limits, and retry rate?
- Did you set gradual-rollout traffic, guardrail metrics, and rollback thresholds?
- Did you verify prices on official pricing pages and record the access date?
- Did you complete a compliance assessment before handling sensitive data?
7. FAQ: common questions on selection and cost
Q1: What are common mistakes in LLM selection?
Common mistakes include: looking only at leaderboards and not business scenarios; looking only at unit price and not the number of turns; looking only at input price and not output cost; ignoring cache, retry, and manual-review costs; going live at full volume without an evaluation set; and having no gradual-rollout or rollback mechanism.
Q2: Why do token bills keep rising?
Three common causes: first, multi-turn conversations cause history context to accumulate; second, output tokens are usually priced higher than input, so output-intensive tasks cost more; third, failed retries, manual review, and invalid calls increase total cost.
Q3: How much can caching save?
How much caching saves depends on prefix reuse, cached-input price, cache-write fees, and TTL. If the repeated-prefix ratio is high, the cache hit rate is high, and the cache price is clearly lower than the regular input price, savings can be significant; otherwise it may not pay off.
Q4: How do you compute the cost of self-hosting an open-source model?
Don’t look only at model-card parameters. Also account for compute cost, inference-optimization cost, operations headcount, monitoring cost, security and compliance cost, and iteration cost. Self-hosting is economical only when call volume is high, data is sensitive, and long-term demand is stable.
The “is it expensive?” debate triggered by each new model release ultimately returns to engineering rationality. Parameter scale, leaderboard rank, and unit-price numbers are only inputs. What truly determines selection success is whether the team has built an evaluation and deployment system centered on real business — quantifiable, gradually rollable, and continuously optimizable. Turn evaluation capability into an engineering asset, and you can make more robust technical decisions in a fast-moving landscape.
