LLM Selection in Practice: Performance, Cost, and Evaluation Methodology

1. Why “Picking a Model by Reviews” Easily Goes Wrong

Whenever a new large language model is released, community discussion tends to split into two camps: one focused on whether it can match the international frontier, the other attacking its pricing as too expensive. Both narratives have a real observational basis, but neither can directly guide engineering decisions, for three reasons:

• Mismatched evaluators: Most public evaluations come from short-term experience, while production environments care about stability and cost curves under long cycles and high traffic.
• Inconsistent cost definitions: Some look only at input unit price, some at promotional prices, and some compare subscription prices against API prices. Different definitions naturally produce conflicting conclusions.
• Highly abstract capability descriptions: Statements like “close to the top tier” cannot answer “is it good enough for my task.”

For developers and enterprises, a more reliable approach is to translate “expensive or not, strong or not” into quantifiable engineering questions: What is the cost per task? What is the quality pass rate? How large is the migration cost? The following sections proceed through four steps: cost, performance, deployment form, and process.

2. Look at API Billing in Full: Unit Price Is Only One Piece of the Cost Puzzle

1. Input and output are priced separately, and the gap is usually significant

Most model services split pricing into input and output, with output unit price generally higher than input. This means that judging whether a model is cheap by its input unit price is nearly meaningless for generation tasks such as copywriting, code, and summarization, because the bill for these tasks is dominated by output tokens. In multi-turn conversations, history messages are repeatedly billed as input, and the cumulative cost cannot be ignored.

2. Context length is a hidden variable

Long-context capability is both a selling point and a cost multiplier. Take RAG (retrieval-augmented generation) as an example: each request injects several retrieved chunks, and the longer the context, the more input tokens per request. Evaluation should be based on the context distribution of real business traffic, not the shortest prompt in official examples.

3. Pay attention to billing discount mechanisms

There are three main cost-saving mechanisms commonly used in the industry:

• Context caching: Repeated prefixes (such as fixed system prompts or long documents) are billed at a lower price once they hit the cache.
• Batch inference: Offline tasks that are not latency-sensitive (data labeling, batch cleaning) go through batch interfaces, usually with a clear discount.
• Tiered pricing: Prices differ across performance tiers or time periods.

The availability and discount magnitude of these mechanisms vary considerably across platforms. When selecting a model, you should calculate the actual unit price based on your own traffic pattern rather than looking only at list prices.

4. Don’t forget failures and retries

Retries caused by rate limiting, timeouts, or non-compliant output formats all generate real bills. For rough estimates, it is advisable to reserve 5%–10% for retry costs; in high-concurrency scenarios, you should also evaluate the impact of rate-limiting policies on business throughput.

A minimal usable monthly cost estimation script:

def estimate_monthly_cost(
    daily_requests: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    input_price: float,     # currency / million tokens (placeholder example, substitute current list price)
    output_price: float,    # currency / million tokens
    retry_rate: float = 0.05,      # extra ratio caused by retries
    cache_hit_ratio: float = 0.0,  # share of input that can hit cache
    cache_discount: float = 1.0,   # cache price / original price
):
    daily_in = daily_requests * avg_input_tokens
    daily_out = daily_requests * avg_output_tokens
    in_cost = (daily_in / 1e6) * input_price
    # Recalculate the difference for the portion that hits cache at the discounted price
    in_cost += in_cost * cache_hit_ratio * (cache_discount - 1)
    out_cost = (daily_out / 1e6) * output_price
    daily_cost = (in_cost + out_cost) * (1 + retry_rate)
    return daily_cost * 30
# Example: 10,000 requests per day, average input 3,000 tokens, output 800 tokens
# print(estimate_monthly_cost(10_000, 3000, 800, 4, 12))

Note: The prices in the script are placeholder examples only. For actual estimates, substitute the target platform’s current list prices and real traffic distribution.

3. Performance Evaluation: Build Your Own Evaluation System Beyond Leaderboards

1. The value and limits of public benchmarks

Public leaderboards are suitable for initial screening: they can roughly sketch capability tiers and narrow a candidate set from dozens down to three to five. But they have three classic problems: data contamination (questions may have leaked into training data, inflating scores), distribution bias (benchmark task distributions differ greatly from business distributions), and ceiling effects (leading models differ very little on conventional benchmarks, so discrimination is insufficient).

2. Build your own evaluation set

It is recommended to sample from real business logs and cover three types of samples:

• Typical samples (60%–70%): representing daily mainstream traffic;
• Boundary samples (20%–30%): long context, multi-constraint instructions, domain-specific terminology;
• Historical failure samples (about 10%): cases that previously caused production issues, used for regression validation.

3. Suggested evaluation dimensions

• Correctness / task completion rate;
• Instruction and format adherence (directly affects the robustness of parsing code);
• Hallucination rate (can be automatically verified with retrieval-based attribution);
• Latency metrics: time to first token and output throughput;
• Stability: output variance across multiple runs of the same input.

4. Human blind testing

Having reviewers score paired outputs (win/tie/loss) without knowing which model produced them can effectively eliminate bias from brand expectations. Two to three hundred representative samples are enough to support preliminary conclusions.

4. Self-Hosting or API Access: Decide with Total Cost of Ownership (TCO)

In recent years, open-weight models have continued to improve, and “deploy an open model yourself” has become an option for many teams. But open weights do not mean free, and the cost structures of the two paths are completely different:

Dimension	API mode	Self-hosted mode
Cost structure	Pay-as-you-go, clear marginal cost	GPU purchase/rental plus operations labor, high fixed cost
Scale effect	Roughly linear growth	The larger the traffic, the lower the unit cost
Data compliance	Data leaves your domain, requires tier assessment	Can be fully private
Customization space	Depends on server-side supported capabilities	Deep customization of inference and post-processing
Iteration speed	Follows platform upgrades at almost zero cost	Version upgrades and tuning are your own responsibility

A rough decision heuristic:

• Modest daily token volume, highly variable traffic, in the validation stage: prioritize APIs for fast iteration;
• Large and stable traffic, high compliance requirements, with an operations team: consider self-hosting, or a hybrid tiering of “self-hosted core paths plus API for long-tail peaks.”

In addition, mainstream cloud platforms typically package model inference services, usage monitoring, log analysis, and alerting as general product categories. Whichever path you choose, it is advisable to include “whether platform-side observability is complete” in your assessment—it directly determines the efficiency of troubleshooting after launch.

5. A Practical Selection Workflow

1. Requirement definition: Clarify task types, latency budget, accuracy floor, per-request context distribution, and compliance level;
2. Candidate convergence: Use public leaderboards and community feedback to roughly screen three to five candidates, eliminating those that clearly fail compliance or latency requirements;
3. Small-scale testing: Run your own evaluation set plus a 1%–5% shadow comparison on real traffic;
4. Cost estimation: Use measured token distributions (not official examples) with the method in Section 2 to estimate the monthly bill, including caching and batch discount mechanisms;
5. Gray launch: Set up dashboards with both quality and cost metrics, with automatic rollback on anomalies;
6. Quarterly review: Models iterate extremely fast, so it is recommended to re-evaluate with the same evaluation set every quarter to keep the selection from going stale.

6. Conclusion

Back to the initial question: “Is being expensive an original sin?” Pricing is essentially a function of capability, supply and demand, and inference engineering cost. Expensive or cheap in itself does not constitute a value judgment. The real question is: under the tasks and traffic you define, does this model offer the optimal cost per task and quality pass rate? When the evaluation framework is objective enough, every new model release is no longer a topic of “taking sides,” but an opportunity to re-optimize effectiveness and cost. This is precisely the greatest advantage of engineering methods over emotional evaluation: quantifiable, reproducible, and iterable.