LLM API Selection: From Unit Price Debates to Cost-Value Evaluation

1. Defining Concepts Before Evaluating New Models

Whenever a flagship LLM is released, the community often repeats the same debate: some users argue that the pricing is too high, while others point out that the input and output unit prices have actually decreased compared to the previous generation. These two perspectives seem contradictory but are actually based on different coordinate systems. The former discusses “perceived cost”—the actual expense of completing a specific business task—while the latter focuses on “listed unit price”—the挂牌 price per million tokens.

For developers making technical selections, this confusion of concepts is an enemy of decision-making. Unit price is merely one variable in the cost function. Under different context management strategies, task success rates, and invocation modes, the final bill for the same unit price can vary several-fold. Therefore, the correct approach to evaluating a new model is to deconstruct it into a reproducible engineering problem: understand the pricing structure, build a total cost model, calibrate performance using scenario-based evaluations, and finally solidify the entire assessment into a pipeline.

2. Reading the Pricing Table: Unit Price is Just the Tip of the Iceberg

The billing for mainstream LLM APIs typically consists of three layers. Comparing only the first layer often leads to distorted conclusions:

1. Input and Output Unit Prices. The price of output tokens is generally significantly higher than that of input tokens, often by a multiple. This means the cost structure varies completely across scenarios: tasks focused on summarization and extraction have a low output ratio and are sensitive to input prices; tasks focused on code generation and long-form writing have a high output ratio and are sensitive to output prices.
2. Context Caching Billing. Modern inference services commonly support prefix caching. When multiple requests carry the same system prompts, tool definitions, or retrieval contexts, the cached portion can be billed at a significant discount. Some services even price cache writes and hits separately. For Agent-like applications where system prompts easily reach tens of thousands of tokens, the cache hit rate directly determines the order of magnitude of the cost.
3. Tiered Billing for Long Contexts. Some services apply a higher tiered unit price for tokens exceeding a certain length threshold. If the average context length of your tasks exceeds 100,000 tokens, this tiered rule must be factored into the calculation.

Consider a purely illustrative calculation: Assume a model’s input price is $20 per million tokens (for demonstration only,不代表 any real product). An Agent application carries a 100,000-token context per request, 90% of which is a fixed prefix eligible for caching, and the cached portion is billed at 20%. The input cost per request would be roughly:

(100,000 × 90% × 20 × 20% + 100,000 × 10% × 20) / 1,000,000 = 0.36 + 0.2 = $0.56

In contrast, the intuitive calculation of “list price × total tokens” would be $2.00—a difference of more than 3x. This is why arguing over unit prices is meaningless; the final bill is shaped by your invocation patterns.

3. A Drop in Unit Price Does Not Equal a Drop in Total Cost: Building a Cost-Per-Task Model

What you should really compare is not “how much per million tokens,” but “how much does it cost on average to complete a business task.” A sufficiently practical simplified model is as follows:

Cost per task ≈ (P_in × N_in + P_out × N_out) × (1 + retry_count) / pass_rate

Here, P_in and P_out are the input and output unit prices, while N_in and N_out are the token consumption per task. Of these four variables, only the unit price is decided by the vendor; the rest are determined by your own engineering capabilities:

• Token consumption structure: More powerful models (especially those with deep reasoning capabilities) consume more thinking tokens. The bill per request might actually increase, but this trades off for an improved pass rate.
• Pass rate: If a new model increases the pass rate for code tasks from 60% to 85%, the costs of retries and manual corrections are drastically amortized. In automated pipelines, this metric often carries more weight than the unit price.
• Context management: Retrieval trimming, history compression, and cache reuse can each change N_in by an order of magnitude.

The following illustrative code integrates these variables into a reproducible estimator:

def cost_per_task(p_in, p_out, n_in, n_out,
                  cache_hit_ratio=0.0, cache_discount=0.8,
                  pass_rate=1.0, avg_retry=0.0):
    """Estimate cost per task (in USD). Price unit: USD/million tokens."""
    effective_in = (
        n_in * cache_hit_ratio * (1 - cache_discount)
        + n_in * (1 - cache_hit_ratio)
    )
    single_cost = (effective_in * p_in + n_out * p_out) / 1e6
    return single_cost * (1 + avg_retry) / pass_rate
# Illustrative parameters: input $20/million, output $60/million, context 100k tokens,
# 90% cache hit rate, pass rate 85%, average 0.2 retries
print(cost_per_task(20, 60, 100_000, 2_000,
                    cache_hit_ratio=0.9, pass_rate=0.85, avg_retry=0.2))

Substituting the parameters of the old and new generation models into this formula yields the true answer to whether an upgrade makes your business more expensive or cheaper. In practice, there are two common conclusions: the unit price drops but single-request token consumption rises, leaving the cost per task roughly flat; or the unit price remains flat but the pass rate significantly improves, leading to a noticeable drop in cost per task. Both conclusions provide far more informational value than the emotional label of “expensive” or “cheap.”

4. Performance Evaluation: Benchmark Scores are Just Entry Tickets

Where does the “pass rate” in the cost model come from? It can only come from your own scenario evaluations. Public benchmarks can be used for initial screening, but directly using leaderboard rankings for decision-making carries several known risks:

• Benchmark saturation: The gap between top-tier models on mainstream public benchmarks is often smaller than the evaluation noise, and the correlation between scores and real-world business performance continues to decay.
• Distribution mismatch: Public benchmarks struggle to cover the long-tail difficulty, domain-specific terminology, and multi-turn context dependencies of real business requests.
• Single-dimension misleading: Long-text compression, complex reasoning, code repair, and tool calling are distinct capability axes. A single aggregate score masks specific shortcomings.

A more reliable approach is to build your own evaluation set. First, perform stratified sampling from real business logs, covering easy, medium, and hard difficulty tiers, with dozens to hundreds of samples per tier to ensure statistical significance. Second, define machine-verifiable acceptance criteria for each task (unit test pass, accurate field extraction, valid format) to reduce the subjectivity of manual scoring. Third, solidify the regression workflow—run the full evaluation set before any model version switch, outputting three metrics: pass rate, average token consumption, and P95 latency, and analyze them in conjunction with the cost model.

5. Open Source Models and Self-Deployment: A Different Ledger

In recent years, the open-source community has produced trillion-parameter Mixture-of-Experts (MoE) models, with several capabilities approaching those of closed-source flagships. This has changed the fundamentals of the selection game. However, the ledger for self-deployment must also be calculated clearly:

• Explicit costs: GPU procurement or rental, cross-AZ disaster recovery, quantization and speculative decoding optimization for the inference framework, and KV cache hit rate tuning. Each of these consumes engineering manpower.
• Implicit costs: Open-source models iterate just as frequently. Version tracking, security auditing, and peak scaling are long-term burdens.
• Applicable boundaries: For scenarios with large and stable request volumes, strict data residency requirements, or the need for deep fine-tuning, self-deploying open-source models is often more cost-effective. For scenarios with low request volumes or a need for rapid validation, the elastic advantages of hosted APIs are more obvious. For teams that do not want to build their own compute infrastructure, you can also evaluate managed model services on cloud platforms (such as Baidu AI Cloud and similar platforms that offer model inference and supporting services), transferring the operational burden to focus on application layer development.

6. Solidifying Evaluation into a Pipeline: Three Must-Do Tasks

1. Cost observability. Record token usage, cache hit rate, latency, and task results for every API call, and integrate them into unified logging, monitoring, and alerting systems (these capabilities are standard product categories on mainstream cloud platforms and can be used directly without custom development). Without observability, all cost optimization is blind tuning.
2. Canary deployment. Do not switch all traffic to a new model at once. First route 5%–10% of real traffic for online A/B testing, observe the cost per task, pass rate, and user-facing negative feedback, and then decide whether to proceed with a full rollout.
3. Multi-model routing. Using different model tiers for different tasks is the architecture with the best cost-performance ratio. A simple rule-based routing example:

def route(task):
    if task.tokens_estimate > 80_000:
        return "long_context_model"   # Dedicated tier for long context
    if task.needs_deep_reasoning:
        return "reasoning_model"      # High reasoning tier
    return "light_model"              # Lightweight fast-response tier

An advanced approach is cascaded routing: first attempt the task with a lightweight model, and only escalate to a higher-tier model upon failure or low confidence, compressing the average cost while ensuring quality.

7. Conclusion

Returning to the original question: whether a new model is “expensive” is fundamentally not a pricing issue, but a structural one. The unit price is a constant provided by the vendor, while token consumption, cache hit rate, and pass rate are variables you can optimize yourself. Rather than arguing in comment sections about whether “expensive is the original sin,” it is better to implement the cost-per-task formula: build evaluation sets, integrate monitoring, conduct canary tests, and deploy routing. Once this pipeline is running, every new model release will no longer be an emotional taking of sides, but a routine upgrade evaluation where conclusions are drawn simply by running the benchmarks.