LLM Model Selection: The Real Cost Ledger—Go Beyond Token Prices to Cost per Resolved Task

3 hr ago0 views

Flagship model price cuts do not automatically translate into lower total cost. This article explains why token unit price alone is misleading and shows how to calculate cost per resolved task using a reusable cost model. It covers input and output tokens, cached input, retries, pass rate, hidden costs, evaluation sets, tiered routing, shadow traffic, canary rollout, cost instrumentation, common anti-patterns, and a practical checklist for choosing and deploying LLM models.

1. Price Cuts Do Not Automatically Mean Cost Savings: Why Price Lists Can Mislead

Every time a flagship model cuts its price, the same discussion appears: “If the price dropped by X%, isn’t this now the most cost-effective option?” The implicit assumption is that if usage stays constant, a lower unit price leads to a proportional drop in total cost. In real systems, the opposite is often true: usage is a function of model behavior, not an external constant.

A more capable model often completes a task in fewer turns. A less capable model may retry, make repeated attempts, or inflate context so much that it consumes more billable tokens. As a result, there are two separate price dimensions:

Pricing dimension: price per million tokens, set by the vendor’s public pricing.
Consumption dimension: actual tokens burned to complete a specific task, driven by model behavior, prompt design, and context management.

Making decisions based only on the first dimension is like predicting total engineering cost from procurement unit price while ignoring rework, retries, and human intervention. The engineering metric that matters is simple: cost per resolved task.

2. Breaking Down Cost: Four Metering Dimensions

Before asking “which model is cheaper,” confirm which quantities multiply to form the bill. Most mainstream platforms meter around token counts, but these can be split into four categories. All four matter.

Metering item	What drives it	Common mistake
Input tokens	System prompt length, retrieval chunks, accumulated conversation history	Counting only the current prompt and ignoring multi-turn history
Output tokens	Generated length, chain-of-thought/reasoning tokens, redundant structured output	Ignoring intermediate tokens from reasoning-oriented models
Cached input	Reuse rate of system prompts and long documents	Not accounting for caching at all, which overestimates cost
Retries and failures	Parse failures, timeouts, failed format validation	Counting only successful calls

In multi-turn conversations, the easiest place to get burned is history accumulation: context carried by the 10th request may be much larger than that of the first. Without explicit truncation, summarization, or retrieval replacement, cost can grow roughly quadratically with the number of turns.

3. A Reusable Cost Model

Writing the metering dimensions as formulas gives the cost per task:

C_task = N_call × (T_in × p_in + T_out × p_out + T_cache × p_cache) × (1 + r_retry)
C_unit = C_task / P_success      # actual cost per successfully completed task

Here, N_call is the number of model calls required to complete the task, r_retry is the retry rate, and P_success is the first-attempt success rate. The denominator is the key: a model with a low unit price but only a 60% pass rate can have an effective unit cost amplified by about 1.67x.

Below is an estimator that can be used directly for offline evaluation. Prices are placeholders and should be replaced with current real prices.

from dataclasses import dataclass, field
@dataclass
class Price:
    """Price per million tokens (placeholder; replace with actual pricing)."""
    input_per_mtok: float
    output_per_mtok: float
    cached_input_per_mtok: float = 0.0
@dataclass
class TaskStat:
    calls: int = 0
    tokens_in: int = 0
    tokens_out: int = 0
    tokens_cached: int = 0
    retries: int = 0
    resolved: bool = False
    latency_ms_total: int = 0
def cost_of(stat: TaskStat, price: Price) -> float:
    if stat.calls == 0:
        return 0.0
    per_call_in = stat.tokens_in / stat.calls
    per_call_out = stat.tokens_out / stat.calls
    per_call_cache = stat.tokens_cached / stat.calls
    unit = (
        per_call_in * price.input_per_mtok
        + per_call_out * price.output_per_mtok
        + per_call_cache * price.cached_input_per_mtok
    ) / 1_000_000
    return unit * stat.calls * (1 + stat.retries / stat.calls)
def cost_per_resolved(tasks: list[TaskStat], price: Price) -> dict:
    total_cost = sum(cost_of(t, price) for t in tasks)
    resolved = [t for t in tasks if t.resolved]
    n = len(tasks)
    return {
        "total_tasks": n,
        "pass_rate": round(len(resolved) / n, 3) if n else 0.0,
        "total_cost": round(total_cost, 4),
        "cost_per_resolved": round(total_cost / len(resolved), 4) if resolved else float("inf"),
        "avg_calls": round(sum(t.calls for t in tasks) / n, 2) if n else 0.0,
        "avg_end_to_end_latency_ms": round(sum(t.latency_ms_total for t in tasks) / n, 1) if n else 0.0,
    }

Illustrative example (numbers are assumed and only demonstrate the difference in metering): For a code modification task, Model A has half the unit price of Model B, but Model A needs 5 calls to pass, while Model B passes in 1 call.

Model A: 5 calls × (80,000 input tokens + 12,000 output tokens), price 2/10 USD per million tokens → about 0.86 USD per resolved task
Model B: 1 call × (60,000 input tokens + 8,000 output tokens), price 4/20 USD per million tokens → about 0.26 USD per resolved task

Even though Model B’s unit price is twice as high, its cost per resolved task is about 70% lower. This is the whole mechanism behind “the more expensive model may be cheaper”: capability translates into token savings.

4. Beyond Tokens: Four Hidden Costs

Token costs alone can produce an overly optimistic conclusion. A complete total cost of ownership also includes:

Latency cost: Higher-capability models may increase time-to-first-token and overall response time. In interactive scenarios, this translates directly into user experience loss. It is advisable to set a separate P95 latency budget for each request path.
Engineering and context-management cost: More expensive models often deserve more investment in prompt compression, retrieval augmentation, and context summarization. These engineering changes themselves have development and maintenance costs.
Failure and human-takeover cost: Models with low first-attempt pass rates shift cost to human review. This often does not appear in the API bill, but it may be the largest single expense.
Observability and governance cost: Without token-level instrumentation, cost aggregation by business line or tenant, and threshold alerts, cost overruns are hard to locate. Cloud platforms generally provide common capabilities such as logging, monitoring and alerting, and metering/billing. Cost dashboards can be built on top of these rather than relying on a single vendor’s proprietary dashboard.

5. Implementation: Route by Task Tier, Not by Model Side

5.1 Build Your Own Evaluation Set

Public leaderboards are only for initial screening. Real business workloads need an evaluation set of 100–300 tasks with reference answers, covering typical input-length distributions, common failure modes, and edge cases. Metrics should not be limited to accuracy. At minimum, record first-attempt pass rate, average calls, cost per resolved task, and P95 latency.

5.2 Tiered Routing

Most businesses are mixtures of tasks. A single model will inevitably waste money. A practical approach is three-tier routing:

def route(task) -> str:
    """Route by task complexity; avoid sending every request to the most expensive model."""
    if task.requires_long_chain_reasoning or task.is_high_risk:
        return "flagship"          # complex reasoning / high-risk changes
    if task.is_structured_extract or task.is_classification:
        return "small"             # light tasks such as extraction, classification, and formatting
    return "mid"                   # regular generation and rewriting
# Fallback escalation: escalate to the larger model if the light model fails validation,
# balancing cost and quality.
def run_with_escalation(task, validators, clients):
    tier = route(task)
    for model in [tier, "flagship"] if tier != "flagship" else [tier]:
        resp = clients[model].complete(task)
        if all(v(resp) for v in validators):
            return resp
    return escalate_to_human(task)

Routing rules should be inferred from offline evaluation data: for each task type, measure pass rate and unit cost across models, plot a quality–cost curve, and use the inflection point as the tier boundary. This curve should be redrawn quarterly as models iterate, because vendor updates can change both unit price and consumption behavior.

5.3 Shadow Traffic and Canary Rollout

Before a new model is fully launched, mirror production traffic and issue shadow calls. Only collect metrics; do not return responses to users. Compare token distributions, pass rate, latency quantiles, and refusal rate. During canary rollout, ramp from 5% to 25% to 100%. At each stage, set automatic rollback thresholds, such as unit task cost exceeding baseline by 30% or first-attempt pass rate dropping by 5 percentage points.

5.4 Put Cost Instrumentation into the Request Path

Each request should carry labels such as business line, tenant, task type, and model tier. Usage fields should be persisted so that bills can be drilled down by dimension:

def audit_log(record: dict):
    # record: {biz_line, tenant, task_type, model_tier, tokens_in,
    #          tokens_out, cached_tokens, latency_ms, passed, cost}
    metrics.increment("llm.cost.usd", record["cost"], tags=record["tags"])
    metrics.histogram("llm.latency.ms", record["latency_ms"], tags=record["tags"])
    if not record["passed"]:
        metrics.increment("llm.retry.total", tags=record["tags"])

Without this data, the question “which model is more cost-effective?” will always be argued based on gut feeling.

6. Five Common Anti-Patterns

Replacing your own evaluation with leaderboard scores: Public benchmark distributions usually differ from business distributions. A leaderboard lead does not mean a lead in business pass rate.
Ignoring multi-turn history inflation: Conversational applications without context limits can see costs spiral out of control as sessions grow.
Missing caching strategy: Stable system prompts, long documents, and few-shot examples that are not reused as prefixes mean paying repeatedly.
Looking only at the API bill: Human takeover, engineering investment, and conversion loss caused by latency are not on the bill.
Switching all at once: Without shadow traffic and canary rollout, problems are most expensive when they appear in production.

7. Checklist

Before deciding “which model to use, and where,” confirm that each of the following has a clear answer:

Is traffic split by task type instead of using a single model globally?
Is there an internal evaluation set that provides cost per resolved task for each task type?
Are cache hit rate, context length, and retry rate measured and optimized?
Is there a dual budget for latency and cost for each request path, with alerts?
Is there a periodic review process that tracks vendor price and capability changes?

Conclusion: A price-cut announcement changes only one coefficient in the cost formula. What determines the bill is how many calls, how many tokens, and how much human effort are required to complete one thing. Shifting the accounting basis from unit price to cost per resolved task, and then routing by task tier, is the engineering solution that saves budget without sacrificing quality.