Beyond Benchmarks: Three Tables and a Task-Cost Ledger for LLM Selection

Every time a flagship large language model launches, public attention tends to focus on two things: benchmark leaderboards and published price sheets. But in real engineering work, the decisions about whether to adopt a model, how long to keep it, and whether it is safe for production often come down to a third table — availability — and a more fundamental question: how much does it actually cost to complete a real business task, including failures and rework?

The industry has repeated this storyline many times: a new model launches with impressive benchmarks and aggressive pricing, only for the provider to apologize days later for rate limiting, quota issues, and unexpected token consumption. That is a useful reminder that evaluating an LLM service requires a broader framework than simply looking at scores and unit prices.

1. Three tables: pricing, capability, and availability

1.1 The pricing table: a “unit price” is really a set of numbers

Modern inference service billing is usually split into three components: input tokens with a cache miss, input tokens with a cache hit, and output tokens. Output pricing is often several times higher than input pricing. Cache-hit pricing may be only about one tenth of the miss price. This means:

• A “unit price” is not one number; it is a set of numbers. Any cost estimate must incorporate assumptions about cache hit rates. Otherwise, bills for long-context workloads can be overestimated by an order of magnitude.
• Comparisons must use the same basis. Per-million-token pricing, the same currency, the same context range, and the same treatment of volume discounts all matter. If any of these differ, cross-service comparisons can be off by several times.
• Watch for hidden terms. Concurrency limits, per-request surcharges, membership tiers, and quota systems all change the real cost structure.

1.2 The capability table: benchmarks are necessary but not sufficient

• Public benchmarks have three systemic biases: data contamination, where test items may have appeared in training data; task-domain bias, where a benchmark’s strengths may not match your business scenario; and saturation effects, where a one- or two-point difference among top models may not be statistically meaningful. A different set of test items could reverse the ranking.
• Different benchmarks measure different capabilities. General question answering, terminal coding, agent tool calling, and long-context understanding are not the same skill set. When selecting a model, prioritize domain-specific evaluations that are structurally similar to your workload.
• Translate scores into task completion rates. Third-party benchmarks are only useful for initial screening. The final decision should be based on measured pass rates on your own task set.

1.3 The availability table: the most overlooked and most revealing dimension

• Rate limiting systems. Two-level thresholds on request count and throughput, rolling time windows, and quotas assigned by account tier all affect real workloads. This is especially damaging for multi-step agent tasks — if a task hits a rate limit at step 15, the tokens consumed in the first 14 steps are still billed in full.
• Stability. Pay attention to peak-hour error rates, queueing latency, incident frequency, and how transparent the provider is in its status communications.
• An underused source of information is how a provider responds after an incident. If an official announcement says the fix involves “reducing unnecessary sub-agent calls” or “compressing unnecessary output,” that is effectively an admission that a large share of the bill comes from intermediate work the model performed on its own initiative. That kind of signal is often more decision-relevant than any benchmark score.

2. Unit task cost: what the bill actually consists of

Start with the full equation:

Task cost = Σ (miss input tokens × miss price + hit input tokens × hit price + output tokens × output price) + retry and rework cost + human verification cost

Breaking this down:

1. Intermediate consumption is the largest variable. In an agent architecture, one user request expands into N model calls: task planning, tool invocation, subtask execution, and result summarization. Each call carries context in the input and may produce lengthy reasoning in the output. The nominal input price is only part of the story. Actual input tokens may be dozens of times larger than the user’s original question.
2. Failures and retries are billed at full price. A task interrupted by rate limiting, an output that fails validation and triggers automatic retry, or a fallback to human review all add real costs that never appear on the published price sheet.
3. Output redundancy directly inflates both cost and latency. A verbose model increases output billing, lengthens end-to-end response time, and blocks downstream stages in a pipeline.
4. A simple illustrative calculation. Suppose a 300-page tender document is roughly 300,000 tokens, and the task requires reading and cross-checking one full pass. The following numbers are only for illustration. If nothing is cached, input alone may reach about 900,000 tokens because the document is read in segments three times. If prefix caching achieves a 90% hit rate, input cost drops by roughly an order of magnitude. If the agent orchestration expands into 20 calls and each call carries 50,000 tokens of context, real input volume easily exceeds one million tokens.

The conclusion: a model with a unit price 80% lower may still cost more per completed task if it requires three times as many loops and produces twice as much redundant output. This is why “best benchmark score” and “lowest task cost” can both be false at the same time — they are not measured on the same axis.

3. Implementation: building a task-level evaluation and metering system

3.1 Build your own task set

Extract 20–50 representative tasks from business logs. Cover four categories: long-document extraction, cross-document verification, code generation and repair, and multi-step tool calling. For each task, define machine-checkable acceptance criteria, such as field comparison, unit test pass, or product schema validation.

3.2 Core metrics

Metric	Meaning	Why it matters
First-pass rate	Share of tasks accepted without human intervention	Determines rework cost
Average rework rounds	Average interactions or retries before acceptance	Amplifies token consumption
Unit task cost	Total spend to complete one task	The real decision metric
End-to-end latency P50/P95	Time from submission to usable result	Affects experience and throughput
Rate-limit trigger rate / error rate	Observed in load testing and production	Availability baseline

3.3 Cost instrumentation

Mainstream inference APIs return usage fields in their responses, such as prompt tokens, completion tokens, and cached tokens. Naming varies slightly, but the semantics are generally consistent. Record usage at the gateway layer and aggregate it by task:

# Gateway-level cost aggregation by task.
# Field names follow common API usage conventions. Prices are illustrative.
PRICING = {  # unit: currency per million tokens
    "input_miss": 10.0,
    "input_hit": 1.0,
    "output": 60.0,
}
def call_cost(u: dict) -> float:
    hit = u.get("cached_tokens", 0)
    miss = u["prompt_tokens"] - hit
    return (miss / 1e6 * PRICING["input_miss"]
            + hit / 1e6 * PRICING["input_hit"]
            + u["completion_tokens"] / 1e6 * PRICING["output"])
class TaskLedger:
    """Aggregate multiple model calls for one business task into a single cost record."""
    def __init__(self, task_id: str):
        self.task_id, self.calls = task_id, []
    def record(self, usage: dict):
        self.calls.append(usage)
    def report(self) -> dict:
        return {
            "task_id": self.task_id,
            "model_calls": len(self.calls),  # Agent loop count: the first cost lever
            "total_cost": round(sum(call_cost(u) for u in self.calls), 4),
        }

The key observation is not the cost of a single call. It is the product relationship between model_calls — how many model calls one business task expands into — and total cost. Reducing the number of calls usually lowers cost more effectively than squeezing the unit price.

3.4 Availability load testing and fallback plans

• Run canary load tests using production-like concurrency profiles to identify rate-limit boundaries and rolling-window behavior.
• Design a fallback chain: switch to a backup model or enter a queue when the primary service is rate limited. Long tasks must support checkpoint resumption — intermediate outputs should be persisted to object storage or a database, so an interrupted task can resume from the checkpoint instead of re-burning the entire task’s tokens.
• Incident credit clauses from the provider can be included in procurement evaluations, but compensation should not be treated as an availability strategy.

3.5 Continuous regression

Model services change almost monthly: parameter refreshes, pricing adjustments, and rate-limit policy updates. Turn the steps above into a pipeline: keep the task set fixed, generate metrics automatically, and compare versions automatically. Any provider-side change should show task-level metric fluctuations before going live.

4. Engineering-side cost reduction checklist

1. Trim context. Use retrieval augmentation instead of stuffing full documents into the prompt. Use rolling context windows for conversation history. Keep stable system prompts at the prefix to improve cache hit rates.
2. Constrain output. Use structured output such as JSON Schema or function calling, set explicit length limits, and require the model to output only conclusions and supporting evidence.
3. Govern agent orchestration. Set both a maximum step count and a token budget. Merge subtasks that can be combined. Run independent tool calls in parallel instead of serially.
4. Route by model tier. Send simple tasks such as classification, extraction, and format conversion to smaller models. Reserve flagship models for difficult tasks. Mainstream cloud platforms and model gateways usually support unified multi-model access with usage observability, which makes routing policies and unified metering easier to implement.
5. Apply budgets and circuit breakers. Set budgets per task or tenant. Trip a circuit breaker and send an alert when the budget is exceeded. For agent-type tasks, enforce a hard budget cap to prevent runaway consumption.

5. Conclusion: change the coordinate system

When model capabilities broadly enter a “good enough” range, the selection criteria should shift from “how far has the model advanced” to “how much does it cost to complete a real task, how many times does it fail, and how many rounds of rework are needed.” The pricing table gives nominal unit prices. The capability table gives an upper bound on ability. But only the availability table and the unit task-cost ledger answer whether a model can run reliably and economically in your business. Models and providers will keep changing, but this framework of three tables plus one ledger can be reused over the long term. That framework is the asset that retains its value in model selection.