Inference Tiers and Cost per Task: An Engineering Method for Evaluating LLM Price-Performance

1. A 0.1 Version Bump Can Double Your Bill

A flagship reasoning model shipped a minor version update: 5.1 to 5.2. On paper, it is three wins at once. Its composite capability score jumped from 49 to 65, setting a new best on public leaderboards; peak output throughput improved by roughly 30%; and the API price dropped about 20%, to $8 per million input tokens and $40 per million output tokens.

Read those three numbers and the verdict is “stronger, faster, cheaper.” Swap the unit of measure to the one the business actually cares about, the single task, and the verdict flips. At the highest reasoning effort setting, average output tokens per task grew from about 118K to 274K, an increase of roughly 132%, and average cost per task rose from $12.99 to $26.50, nearly doubling.

The lesson here is not that the new model is a bad deal. It is a measurement problem that gets ignored for a long time: unit price is a procurement metric, task cost is an operating metric. Token usage and call counts are jointly determined by model behavior, the reasoning effort setting, and how the application orchestrates the model. If any one of those gets out of control, the savings from a price cut get swallowed. The rest of this article breaks that down into a computable, verifiable engineering method.

2. Break “Price-Performance” Into Three Orthogonal Dimensions

A single benchmark score cannot support a selection decision. It covers only one dimension and conflicts noticeably with the other two. A three-dimensional framework works better:

Dimension	Core metrics	How to collect	Common misread
Capability	First-attempt task success rate, evaluation set score	Offline evaluation set	Substituting public leaderboards for business data
Cost	Cost per successful task, per-task token usage	Call log aggregation	Looking only at price per million tokens
Latency	Time to first token, end-to-end P50/P95	Trace instrumentation	Looking only at average throughput

All three must be recorded at the same time, because they are three facets of the same transaction. Lengthening the reasoning chain to raise capability pushes up both cost and latency at once. Cutting the reasoning budget to lower latency pulls down the first-attempt success rate, which then pushes cost back up through retries. Any single-dimension optimization gets paid back on the other two.

3. The Single-Task Cost Model

The cost of an LLM task can be written as:

C_task = Σ_r ( T_in,r · p_in + T_cache,r · p_cache + T_out,r · p_out )
C_success = C_task / P_success        # real unit cost after retries and failures

where r is the r-th model call, and T_out includes both the explicit reply and the internal reasoning content. This is the key point: most providers bill the reasoning process as output tokens, and the output price is typically 4 to 5 times the input price (here, $40 versus $8). So reasoning length is the dominant line item on the bill, not the input context.

Here is a cost estimator you can plug straight into your pipeline:

from dataclasses import dataclass, field
PRICE = {"input": 8.0, "cache_read": 0.8, "output": 40.0}  # USD / million tokens
@dataclass
class Call:
    in_tok: int
    cache_read_tok: int = 0
    out_tok: int = 0          # includes reasoning tokens
@dataclass
class TaskMeter:
    calls: list[Call] = field(default_factory=list)
    budget_usd: float | None = None
    success: bool = False
    def record(self, call: Call) -> None:
        self.calls.append(call)
        if self.budget_usd is not None and self.cost() > self.budget_usd:
            raise BudgetExceeded(self.cost())
    def cost(self) -> float:
        total = 0.0
        for c in self.calls:
            total += c.in_tok * PRICE["input"] / 1e6
            total += c.cache_read_tok * PRICE["cache_read"] / 1e6
            total += c.out_tok * PRICE["output"] / 1e6
        return total
    def cost_per_success(self, p_success: float) -> float:
        # the lower the first-attempt success rate, the more retries amplify cost
        return self.cost() / max(p_success, 1e-6)

Check the observed behavior against this model. Output tokens grew 2.32x while the price fell to 0.8x, so the output portion of the cost became about 1.86x larger. Add the input context that gets re-billed on every call plus the cache misses across multiple rounds, and the total lands at roughly 2x, which matches the measured ratio of 26.50 / 12.99.

Three conclusions follow. First, evaluating a new model means recomputing task cost, not unit price. Second, cache hit rate has as much effect on the bill as the reasoning effort setting does. Third, do not treat “average output tokens” as a constant during estimation; it varies significantly with reasoning effort and task difficulty.

4. Reasoning Tiers Follow a Diminishing-Returns Curve, and the Middle Tier Is Often the Inflection Point

Different reasoning effort settings on the same model are just different sample points on one capability-cost curve, and that curve is concave. Going from a little to a moderate number of reasoning tokens produces a large capability gain. Keep adding and the gain decays quickly while cost grows linearly, or even super-linearly.

The data in this example happens to give two comparable curves: the new model at its middle reasoning tier scores 62 at about $9.08 per task, while the previous flagship at its highest tier scores 59 at about $10.04 per task. In other words, the new model’s default tier beats the old model’s maximum tier at a lower cost. That is a more useful engineering conclusion than “the new model can reach 65,” because it hands you a default configuration you can adopt directly.

For a quantitative selection rule, use marginal capability gain per unit cost:

ROI(tier) = ( Score(tier) - Score(baseline) ) / ( Cost(tier) - Cost(baseline) )

Compute the marginal ROI of each tier against the one below it, take the tier just before ROI drops off sharply as the production default, and reserve the higher tiers for the few task types where they demonstrably pay off.

5. Agents and Coding Tasks: Retries Make Cost Non-Linear

The cost structure of coding tasks is more complex than that of Q&A tasks, because a patch that “looks correct” often fails later tests, triggering multiple rounds of repair, and every round re-bills the inflated context.

The comparison in this example is clear. At maximum reasoning intensity, the model scores 49.1% on the code evaluation, a new best. At the middle tier it scores 45.3%, essentially tied with the previous generation’s highest tier at 45.5%, at roughly half the per-task cost. On tasks that require external verification (compilation, tests, static analysis), first-attempt success rate and average number of repair rounds predict the bill better than a benchmark score does.

The corresponding engineering move is budget guards and fast failure, not an unbounded “think again”:

MAX_TURNS = 6
MAX_TASK_BUDGET_USD = 3.0
async def run_agent(task, meter: TaskMeter):
    for turn in range(MAX_TURNS):
        tier = "high"
        if turn >= 2 or task.risk == "high":
            tier = "max"            # only escalate after a failure or for high-risk tasks
        resp = await model.call(task.context, reasoning=tier)
        meter.record(resp.usage)
        if await verifier.check(resp.patch):   # tests/compilation act as a cheap acceptance gate
            meter.success = True
            return resp.patch
        task.context = compress(task.context + resp.notes)  # compress history to suppress context growth
    raise TaskFailed(f"exhausted, cost={meter.cost():.2f}")

Three design principles matter here. Leave acceptance to deterministic tools rather than model self-assessment. Compress context on demand, or replace long-context stacking with retrieval. And trigger a reasoning tier upgrade only when the task has already failed or its risk level is high, instead of running everything at maximum by default.

6. Latency-Constrained Scenarios: Real-Time Loops Cannot Absorb Long Reasoning

If the consumption scenario is exploratory coding where waiting is tolerable, a user scrolling their phone for tens of seconds of deep thinking is fine. Put the same model into a real-time interaction or embodied AI scenario and the logic inverts immediately. Feasibility for those systems is decided by loop frequency, so you have to budget end-to-end latency first:

T_e2e = T_sense + T_decide + T_plan + T_act + T_comms

If sensing takes 30ms, actuation 20ms, and communication 20ms, while the decision stage waits seconds to tens of seconds for multi-step deep reasoning, the effective frequency of the control loop collapses to a fraction of a hertz. By then the environment has already changed, and the plan no longer applies even if it was perfectly correct. No benchmark score can compensate for a timing mismatch.

The workable approach is layering, assigning “thinking depth” to the layer that suits it:

1. Low-frequency layer (seconds to minutes): the large model handles task decomposition, root-cause analysis of anomalies, and policy re-planning. Call volume is low, so a high reasoning tier is affordable.
2. Mid-frequency layer (hundreds of milliseconds): a small or medium-sized model, or a specialized policy model, handles local decisions and state judgment.
3. High-frequency layer (tens of milliseconds): rules, controllers, or lightweight models close the loop and guarantee a deterministic upper bound on latency.

Supporting measures include a hard latency cap on high-frequency decisions with a degraded fallback allowed, asynchronous pre-warming and caching of anything pre-computable, and result caching plus policy reuse for recurring scenarios. Once the system is layered this way, the large model’s long reasoning only shows up at the low-frequency decision points where it actually creates value, and both cost and real-time behavior stay under control.

7. Implementation Checklist

1. Build your own business evaluation set: 50 to 200 tasks drawn from the real distribution, covering both high-frequency and long-tail cases. Do not use public leaderboards directly as a procurement basis.
2. Record four dimensions of data: reasoning tier, first-attempt success rate, token usage (split into input, cache, and output), and P50 and P95 end-to-end latency.
3. Use cost per successful task as the primary metric: fold retries and failures into the denominator so the number reflects real unit economics.
4. Start from the middle reasoning tier by default: escalate only for high-risk or already-failed tasks, and put the escalation rule in code rather than relying on human judgment.
5. Budget guards and circuit breakers: set per-task, per-business-line, and per-day budget caps; when a cap is hit, degrade or queue instead of letting a single point run away.
6. Cost governance infrastructure: ship call logs to a logging service, aggregate tokens and spend by business line through monitoring and alerting, and pair that with container platform autoscaling and asynchronous load buffering through a message queue. On Baidu AI Cloud, these general-purpose capabilities can be composed directly into a token and cost observability loop.
7. Separate offline from online workloads: send batch jobs through an asynchronous low-priority lane and online requests through a low-latency lane, so they do not compete for the same quota.
8. Run regression evaluations on a schedule: re-run the evaluation set after every model version change, and focus on whether the task-cost curve has shifted, not just on the score.

8. Conclusion

Three judgments are worth taking away:

• Public leaderboard scores are a screening threshold, not a procurement basis. What determines the bill is task-level token usage, the reasoning tier, and the number of calls.
• Make cost per successful task the primary metric, use marginal ROI to pick the reasoning tier, and remember that the middle tier is often the price-performance inflection point.
• In real-time and embodied scenarios, deployability is decided by architectural layering rather than peak capability. Long reasoning has to be placed in the low-frequency decision layer, and the high-frequency loop needs a deterministic latency upper bound.

Model capability is still evolving quickly, but the evaluation methodology should stabilize first. Unified units of measure, collectable data, and reproducible decisions are what let you judge quickly whether the next version bump is worth it.