LLM Selection: How to Evaluate Token Efficiency and Per-Task Cost
Matching a benchmark score does not mean matching cost-efficiency. This article breaks LLM selection into three calculable, optimizable dimensions: where capability comes from, why token efficiency differs, and how cost should actually be computed. It explains total versus activated parameters, the role of post-training as both a capability driver and a token-consumption amplifier, and why "price per million tokens" can be misleading. It then introduces a token-efficiency ratio, a break-even unit-price formula, a runnable cost-estimation snippet, and a full-chain TCO view, before closing with a practical optimization checklist and a data-driven evaluation methodology. Readers will learn how to move model selection from gut feeling to reproducible numbers.
1. The Problem: “Matching the Score” Is Not “Matching the Value”
Over the past two years, the competitive focus of large models has shifted noticeably. Early on, it was about parameter scale; then it moved to benchmark scores; and now more and more technical teams are asking a far more practical question: for the same quality of output, how many tokens and how much money do I have to pay?
A typical scenario: a newer model version, through post-training optimization, matches a flagship model with tens of times more parameters on a third-party composite intelligence index, and its API unit price is significantly lower—it looks like an overwhelming cost-efficiency advantage. But once you switch the accounting basis from “price per million tokens” to “actual cost per task,” the advantage shrinks dramatically—because that lower-priced model often consumes far more output tokens to produce the same answer.
This is not a problem with any single product, but a structural phenomenon emerging across the industry. This article tries to break the issue into three calculable, optimizable dimensions: where capability comes from, where token efficiency differs, and how cost should be computed.
Note: All figures in this article are illustrative, used only to demonstrate accounting methods and reasoning logic; they do not represent the measured performance of any specific model or product.
2. Where Capability Comes From: The Triangle of Base Scale, Sparse Activation, and Post-Training
2.1 What Total Parameters and Activated Parameters Each Determine
Most mainstream large models today adopt a Mixture-of-Experts (MoE) sparse architecture, so it is necessary to distinguish two completely different numbers:
- Total parameters: determine memory/VRAM footprint, weight-loading time, deployment threshold, and the service ceiling per unit of hardware. It is essentially a “capacity cost.”
- Activated parameters: determine the actual compute per token in a forward pass, directly affecting throughput, latency, and inference compute per unit. It is essentially a “speed cost.”
Neither has a linear relationship with “how smart the model is.” Understanding total parameters as the capability ceiling and activated parameters as response speed is a relatively accurate intuition. When a model’s activated parameters are far smaller than its competitor’s yet it achieves a similar composite score, it indicates that the incremental capability does not come from larger dense computation, but from higher-quality training signals and fuller utilization of capacity.
2.2 Post-Training: The Main Battlefield for Capability Gains, and an Amplifier of Token Consumption
The center of gravity for capability improvement has clearly shifted from pre-training to post-training, with main techniques including instruction fine-tuning, preference alignment, reinforcement learning with verifiable rewards, tool-calling and long-horizon task trajectory training, and distilling reasoning ability from stronger models.
The problem is that these methods, while improving “answer accuracy,” naturally lengthen the reasoning process. Reinforcement learning teaches the model to “think longer and more carefully,” because what it is repeatedly rewarded for during training is final correctness, not reasoning length. This gives rise to two typical side effects:
- Overthinking: generating lengthy reasoning on simple problems, with an extremely poor input-to-output ratio;
- Uncontrollable reasoning length: the same model, under different difficulty levels and different prompts, can vary by several times in reasoning tokens.
2.3 “Capability Density” Is Real Progress, but Not a Free Lunch
A small-activation-parameter model catching up to a large-parameter model is essentially an improvement in capability density—the effective capability carried per unit of compute has increased. This is real engineering progress worth acknowledging. But it also brings an invisible bill: part of the density gain is paid for with more reasoning tokens. This naturally leads the discussion to token efficiency.
3. Token Efficiency: The Badly Underestimated Second Cost Axis
3.1 What Token Efficiency Is
It can be defined with a simple ratio:
token efficiency ≈ task quality score / total reasoning tokens per task (average)
The “total reasoning tokens” must include reasoning tokens and output tokens; counting only input length or only the final answer will systematically underestimate cost.
3.2 Why It Is the Dominant Cost Factor
In real applications, input length is often relatively stable (prompts, documents, retrieved snippets), while the difference in reasoning and output tokens can reach several times. Suppose two models have the same quality and the same input tokens; then:
per-task cost ≈ input tokens × input unit price + output tokens × output unit pricecost ratio ≈ output token ratio × output unit price ratio
Illustrative example: model A averages about 41K reasoning tokens per task, model B about 25K, so A is about 1.64× B. This means A is truly cheaper on a per-task basis only when A’s output unit price is below about 61% of B’s (i.e., 1/1.64). This gives a break-even formula you can apply directly:
break-even unit price multiplier = competitor output tokens / own output tokens
This also explains the “scissors gap” phenomenon where “half the token unit price only saves 20% of per-task cost”: the unit-price advantage is largely eaten up by the token-consumption disadvantage.
3.3 The Amplifying Effect in Agent Scenarios
This is not obvious in single-turn Q&A, but is dramatically amplified in Agents, multi-turn dialogues, and long-horizon tool calls, for the following reasons:
- Every turn in a multi-turn dialogue resubmits the full history, so already-generated reasoning content is repeatedly billed as context;
- In an agent loop, a single failed tool call or a single format-parsing error triggers a full re-run of the reasoning;
- Long reasoning occupies the context budget, compressing available working memory, which may actually push up the number of turns.
Therefore, when evaluating token efficiency, be sure to measure at the task level rather than the single-call level, especially for tasks involving more than 5 rounds of interaction.
4. Cost Accounting: The Correct Basis, from “Unit Price” to “Per-Task Cost”
4.1 A Runnable Estimation Implementation
from dataclasses import dataclass@dataclassclass Pricing:name: strprice_in_per_mtok: float # input price (per million tokens)price_out_per_mtok: float # output price (per million tokens, including reasoning tokens)cache_hit_ratio: float = 0.0 # prefix cache hit ratio, 0–1cache_discount: float = 0.0 # discount applied to the cache-hit portion, 0–1retry_factor: float = 1.0 # expected number of calls (including retries / reruns)def task_cost(p: Pricing, in_tokens: int, out_tokens: int) -> float:"""Compute the expected cost of a single task (unified currency unit)."""billable_in = in_tokens * (1 - p.cache_hit_ratio * p.cache_discount)cost = (billable_in * p.price_in_per_mtok+ out_tokens * p.price_out_per_mtok) / 1_000_000return cost * p.retry_factor# Illustrative comparison: lower unit price, but higher token consumptiona = Pricing("Model A", 1.40, 4.40)b = Pricing("Model B", 3.00, 15.00)cost_a = task_cost(a, in_tokens=6_000, out_tokens=41_000)cost_b = task_cost(b, in_tokens=6_000, out_tokens=25_000)print(cost_a, cost_b, f"{(cost_b - cost_a) / cost_b:.1%} cost difference")
Applied to a business, what you really need to look at are three derived metrics:
- Per-task cost: the cost unit the business side understands most easily;
- Cost per unit of quality:
per-task cost / task success rate, counting failed reruns; - Value frontier (Pareto frontier): on the “quality–cost” plane, keep only the options that are not simultaneously dominated by any other configuration.
4.2 Basis Details That Must Be Included
- Whether reasoning tokens are billed separately and whether they count toward the output quota;
- Whether there are batch/async discounts, cache-hit discounts, and tiered pricing;
- Hidden duplicate billing from truncation and retransmission when input is very long;
- Queuing and retries caused by rate limiting, which directly raise
retry_factor; - Free tiers and trial quotas can only be used for evaluation, not as a cost baseline.
5. The Token Bill Is Not Enough: A Full-Chain TCO View
Token costs are often only part of the total cost of an intelligent application. A complete total cost of ownership (TCO) at least includes:
- Latency cost: time to first token and total duration. Long reasoning significantly raises P95 latency, which in interactive scenarios such as customer service and code completion directly translates into experience loss and concurrent resource consumption.
- Infrastructure cost: if using self-hosted or private deployment, total parameter scale determines VRAM and card count, while activated parameters determine throughput; here the concept of “unit price” must be replaced with “compute cost per token + fixed idle cost.”
- Architecture and integration cost: prompt maintenance, output structure validation, evaluation set construction, monitoring/alerting and log analysis, canary and rollback mechanisms.
- Risk cost: rework caused by unstable outputs, manual review triggered by hallucinations, and additional pipeline costs from data compliance and audit requirements.
A cheap but slow engineering solution that needs more validation may have a higher real cost than a slightly more expensive but stable one.
6. An Actionable Optimization Checklist for Engineering
- Reasoning budget control: prefer interface forms that support an upper limit on reasoning length; explicitly lower the reasoning budget for simple tasks and relax it for complex ones, combined with task classification for dynamic allocation.
- Model routing and cascading: use lightweight models or rules to handle high-confidence simple requests, escalating to strong models only in low-confidence, multi-step reasoning, and tool-orchestration scenarios. This is one of the highest single-item returns.
- Prompt slimming: remove duplicate instructions, compress context, and replace verbose natural language with structured input, directly reducing input tokens and unnecessary reasoning triggers.
- Prefix-cache-friendly design: place stable, unchanging system prompts and knowledge snippets first and variable content last to improve the cache hit rate; note that output tokens are usually not cacheable.
- Context engineering and retrieval constraints: limit the number of retrieved snippets, deduplicate, and truncate by relevance, avoiding “the more context you stuff in, the more expensive it gets—and the worse it gets.”
- Retries and idempotency: design idempotency keys and fast-fail paths for tool calls to avoid full re-runs of reasoning.
- Structured output constraints: use explicit schema constraints to reduce format-correction rounds, indirectly lowering total tokens.
- Batching and asynchronization: route offline tasks through batch interfaces and queues for peak-shaving, reducing per-unit cost.
7. Evaluation and Selection Methodology: Turning “Gut Feeling” into “Data”
- Build your own task set: sample 100–500 entries from real business logs, covering high-frequency, long-tail, boundary, and adversarial samples, and annotate standard answers and acceptable answer sets.
- Unify the evaluation basis: fix temperature, random seed, maximum output length, and system prompt; record input tokens, reasoning tokens, output tokens, and elapsed time for each call.
- Look at stability, not single-run peaks: sample the same task multiple times and compute the variance of the success rate, avoiding being misled by a one-off high score.
- Do sensitivity analysis: separately vary token unit price and token consumption, and observe whether the ranking flips, to find the robust range of conclusions.
- Canary validation: launch at low traffic, track cost, latency, success rate, and human feedback with a monitoring/alerting system, and set automatic rollback thresholds.
- Re-evaluate periodically: models iterate frequently, and the validity period of a selection conclusion is usually measured in months; it should be incorporated into routine regression.
8. Conclusion
“Matching the composite evaluation score” is a technical progress worth acknowledging, showing that the combination of post-training and sparse architecture is continuously improving the capability density per unit of compute. But engineering decisions cannot stop at the leaderboard; at least three more questions must be asked:
- For this score, how many tokens were consumed, and how many of them are truly necessary reasoning?
- Converted to per-task cost, how much of the price advantage remains?
- After including latency, stability, infrastructure, and operations, is the total cost still advantageous?
It is recommended to fix the evaluation order as: first check whether quality meets the bar → then check token efficiency → then check per-task cost → finally check full-chain TCO. Only by comparing under the same basis does a price tag carry decision-making meaning. The deciding factor in future model competition very likely lies not just in “who is smarter,” but in “who thinks more economically when reaching the same level of intelligence.”
