A New Metric for LLM Selection: From “Price per Token” to “Useful Intelligence per Dollar”

In discussions about large language model (LLM) selection over the past two years, one habit has become remarkably common: comparing models by “price per million tokens.” The metric is intuitive and easy to share, but it almost always leads to the wrong conclusion. The reason is simple: token price measures input and output volume, while what businesses actually buy is completed work. A model with a very low unit price may still cost more overall than a pricier model if it requires three retries, two rounds of manual review, and eventual rework.

This article proposes an evaluation framework that is closer to engineering reality: Useful Intelligence per Dollar, or UI/$. It is not a laboratory-grade measurement, but a structured thinking tool for technical selection and solution review.

Why Token Price Is a Misleading Metric

Consider a typical engineering scenario. A team needs to use a model to extract and validate structured content from 1,000 documents. Option A uses a low-cost model with a per-call cost of about 0.2 yuan. Its first-pass success rate is only 55%, failed samples require an average of 2.5 retries, and about 20% of results need human review at a cost of roughly 1.5 yuan per item. Option B uses a model priced at about three times Option A, with a per-call cost of 0.6 yuan, a first-pass success rate of 92%, and a human review rate reduced to 5%.

A rough estimate:

• Option A total cost ≈ 1,000 × 0.2 × (1 + 0.45 × 1.5) + 1,000 × 0.2 × 1.5 ≈ 200 × 1.675 + 300 ≈ 635 yuan
• Option B total cost ≈ 1,000 × 0.6 + 1,000 × 0.05 × 1.5 ≈ 600 + 75 = 675 yuan

The total costs are close, but Option A also consumes additional engineering schedule, staffing, and waiting time. Once delay costs and incident risk are included, Option A becomes less economical on a per-unit basis. This example shows that lower unit price does not mean lower total cost, and lower total cost does not mean higher delivered value. What really needs to be measured is how much directly deliverable work each dollar ultimately produces.

Defining and Breaking Down UI/$

To make the discussion computable, here is an estimation formula:

UI/$ = (Deliverable task value × First-pass success rate × Scaling coefficient)
       ÷ (Model cost + Retry cost + Human review cost + Latency cost + Ops cost + Incident cost)

Both the numerator and the denominator deserve closer examination.

The Numerator: What Counts as “Useful”

• Deliverable task value: whether the result can directly enter downstream workflows without rework. It could be “generated code passes compilation and unit tests,” “extracted fields are complete and format-compliant,” or “analysis conclusions are accepted by business stakeholders.”
• First-pass success rate: the proportion of outputs that meet acceptance criteria on the first attempt. This is the key distinction between “looks smart” and “actually usable.”
• Scaling coefficient: whether unit value holds when task volume grows 10x or 100x. It reflects model stability under long-tail inputs, concurrency pressure, and longer contexts.

The Denominator: Cost Is More Than Tokens

• Model cost: direct input and output token expenses.
• Retry cost: extra API calls and orchestration overhead caused by failed attempts.
• Human review cost: manual checking, correction, and fallback effort, which is usually the most underestimated item.
• Latency cost: the impact of first-token latency and task turnaround time on business processes, especially in interactive scenarios.
• Ops cost: rate limits, quotas, monitoring, canary releases, version switching, and related engineering burden.
• Incident cost: rework, customer complaints, and trust loss caused by incorrect outputs reaching production.

Once the full denominator is written out, it becomes clear that many “cheap models” are not actually cheap.

A Five-Dimension Evaluation Framework

To make UI/$ operational, it can be divided into five scorable dimensions with a total weight of 100%.

Dimension	Weight	What It Means
Useful work capability	40%	Whether the model can complete real tasks such as analysis, coding, agent orchestration, and document generation
Total cost per successful task	25%	Not just tokens, but also retries, verbose output, and human review
Reliability	15%	First-pass success rate, factual accuracy, and tool-calling stability
Speed	10%	First-token latency, generation speed, and task turnaround time
Deployment and scalability	10%	API availability, private deployment, ecosystem, and operational complexity

Useful Work Capability (40%)

This is the highest-weighted dimension. Evaluation should be based on task families rather than individual questions. A practical approach is to build a task set that covers your own business: code completion and refactoring, structured extraction, multi-step agent workflows, long-document summarization and question answering, multimodal understanding, and more. Each task type should have explicit acceptance criteria, such as “code passes compilation and unit tests,” “extraction F1 score meets the threshold,” or “agent workflow completes within a limited number of steps.”

Public benchmarks can serve as reference points, but they cannot replace in-house evaluation. Widely used capability indices are typically based on English, text-only evaluations, and their coverage of non-English contexts, multimodal inputs, private deployment, and enterprise-specific workflows is limited. A high ranking on raw capability does not necessarily mean optimal unit economics in your own scenario.

Total Cost per Successful Task (25%)

The recommended metric is “full cost per task” rather than “price per token.” A practical method is to record the entire call chain for each task: number of calls, token consumption, retry count, human intervention time, and end-to-end duration. Aggregate this data and divide by the number of successful tasks to get the “cost per successful task.” This is the number that should actually drive procurement and solution review.

Reliability (15%)

Reliability has three layers: first-pass success rate, factual accuracy, and tool-calling stability. For agent-based applications, the validity of tool-call parameters, retry policies, and idempotency handling are especially critical. It is advisable to include adversarial inputs and boundary inputs in evaluation to observe whether the model fabricates nonexistent interfaces or parameters.

Speed (10%)

The impact of speed on unit economics is often underestimated. First-token latency determines interactive experience, generation speed determines batch throughput, and task turnaround time determines business process waiting cost. In agent scenarios, the cumulative latency of multi-step sequential calls significantly amplifies total elapsed time, so “end-to-end latency per task” should be measured separately.

Deployment and Scalability (10%)

This dimension focuses on engineering feasibility: API availability and rate-limit policies, support for private deployment, ecosystem and toolchain maturity, and operational complexity. The scaling coefficient is most visible here—when concurrency grows from 10 to 1,000, rate limits, quotas, and failure recovery strategies directly change unit cost.

Implementation: A Workable Evaluation Process

The framework can be put into practice through the following steps.

Step 1: Define the task set and acceptance criteria. Extract 50 to 200 representative tasks from real business workloads, covering both high-frequency and long-tail cases, with automatable acceptance rules.

Step 2: Build a cost model. Record token usage, retries, human review, and duration data for each call in a configurable way. Illustrative pseudocode:

def unit_cost(records):
    total = 0.0
    for r in records:
        total += r.model_fee
        total += r.retry_count * r.model_fee
        total += r.human_review_minutes * HUMAN_RATE
        total += r.latency_seconds * LATENCY_RATE
        total += r.ops_amortized
        total += r.incident_cost
    success = sum(1 for r in records if r.passed)
    return total / max(success, 1)

Step 3: Calculate an approximate UI/$ value. The numerator is estimated as “deliverable task value × first-pass success rate × scaling coefficient,” and the denominator is the full cost per successful task from the previous step. The value term can use a conversion method agreed upon by the business side, such as “labor hours saved” or “improvement in downstream pass rate.”

Step 4: Run sensitivity analysis. Focus on three variables: how total cost changes when first-pass success rate drops by 10%, whether rankings reverse when the human review ratio doubles, and whether a 10x increase in concurrency changes conclusions through latency or rate limiting. Many selection decisions flip under sensitivity analysis.

Step 5: Validate with small-scale canary testing. Run A/B comparisons on real traffic, continuously collect first-pass success rate and human intervention data, and update UI/$ estimates with a rolling window.

Common Pitfalls and Engineering Recommendations

Pitfall 1: Treating public benchmarks as selection conclusions. Benchmarks are reference points, not answers. An in-house task set is the real basis for decision-making.

Pitfall 2: Ignoring the long-term accumulation of human review costs. Human review costs tend to grow linearly over time, while model costs may decline with scale. The two trends move in opposite directions and must be modeled together.

Pitfall 3: Using average latency instead of tail latency. Business experience is determined by P95 and P99, not the mean.

Pitfall 4: Equating “open weights” with “fully open source.” Open weights usually mean only that the parameters are available, not that the training data, training code, and license are fully open. Licensing scope and commercial terms should be confirmed separately during evaluation.

Engineering recommendations:

• Record the complete call chain for each task to create a traceable cost ledger.
• Treat first-pass success rate as a core SLO and monitor it together with the human review ratio.
• Introduce idempotency and retry limits in agent orchestration to avoid unlimited retries inflating costs.
• Build fallback strategies for long-tail inputs, with explicit rules for when to escalate to a stronger model or route to human review.
• Re-run the evaluation set periodically to prevent capability drift caused by model version upgrades.

Conclusion

The core claim of Useful Intelligence per Dollar is straightforward: do not ask how much a million tokens cost. Ask how much it costs to actually finish a piece of valuable work. This requires shifting the evaluation perspective from “call price” to “delivered outcome,” and from “single-point capability” to “unit economics.” For developers and enterprises, the value of this framework lies not in producing a precise score, but in providing a structured language for discussion—one that puts capability, cost, reliability, speed, and scalability on the same table for simultaneous scrutiny, leading to selection decisions that are closer to reality. In an era when cloud providers offer multiple models and deployment options, this delivery-value-anchored evaluation method is worth more investment than any single benchmark.