How to Choose LLMs: A Guide to Useful Intelligence per Dollar (UI/$)

    AI model leaderboards that once drove LLM selection are losing relevance as top models converge in capability and inference prices keep falling. This guide introduces a business-oriented metric—Useful Intelligence per Dollar (UI/$)—that shifts evaluation from benchmark scores to end-to-end cost accounting. It breaks down the six cost components of completing a real task (model fees, retries, human review, latency, operations, and incidents), shows why a low per-token price does not guarantee a low total cost, and explains how to combine deliverable value, first-pass success rate, and a scaling factor into one comparable score. It also examines how the industry is moving from intelligence contests to value competition, and lays out a five-step path for enterprises to build their own UI/$ evaluation system—from task sampling and unified evaluation sets to end-to-end instrumentation, per-task-type accounting, and regression reviews—along with cost-conversion templates and common pitfalls to avoid.

    From Intelligence Contests to Value Competition: How to Choose LLMs

    1. Starting Point: How Should You Choose an LLM?

    Over the past few years, when choosing an LLM, enterprises often looked at capability leaderboards first: whoever had the higher benchmark score advanced to the next round of evaluation. This worked well when capability gaps were wide and price differences were small, because it reduced a complex technical decision to a single comparable number.

    But as mainstream models’ capabilities have gradually converged, the score gaps between top models have narrowed, and inference prices keep falling, the “rankings-first” selection logic starts to feel thin. The question enterprises need to answer is shifting from “whose model is stronger” to “how do we evaluate LLM cost-effectiveness,” “how do we compare LLM costs,” and “how do we calculate the cost per task”—ultimately landing on a more business-oriented question: how much value can each dollar deliver?

    This shift comes from three factors:

    1. Scores don’t include cost information. Two models with similar capabilities may differ: one passes on the first try, while another needs multiple retries. A low unit price doesn’t necessarily mean a low total cost.
    2. Benchmark tasks aren’t business tasks. Public benchmarks have relatively fixed tasks, formats, languages, and scoring criteria, but real business involves document standards, tool calls, exception handling, and terminology requirements that all affect results.
    3. Scores are averages, not distributions. Two models with the same average pass rate may perform completely differently on low-difficulty vs. high-difficulty tasks, which directly determines whether you need a human fallback.

    So the focus of model selection is shifting from “benchmark comparison” to “end-to-end cost accounting.” This article proposes a core metric: Useful Intelligence per Dollar, or UI/$ for short. It tries to answer: in real business, when you spend one dollar, how much directly deliverable value do you get back?

    2. Core Proposition: Value Is Determined by the Complete Delivery Process

    1. A Different Yardstick: The Cost of a Successful Task

    Traditional cost accounting usually only looks at the unit price of input and output tokens, treating one model call as one successful delivery. In reality, a single call isn’t always successful. What business actually cares about isn’t “how much does one call cost,” but “how much does it cost to actually complete one piece of valuable work.”

    So we can introduce two measures:

    • Total cost per task: the total price paid to complete a task on average, including model fees, retries, human review, latency, operations, and incidents.
    • Cost per successful task: based on total cost per task, converted to a per-successful-delivery basis using the first-pass success rate.

    A more business-oriented expression is:

    Cost per successful task = (model fees + retry cost + human review cost + latency cost + operations cost + incident cost) ÷ number of successful tasks

    If you can’t precisely convert the success rate yet, you can first measure the total cost per task, then use the first-pass success rate to correct the value side:

    UI/$ = deliverable task value × first-pass success rate × scaling factor ÷ total cost per task

    Here is a sample table showing “a low price doesn’t mean a low cost”:

    Cost item Model A (low price) Model B (high price)
    Model fee per call (relative) 1.0 3.0
    Average number of calls (incl. retries) 3.2 1.1
    Model fees 3.20 3.30
    Human review cost (amortized) 1.60 0.30
    Latency cost (amortized) 0.48 0.11
    Operations & incident cost (0 in this example) 0.00 0.00
    Total cost per task 5.28 3.71

    Model A’s per-call model fee is only one-third of Model B’s, but after adding retries, human review, and waiting, its total cost per task is actually about 40% higher. The reason isn’t the price itself, but the hidden costs. Model prices change, business value changes, and scenario constraints change, so selection conclusions must be based on measured data at the current point in time—you can’t directly apply historical leaderboards or old experience.

    2. The Six Cost Items Come from Different Sources

    These six cost items don’t come from the same source, and the ways to control them differ:

    Cost item Main source Example control measures
    Model fees Input/output token billing, inference resource usage Context trimming, result caching, routing
    Retry cost Repeated calls due to failed or substandard output Structured output, automatic validation, budgeted retries
    Human review cost Manual review, correction, rework Confidence scoring, risk-based sampling, automatic acceptance
    Latency cost Slow first-token response, long output, queuing Streaming output, prefix reuse, priority scheduling
    Operations cost Gateway, auth, monitoring, scaling, version upgrades Unified gateway, automated ops, version canary
    Incident cost Wrong output, sensitive content, compliance risk Input/output review, canary release, rollback

    It’s worth noting that latency cost and incident cost can’t always be directly converted into money. Enterprises should determine conversion coefficients through internal interviews, historical tickets, customer complaint records, compensation data, and business-loss estimates, and keep them consistent throughout the evaluation period. Otherwise, evaluation results from different times or teams are hard to compare.

    3. Capability Leaderboards Are Anchors, Not Conclusions

    Capability leaderboards still have value—they help enterprises quickly shortlist candidate models. But they can only serve as screening clues, not final selection conclusions, for these reasons:

    • Limited language and modality coverage. Many benchmarks focus on English plain text, so they struggle to cover Chinese-context understanding, writing, and domain knowledge, and can’t reflect multimodal capabilities like images, documents, and audio/video.
    • Weight settings vary by leaderboard. Different leaderboards divide capability dimensions differently. For example, a capability index might weight coding at 30%, scientific reasoning at 25%, Agent tasks at 25%, and general understanding at 20%, summing to 100%, but the specific weights must follow the official definition and shouldn’t be extrapolated arbitrarily.
    • Deployment forms are ignored. Public cloud API, dedicated instances, and on-premises deployment each have trade-offs in data compliance, network isolation, and elastic scaling, none of which show up in benchmark scores.
    • Update cadence isn’t synchronized. Leaderboards have fixed update cycles, while model iteration, price changes, and tool-ecosystem shifts often happen faster.

    So the proper role of capability leaderboards is as candidate anchors. Final conclusions must return to the enterprise’s own task set and be re-measured.

    4. Scaling Factor: Measuring “Can Each Dollar Buy More Value at Scale?”

    The scaling factor answers: when usage grows, does the value one dollar buys increase or decrease? It’s usually affected by three types of factors:

    1. Marginal cost changes. Higher cache-hit rates and batch-processing discounts may dilute per-call cost; but peak scaling, rate-limiting smoothing, and long-context calls may push unit cost up.
    2. Changes in per-unit labor input. After automated validation and process standardization improve, the marginal human-review input drops.
    3. Hidden cost amplification. As call volume rises, the impact of a single content-safety incident or service outage also grows, so risk costs need to be counted into the scaling cost.

    So the scaling factor shouldn’t be set by guesswork at a single usage tier, but characterized through measured curves at different usage tiers.

    3. Building the Metric: Quantifying “Useful Intelligence”

    1. A Discussable, Iterable Metric: UI/$

    To put “capability” and “cost” in the same table, we can build an illustrative metric:

    UI/$ = (deliverable task value × first-pass success rate × scaling factor) ÷ total cost per task

    Breaking it down:

    • The numerator answers “is it valuable?”: how much directly deliverable value one dollar ultimately buys back. Deliverable task value can be converted from labor hours, business revenue, task settlement price, or internal service unit price.
    • The denominator answers “is the cost high?”: not just the model quote, but all the model fees, retries, human review, latency, operations, and incident costs incurred during task completion.
    • First-pass success rate answers “is it reliable?”: only counting tasks that pass acceptance without retries or manual edits.
    • Scaling factor answers “is it still worth it after scaling?”: whether value and cost change together as usage grows.

    If you’ve already converted cost into “cost per successful task,” you can use an equivalent form:

    UI/$ = deliverable task value × scaling factor ÷ cost per successful task

    2. Composite Scoring Framework

    To compare “capability” and “cost” in the same table, you can set a 100-point composite score. Here are illustrative weights:

    Dimension Weight Meaning
    Useful work capability 40% Can it complete real tasks like analysis, coding, Agent tasks, and document generation
    Total cost per successful task 25% Not just token unit price, but also retries, verbose output, and human review overhead
    Reliability 15% First-pass rate, factual accuracy, stability of tool calls and structured output
    Speed 10% First-token latency, generation speed, task turnaround time
    Deployment & scaling 10% API availability, on-premises deployment options, ecosystem tools, ops complexity

    Note that these weights are illustrative, not the standard answer. Weight settings should satisfy two conditions:

    • Weights sum to 100%.
    • Weights must be calibrated to business goals, task types, sample sizes, and risk appetite, and subjected to sensitivity analysis. For example, does the ranking change noticeably after raising the reliability weight by 5%; does the conclusion reverse after raising the cost weight by 10%. If a small weight adjustment completely changes the conclusion, it means the sample size is insufficient or the metric is still unstable.

    Weights can differ by scenario:

    • Intelligent customer service: first-contact resolution and response latency are most critical, so reliability and speed weights can be raised.
    • Code assistance: tool calls, multi-turn revisions, and structured output matter more, so useful-work capability and total-cost-per-successful-task weights can be raised.
    • Document and content production: throughput, format compliance, and long-text handling matter more, so total-cost-per-successful-task and deployment/scaling weights can be raised.

    3. Illustrative Calculation Code

    The code below illustrates the calculation approach. To stay consistent with the earlier table, it treats “model fee per call” directly as a relative value and amortizes human review, latency, operations, and incident costs per task. Model fees accumulate by average number of calls; human review, latency, operations, and incident costs are no longer multiplied by the number of calls. Operations and incident costs are set to 0 only to align with the table; in production they should be filled in based on actual business.

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class TaskCostProfile:
    4. """Per-task cost profile. Values are illustrative and should be replaced with measured data."""
    5. model_cost_per_call: float = 0.0 # Model fee per call; may include token unit price and input/output usage conversion
    6. avg_calls: float = 1.0 # Average number of calls, including retries on failure
    7. review_cost_per_task: float = 0.0 # Human review cost amortized per task
    8. latency_cost_per_task: float = 0.0 # Latency cost amortized per task
    9. ops_cost_per_task: float = 0.0 # Operations cost amortized per task
    10. incident_cost_per_task: float = 0.0 # Incident cost amortized per task
    11. def single_task_total_cost(profile: TaskCostProfile) -> float:
    12. """Total cost per task = average calls * model fee per call + per-task amortized costs."""
    13. model_fee = profile.model_cost_per_call * profile.avg_calls
    14. return (
    15. model_fee
    16. + profile.review_cost_per_task
    17. + profile.latency_cost_per_task
    18. + profile.ops_cost_per_task
    19. + profile.incident_cost_per_task
    20. )
    21. def ui_per_dollar(
    22. value_per_success: float,
    23. profile: TaskCostProfile,
    24. first_pass_rate: float,
    25. scale_factor: float = 1.0,
    26. ) -> float:
    27. """UI/$ = deliverable task value * first-pass success rate * scaling factor / total cost per task."""
    28. if first_pass_rate <= 0:
    29. raise ValueError("First-pass success rate must be greater than zero")
    30. cost = single_task_total_cost(profile)
    31. if cost <= 0:
    32. raise ValueError("Task cost must be greater than zero")
    33. return value_per_success * first_pass_rate * scale_factor / cost
    34. if __name__ == "__main__":
    35. # Model A: low price but more calls, higher human review and latency
    36. model_a = TaskCostProfile(
    37. model_cost_per_call=1.0,
    38. avg_calls=3.2,
    39. review_cost_per_task=1.6,
    40. latency_cost_per_task=0.48,
    41. ops_cost_per_task=0.0,
    42. incident_cost_per_task=0.0,
    43. )
    44. # Model B: higher unit price but fewer calls, lower human review and latency
    45. model_b = TaskCostProfile(
    46. model_cost_per_call=3.0,
    47. avg_calls=1.1,
    48. review_cost_per_task=0.3,
    49. latency_cost_per_task=0.11,
    50. ops_cost_per_task=0.0,
    51. incident_cost_per_task=0.0,
    52. )
    53. cost_a = single_task_total_cost(model_a)
    54. cost_b = single_task_total_cost(model_b)
    55. print(f"Model A total cost per task: {cost_a:.2f}")
    56. print(f"Model B total cost per task: {cost_b:.2f}")
    57. # Assume a deliverable task is worth 10 dollars; Model A first-pass rate is 62%, Model B is 94%
    58. score_a = ui_per_dollar(10.0, model_a, first_pass_rate=0.62)
    59. score_b = ui_per_dollar(10.0, model_b, first_pass_rate=0.94)
    60. print(f"Model A UI/$: {score_a:.2f}")
    61. print(f"Model B UI/$: {score_b:.2f}")

    The results can be read as:

    • Model A total cost per task: 5.28
    • Model B total cost per task: 3.71
    • Model A UI/$: 1.17
    • Model B UI/$: 2.53

    These results show: even if Model A has a lower unit price, if its first-pass success rate isn’t good enough and retries and review costs are high, its value per dollar may still be lower than Model B’s.

    Based on this method, you can form a decision matrix like the one below:

    Candidate First-pass rate Avg. calls Total cost per task UI/$ Positioning suggestion
    Candidate A 62% 3.2 5.28 1.17 Routing fallback
    Candidate B 94% 1.1 3.71 2.53 Primary model
    Candidate C 88% 1.4 4.50 1.96 Supplement for complex tasks

    4. Industry Impact: From Intelligence Contests to Value Competition

    1. Competition Shifts from Parameters to “Effect + Cost”

    In the past, launch events often featured parameter counts, context length, and benchmark scores. As capability gaps narrow and prices keep falling, enterprises care more about whether a model can reliably complete business tasks and whether per-task cost is controllable. Vendors need to deliver on model capability, inference cost, tool ecosystem, and deployment flexibility simultaneously to gain an edge in specific business scenarios.

    2. Pricing Shifts from “Per Token” to “Per Result”

    Per-token billing isn’t friendly to cost estimation: the longer the output and the more retries, the harder the bill is to control. Per-result billing, subscription-quota billing, and volume-package billing turn “counting tokens” into “counting tasks,” which naturally fits the UI/$ approach.

    However, per-result billing also has boundaries:

    • It suits scenarios with clear acceptance criteria, determinable results, and clear responsibility boundaries, such as code generation, document structured extraction, and customer-service ticket classification.
    • It doesn’t suit fully subjective, exploratory, or controversial scenarios. These require human review, appeal mechanisms, and dispute-resolution terms.
    • Contracts need to specify acceptance criteria, failure/retry rules, charging boundaries, responsibility division, and data-security requirements.

    For enterprises, budget is no longer a hard-to-explain number, but something that maps to “how many tasks are completed each month.”

    3. Evaluation Focus Shifts from Benchmarks to Business Scenarios

    General leaderboards are still useful, but enterprises care more about whether model output is usable, compliant, and deliverable under their own document standards, terminology systems, permission constraints, and toolchains. This brings two changes:

    • Building in-house evaluation sets. Enterprises start turning anonymized real tasks into evaluation sets, forming versioned regression baselines.
    • Engineering evaluation. Evaluation results need to be automatically compared, continuously tracked, and sensitive to model version changes.

    4. Architecture Shifts from “Single Model” to “Tiered Routing”

    A typical cost-optimization approach is tiered routing:

    • Small-parameter or low-cost models handle high-frequency simple tasks like extraction, classification, and rewriting.
    • High-capability models concentrate on complex reasoning, long-chain tasks, and high-risk tasks.
    • Intermediate results and stable outputs go through caching.
    • The access layer supports a unified gateway, canary switching, and rollback to avoid business disruption when swapping models.

    5. Changes in Demand Structure

    The more specific and workflow-aligned a task is, the higher the delivery value per unit of intelligence. This drives two types of demand growth:

    • Customized capabilities for vertical scenarios, such as retrieval augmentation combined with an enterprise knowledge base.
    • Controllable, auditable deployment methods, such as on-premises deployment, dedicated resource isolation, permission auditing, and log retention.

    Reliance on general leaderboards will decline, while the combination of “in-house evaluation + business metrics” will rise.

    6. Selection Methods and Team Collaboration

    • From intuition to data. Selection shifts from “guesswork” to a “measure–compare–decide” loop.
    • From individual to team. Evaluation, cost accounting, and observability need to be done jointly by R&D, algorithm, ops, business, and security teams.
    • Delivered value becomes a shared language. In cross-team communication, “what’s the first-pass rate for this type of task, what’s the per-task cost, what’s the incident risk” builds consensus more easily than “what’s the model’s rank.”

    5. Implementation Path: How Enterprises Build an “Intelligence per Dollar” Evaluation System

    Step 1: Define Tasks and Value Measures

    • Sample 50–200 representative tasks from real business, covering high-frequency, low-frequency, and high-risk scenarios.
    • Define acceptance criteria for each task, such as correct format, complete fields, factual accuracy, successful tool calls, and risk-content blocking.
    • Standardize value conversion, using labor hours, internal settlement prices, or business-revenue estimates. The key is to stay consistent throughout and avoid switching measures midway.

    Step 2: Build a Unified Evaluation Set

    • Each sample includes input, key expected-output points, automatic validation rules, and human-review points.
    • Score as automatically as possible, using rules, tests, or structured comparison; subjective tasks can use two-person review with spot checks.
    • Version-manage the evaluation set; when adding tasks, don’t delete old ones, ensuring results are traceable and comparable.

    Step 3: End-to-End Instrumentation

    Log the full chain for every call as the basis for cost and reliability analysis. Fields you can collect include:

    • Model fees: input and output token counts, model version, billing unit price.
    • Retry count: how many times the same task was resubmitted or regenerated.
    • Human time: actual time spent in the review step.
    • Latency distribution: P50/P95 of first-token latency and end-to-end latency.
    • Incident count: compliance blocks, rework from wrong output, customer complaints, or compensation.

    A simple instrumentation approach:

    1. import time
    2. def call_with_telemetry(client, payload, task_id):
    3. started = time.perf_counter()
    4. resp = client.chat(payload)
    5. usage = getattr(resp, "usage", None)
    6. return {
    7. "task_id": task_id,
    8. "input_tokens": usage.prompt_tokens if usage else None,
    9. "output_tokens": usage.completion_tokens if usage else None,
    10. "elapsed_ms": (time.perf_counter() - started) * 1000,
    11. "status": "ok",
    12. }

    The collected details first go into a message-queue buffer, then into logging and data-storage services for later aggregation and analysis. Enterprises can also use common cloud capabilities—container platforms, message queues, logging services, and monitoring/alerting—such as the relevant components provided by platforms like Baidu AI Cloud, to build a unified access layer and continuously observe the call chain.

    Step 4: Account and Compare by Task Type

    Group samples by task type and calculate total cost per task, first-pass success rate, UI/$, and scaling factor for each. The point isn’t to find one company-wide “best model,” but to find the right candidate combination for each type of task.

    For example:

    • Simple customer-service Q&A: focus on first-contact resolution, response latency, and per-call cost.
    • Code generation: focus on runnability, test pass rate, tool-call success rate, and rework count.
    • Document summarization: focus on format compliance, factual consistency, long-text handling, and amount of manual editing.
    • High-risk review: focus on incident cost, compliance block rate, auditability, and human fallback capability.

    Step 5: Build Regression and Review Mechanisms

    Model version upgrades, price changes, prompt changes, and business traffic shifts can all change conclusions. It’s recommended to rerun the evaluation set monthly or after each major change, and output trend curves:

    • Continuously declining UI/$ means value is leaking away.
    • Rising per-task cost with an unchanged pass rate calls for checking retry logic, prompt design, or context length first.
    • A higher pass rate but a larger cost increase means the benefit doesn’t cover the cost, requiring a re-tradeoff.
    • Rising incident cost while model unit price falls may indicate that low-price models lack stability, requiring stronger input/output review or routing fallback.

    Cost Conversion Templates: Incident Cost and Latency Cost

    To make cost measurement more actionable, you can add the following fields:

    Incident cost

    Incident cost = (error rate × cost per rework) + (complaint rate × cost per complaint handling) + (compliance block rate × cost per compliance review) + (sensitive event rate × cost per emergency response)

    Fields you can collect:

    • Error rate: number of wrong outputs ÷ total tasks.
    • Complaint rate: tasks that triggered complaints ÷ total tasks.
    • Compliance block rate: tasks blocked by security or compliance policies ÷ total tasks.
    • Average handling time: average labor time for rework, review, and complaint handling.
    • Loss per incident: can be converted from compensation amounts, brand-loss estimates, or internal labor-hour costs.

    Latency cost

    Latency cost = average wait time × unit-time cost

    It can also be written as:

    Latency cost = (number of timed-out tasks × loss per timeout) + (average wait time × labor waiting cost)

    Fields you can collect:

    • Average first-token latency.
    • Average end-to-end latency.
    • P95 latency.
    • Timeout rate.
    • Compensation, payout, or manual-intervention cost after a timeout.

    When complete historical data isn’t available early on, you can approximate using internal interviews and labor-hour costs, then continuously calibrate during subsequent runs.

    6. Common Pitfalls and Cautions

    1. Looking only at unit price while ignoring success rate. If a low-price model’s first-pass rate is too low, retries and human review push costs up, and the conclusion may be completely reversed.
    2. Ignoring long output and verbose context. The longer the output, the more latency and token consumption beyond the unit price; in multi-turn conversations, continuously accumulating history also makes cost grow linearly with the number of turns.
    3. Underestimating incident cost. The impact of wrong output often goes beyond a single call’s fee, potentially causing compliance, brand, complaint, and trust losses.
    4. Directly extrapolating business effect from leaderboard scores. Leaderboard language, task types, and scoring measures are inconsistent with business, so localized validation is required.
    5. Treating one evaluation as final. Model versions, prices, prompts, and data distributions all change, so evaluation needs to be periodic and versioned.
    6. Metrics can be “gamed.” Too narrow a task set, evaluation-data leakage, or deliberate tuning can inflate UI/$. Task sets should stay diverse and be updated regularly.
    7. Ignoring cost-estimation bias. Incident cost, latency cost, and value conversion are often subjective, so keep range estimates and periodically check whether assumptions still hold.

    7. Conclusion

    The value of “Useful Intelligence per Dollar” isn’t in giving models a new ranking, but in shifting the evaluation perspective from “whose score is higher” to “how much value one dollar delivers.”

    For developers, it provides an actionable accounting framework: putting retries, human review, latency, operations, and incident costs into the same ledger, and making decisions with data rather than intuition.

    For enterprise managers, it provides a shared language for talking with the business: budget maps to delivery volume, effect maps to reliability, and scale maps to marginal efficiency.

    For individual users, rather than chasing “number one on the leaderboard,” it’s better to try several tools and compare which one can get the job done right in one shot on your real tasks. Which model or platform ranks first changes with time and scenario, but the pursuit of “getting valuable work truly done with less money” stays constant. This is also the fundamental reason the industry is moving from “intelligence contests” to “value competition.”