LLM Model Selection in Practice: A Two-Dimensional Evaluation of Intelligence Index and Unit Cost

    Reading benchmark scores as a leaderboard is a dangerous simplification: a single score hides the failure modes, retries, and context bloat that dominate real production costs. This article argues for evaluating large language models on two dimensions — an intelligence index that compresses capability into a comparable number, and unit task cost measured per successful task rather than per call. It walks through cost-per-success calculation, an anonymized comparison dataset, and Pareto-frontier analysis showing why "great value" is not the same as "optimal," along with knee-point and endpoint reasoning. It then covers evaluation-set design for reproducibility, difficulty-tiered routing with a failure-escalation loop, common cost-reduction techniques and their trade-offs, and the metrics a monitoring dashboard should track, closing with a checklist of common selection pitfalls.

    1. Why an “11-Point Benchmark Gap” Is Not Enough to Drive a Selection Decision

    Reading a standardized benchmark result as a leaderboard is the most common — and most dangerous — simplification. A leaderboard gives you a scalar: a score. What production environments actually care about is a vector: capability, unit cost, latency distribution, stability, context-length ceiling, and operability. When a new model’s intelligence index is 11 points below a leading domestic model and 22 points below an overseas frontier benchmark, yet its per-task cost is only one-eighth to one-tenth of the latter, that conclusion is nearly useless in an engineering context: it tells you neither which task types the score gap appears on, nor whether the lower score can be recovered through prompt engineering, retrieval augmentation, or fine-tuning.

    More importantly, a leaderboard score rests on two assumptions that usually don’t hold:

    1. Comparable capability: the same score can correspond to completely different failure modes across models. One model might lose 3 points on long-chain reasoning, another 3 points on instruction-following and format stability — and the damage to a production system is entirely different.
    2. Comparable cost: the “price per 1K tokens” on a price sheet does not equal per-task cost. Retries, context bloat, failed reruns, and redundant output can multiply the actual cost several times over.

    So the first step in selection is not to find the “strongest model,” but to build an evaluation coordinate system that expresses capability and cost at the same time.

    2. A Two-Dimensional Framework: Intelligence Index and Unit Task Cost

    2.1 Intelligence Index: Compressing Multidimensional Capability into a Single Comparable Number

    An intelligence index usually comes from the weighted pass rate of a standardized task suite — for example, categories such as code generation, multi-step reasoning, long-document summarization, structured output, and tool calling. Each category gets a score from 0 to 1, which is then weighted by business importance and mapped to a 100-point scale. It solves the “cross-comparison” problem, but to be trustworthy it must satisfy three conditions:

    • The task distribution matches real traffic, not the public leaderboard;
    • The scorer is reproducible — rule-based checks first; model-based scoring needs a fixed version and random seed;
    • Report percentiles, not just averages — especially P50/P90 pass rates, since long-tail failures are often the real source of production incidents.

    2.2 Unit Task Cost: An Often-Underestimated Metric

    Per-task cost should be defined as “the total cost consumed to complete one deliverable task,” not the cost of a single call. An illustrative calculation, including duplicate billing from retries:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class Trial:
    4. score: float # 0~1, score of a single attempt
    5. input_tokens: int
    6. output_tokens: int
    7. retries: int
    8. price_in: float # price per 1K input tokens
    9. price_out: float # price per 1K output tokens
    10. def unit_cost(t: Trial) -> float:
    11. tokens_in = t.input_tokens * (1 + t.retries)
    12. return (tokens_in * t.price_in + t.output_tokens * t.price_out) / 1000
    13. def cost_per_success(tasks: list[Trial]) -> float:
    14. """Priced per success — a stricter and more realistic view for high-failure-rate models"""
    15. total = sum(unit_cost(t) for t in tasks)
    16. success = sum(1 for t in tasks if t.score >= 1.0)
    17. return total / success if success else float("inf")

    cost_per_success is a better basis for decisions than unit_cost: a model with a cheap per-call price but a low pass rate — needing an average of 1.8 retries to succeed — may end up costing 1.5x a model that is cheaper per token and passes on the first try.

    2.3 An Anonymized Illustrative Dataset

    The table below shows illustrative results for a set of standardized tasks (data anonymized, used only to explain the method, and not representative of any specific product):

    Candidate Intelligence index Unit task cost Relative position
    Overseas frontier ~57 High Capability ceiling reference
    Leading domestic model ~46 Medium-high First tier
    New model under test X 35 ~1 Lower bound of the low-cost region
    Comparison model Y 36 Lower than X Dominator on the Pareto frontier
    Comparison model Z ~36 ~8 Clearly dominated
    Comparison model W ~33 ~10 Worse on both dimensions

    Two conclusions follow directly. X has roughly the same capability as Z and W but costs an order of magnitude less, showing that on this standardized suite X already has substitution value. Meanwhile, X is 11 points behind the leading domestic model and 22 points behind the overseas frontier, showing that catching up is still underway and that the gap is compressing in the latter half — which is exactly what “stunning cost-performance but still short of the top” fully means.

    3. The Pareto Frontier: Why “Great Value” Is Not the Same as “Optimal”

    On the “cost–capability” plane, if one candidate is no worse than another on both metrics and better on at least one, the latter is dominated. The polyline connecting all non-dominated candidates is the Pareto frontier.

    1. def pareto_front(points: list[tuple[float, float]]) -> list[tuple[float, float]]:
    2. """points: [(cost, score)]; returns front points sorted by ascending cost"""
    3. ordered = sorted(points, key=lambda p: (p[0], -p[1]))
    4. front, best_score = [], float("-inf")
    5. for cost, score in ordered:
    6. if score > best_score:
    7. front.append((cost, score))
    8. best_score = score
    9. return front

    Against the table above, X is not on the frontier, because Y achieves a higher score at a lower cost — X is completely dominated. This is an easily overlooked conclusion: “good value” is relative to higher-priced models in the same score band; from a global view, X is merely a “low-cost candidate close to the frontier,” sitting at the lower boundary of the high-value region. If selection relies only on marketing claims like “cheaper at the same score,” it’s easy to fall into the dominated region.

    Two special points on the frontier deserve attention:

    • Knee point: the point of maximum curvature on the frontier, above which spending more yields sharply diminishing score gains — usually the sweet spot for general business use;
    • Endpoints: the cheapest candidate (suitable for high-volume, low-risk coarse-filtering tasks) and the strongest candidate (suitable for high-value, low-tolerance tasks). Endpoints are often not covered by a single model, but instead serve as inputs to a routing strategy.

    4. Evaluation Set Design: Making Conclusions Reproducible

    An evaluation system that can support decisions should include the following elements:

    • Task sampling aligned to the real distribution: sample by the proportions of online intent categories, not uniformly;
    • Difficulty stratification: label each task with a difficulty level to support tiered routing later;
    • Failure-mode classification: distinguish format errors, factual errors, missed instructions, over-length truncation, tool-call failures, and so on — more instructive than a total score;
    • Contamination and version-drift protection: keep a private task set that never enters any public corpus, pin the model version and evaluation-script hash, and rerun quarterly to detect metric drift from silent upgrades;
    • Cost observability: log token usage, retry count, and end-to-end latency on every call, and aggregate them into a “cost per successful task” dashboard.

    5. Implementation: Tiered Routing and Cost Governance

    The end product of two-dimensional evaluation is not “picking one model” but “picking a combined strategy.”

    5.1 Difficulty-Tiered Routing

    Put low-cost models that aren’t on the frontier on the high-traffic, low-difficulty path, and reserve the high-capability frontier endpoint for hard cases:

    1. TIERS = [
    2. {"max_difficulty": 1, "model": "small-fast"}, # formatting, extraction, classification
    3. {"max_difficulty": 2, "model": "mid-balanced"}, # summarization, general Q&A
    4. {"max_difficulty": 3, "model": "large-strong"}, # multi-step reasoning, complex code
    5. ]
    6. def route(task: dict) -> str:
    7. for tier in TIERS:
    8. if task["difficulty"] <= tier["max_difficulty"]:
    9. return tier["model"]
    10. return TIERS[-1]["model"]

    This needs a paired escalation loop: if a small model’s output fails the validator (e.g., JSON parsing fails or an assertion doesn’t hold), automatically escalate to a stronger model and rerun, instead of returning an error directly. This raises the overall average cost, but “cost per successful task” usually drops.

    5.2 Common Cost-Reduction Methods and Their Trade-offs

    • Semantic caching: vector-nearest-neighbor matching for highly repetitive requests, returning cached results on a hit. You need a similarity threshold and a TTL to avoid caching stale information.
    • Context compression: rerun-then-truncate retrieved results and summarize conversation history — input tokens are often the bulk of the cost.
    • Output constraints: use structured output or schema constraints to cut redundant text, directly compressing output tokens.
    • Batching: use batch endpoints for offline tasks, trading latency for unit price.

    5.3 Metric Definitions

    A monitoring dashboard should include at least: cost per request, cost per successful task, escalation rate, cache hit rate, P50/P95 latency, and difficulty-stratified capability pass rate. Only by putting capability and cost on the same dashboard and comparing them over the same time window can you catch problems like “a silent model upgrade doubled the cost” or “prompt bloat made capability look better while cost spiraled out of control” in time. With a combination of general-purpose cloud capabilities — model serving, container platforms, object storage, logging, and monitoring/alerting — you usually don’t need to build the entire instrumentation and dashboard stack from scratch; the general-purpose capabilities offered by platforms such as Baidu AI Cloud can serve as a reference for implementation.

    6. A Checklist of Common Pitfalls

    1. Substituting unit price for per-task cost, ignoring retries and context bloat;
    2. Looking only at averages, not P90 pass rates or the failure-mode distribution;
    3. Treating “cheaper at the same score” as automatically “optimal,” ignoring Pareto dominance;
    4. Building the evaluation set to mirror the public leaderboard, so conclusions fail on private traffic;
    5. Doing one-off selection only, with no version-drift monitoring or periodic reruns;
    6. Routing everything to the strongest model for peace of mind, and leaving cost governance until after the bill arrives.

    7. Conclusion

    A benchmark’s “how many points behind, how many times cheaper” only has decision value once it is placed back into the two-dimensional capability–cost coordinate system. The standard for whether a low-cost model is worth adopting is not whether it’s cheaper than the high-priced model, but whether it sits near the Pareto frontier, whether the knee point falls within the business’s acceptable capability range, and whether — combined with tiered routing and a failure-escalation loop — it can push “cost per successful task” below the target line. For most businesses, the pragmatic path is: let a low-cost model carry the bulk of the traffic, let the strong model cover hard cases and high-value scenarios, and use a reproducible evaluation set and cost dashboard to continuously calibrate this dividing line — this is both the optimal solution under the current value window and an architecture that won’t need to be rebuilt as model capabilities keep catching up.