Don't Pick LLMs by Leaderboard Alone: Cost and Use Case Matter More

    Leaderboard scores are a weak basis for production LLM selection. This article explains ranking limitations, how price gaps dominate when quality is close, how to route cheap vs. capable models, which capability/cost/ops/risk metrics to record, how to score image generation, how to build an internal eval set, how repeated context inflates spend, and how to decide with cost per valid result.

    1. Why leaderboards are a limited signal

    Many teams start model selection with leaderboards, parameter counts, and overall scores. In production, outcomes more often depend on call cost, latency, reliability, context handling, error types, and fit to the actual workflow.

    One internal estimate cited here covered 44,205 mainstream use cases at about 31,000 yuan. The top two models scored 95.4 and 95.3, while input and output prices differed by nearly 20x. The scoring scheme, sample mix, and statistical definition were not fully public, and a 0.1-point gap without a confidence interval may not be statistically meaningful. The numbers are used only to illustrate evaluation method, not as an authoritative ranking.

    Overall scores help form a high-level view; they should not replace enterprise selection, for three reasons.

    First, tasks demand different capabilities. A model that shines at complex reasoning may be unstable at structured output or format following. A slightly lower overall score may still be better for support summaries, ticket classification, and bulk extraction.

    Second, averages hide uneven skill. A high mean across ten task types can still hide two weak areas. If the business depends on those two, the mean is nearly useless.

    Third, public tests differ from real data. Spoken language, missing fields, duplicates, errors, internal jargon, and permission boundaries often do not appear in standard items.

    Leaderboards answer “how strong is this model in general.” “Does it fit our system” needs your own data.

    2. When scores are close, price dominates

    If two models are similar in quality, price becomes the main decision factor.

    Estimate per-call text cost as follows. Prices are usually per million tokens, so divide by 1,000,000.

    Per-call cost = input tokens × input unit price ÷ 1,000,000 + output tokens × output unit price ÷ 1,000,000

    Monthly cost = average daily requests × per-call cost × days in the month

    Do not multiply billed tokens again; they are already in per-call cost.

    At low trial volume, price gaps may look small. At tens of thousands of requests per day, they hit infrastructure budget.

    A worked example uses hypothetical unit prices, not a vendor quote. Assume 100,000 requests per day, 3,000 input tokens and 500 output tokens each. Model A: 1 yuan per million input tokens, 4 yuan per million output. Model B: 0.05 and 0.2 yuan respectively—20x cheaper.

    Model A per-call cost: 3000 × 1 ÷ 1,000,000 + 500 × 4 ÷ 1,000,000 = 0.005 yuan. About 500 yuan/day, ~15,000 yuan/month (30 days).

    Model B per-call cost: 0.00025 yuan. About 25 yuan/day, ~750 yuan/month.

    This assumes identical input/output tokens, cache hit rates, and billing rules. Longer outputs or repeated history change the gap.

    Ask:

    • How many input and output tokens does a typical request use?
    • Does output often run long?
    • Must you resend history or documents?
    • Can you cache or reuse results?
    • What is the business cost of one error?
    • Does model price match business value?

    3. Cheap models and strong models fit different work

    Low-cost models usually win on bulk, simple, high-concurrency work:

    • text classification
    • keyword extraction
    • content deduplication
    • ticket routing
    • simple summarization
    • standard template generation
    • large-scale preprocessing

    Goals are clear, output structure is stable, and error bounds are known. Rules, retries, and sampling can contain risk.

    Work that needs stronger reasoning includes complex code review, multi-step analysis, high-stakes decision support, long-document cross-checking, constrained plan generation, and professional Q&A that must cite grounds. Errors are expensive; a stronger model is usually justified.

    A reasonable pattern is different models for different requests, not one model for everything.

    4. Layered model routing

    Route by difficulty, risk, and business value. A common three-layer split:

    Layer 1 — low complexity. Fixed format, clear goal, short output. Prefer cheaper models. Prompt templates and field checks keep output stable.

    Layer 2 — medium complexity. Some context understanding, low-stakes conclusions. Use a balanced model, with retries and result scoring.

    Layer 3 — high complexity. Multi-step reasoning or high accuracy. Use a stronger model, cap call rate and context length, and add programmatic or human review.

    Signals: request length, task type, number of constraints, citation requirement, historical failure rate, user tier, business risk, expected output length.

    Sketch:

    If the task is fixed-format extraction, call a low-cost model;
    else if it needs complex reasoning or is high-risk, call a high-capability model;
    else call a balanced model.
    If format checks fail, retry once; if still failing, escalate one layer.

    After routing is live, add three safeguards.

    Fallback: on rate limits, timeouts, or outages, switch to a backup model and log the event.

    Monitoring: track volume, failure rate, average latency, and unit cost per layer; alert owners on anomalies.

    Cost circuit breaker: set a daily budget per line of business; near the cap, drop routing tier or pause non-critical traffic.

    This is usually cheaper than always using the top model and more reliable than always using the cheapest.

    5. Eval reports should capture four metric families

    A useful report covers capability, economics, engineering, and risk.

    Category Main metrics Notes
    Capability accuracy, task completion, format adherence, factual consistency, runnable code rate, complex-reasoning success quality on target tasks
    Economics input price, output price, average per-call cost, long-context cost, bulk-job total spend real spend
    Engineering average latency, time to first token, timeout rate, rate-limit frequency, availability, success after retry production behavior
    Risk sensitive-content handling, wrong-answer rate, hallucination tendency, over-permission answers, leak risk, output uncontrollability potential loss

    A single overall score cannot tell “stable across the board” from “strong in a few areas, weak elsewhere.” Each metric needs a collection method, refresh cadence, and pass line.

    6. Image generation needs its own rubric

    Text models can be scored with answer match, human raters, or programmatic checks. Image generation is not “does it look good.” Score five dimensions:

    1. Prompt adherence (25%): subject, count, action, scene, style. For multiple objects, is anything missing?
    2. Detail quality (25%): anatomy, hands, text, object edges, lighting, perspective. Pretty images often hide errors.
    3. Style consistency (15%): color, character traits, and look across a series.
    4. Usability (20%): size, background, subject padding, text regions—can design use it in production?
    5. Cost and speed (15%): cost per image, generation time, retry rate, batch throughput.

    Automatic metrics help pre-filter large batches; humans judge aesthetics, composition, and commercial usability. Combining both reduces single-metric bias.

    7. Build an internal, real-workflow test set

    Public evals are reference only. Decisions should rest on your data, built from real processes, not a handful of ad-hoc questions.

    Split by: high-frequency, high-value, high-risk, long-text, edge/anomaly, multi-turn, structured output, and image generation/edit.

    Each class needs scoring rules. For extraction, check both field correctness and format consistency. For code, check runnability plus error handling, resource cleanup, and security.

    Use a one-page scorecard per class: task type, sample size, scoring items, weights, acceptance threshold. Set thresholds before testing so they are not moved after seeing results.

    Desensitize data. Do not send real PII, secrets, or internal sensitive material to external services.

    8. Hidden cost of repeating context

    Teams often watch unit price and miss context billed again and again. In multi-turn chat, long-document analysis, and RAG, history can be charged as input every time.

    Suppose every request carries the same 20,000-token document, 10,000 calls. With no cache, input is billed at full price. With 80% cache hits billed at 10% of list price, input cost is about 28% of the uncached figure: 0.8 × 0.1 + 0.2 × 1 = 0.28.

    Cache has quality risk. If the source document updates but cache keeps the old version, answers use stale material. Version cache entries and invalidate on source change.

    Other optimizations:

    • periodic summaries of history
    • keep only fragments relevant to the current question
    • separate fixed instructions from dynamic content
    • cap useless output length
    • retrieve long documents in segments
    • stop generation once the result is good enough

    These cut spend and context noise, which often stabilizes answers.

    9. Decide with cost per valid result

    When scores are close and prices differ a lot, compare cost per valid result:

    Cost per valid result = total call cost ÷ number of valid results

    A valid result must pass programmatic format checks and business acceptance (sampled human review or business rules). Format-only is not enough; well-formed answers can still be wrong.

    Example. Model A: 5,000 yuan week, 10,000 outputs, 8,500 valid → ~0.59 yuan each. Model B: 800 yuan, 10,000 outputs, 7,600 valid → ~0.11 yuan each.

    Here B is cheaper on both unit price and cost per valid result. If B’s valid rate were 60% (6,000 valid), unit cost would rise to ~0.13 yuan, still below A. Only if valid rate falls far enough, or human review cost jumps, does the expensive model win. Verify with your data.

    Two rules of thumb: if a cheap model already meets the required valid rate on the target task, it is usually the better choice. If a stronger model clearly cuts review, reduces business errors, or handles hard work, a higher price can still be rational.

    Mature setups are rarely a single model. They combine a real test set, cost monitoring, routing, quality checks, and human fallback. When building on cloud APIs, wire model calls to logs, alerts, storage, and access control, and keep watching live behavior.

    A leaderboard rank only describes a specific test. What matters is whether the model reliably produces valid results on your work at a cost you can sustain.