LLM Selection in Practice: Three Tables to Evaluate a New Model's Real Value

Whenever a new large language model (LLM) ships, the information typically arrives in a fixed sequence: benchmark screenshots go viral first, then the official pricing page lands, and finally—if you are unlucky—users start complaining about rate limits and unexpected quota drains in community forums. Many teams follow the same order when making decisions: get excited by the scores, get tempted by the price, and then pay for availability problems in production.

A more rational reading order is the reverse: check the pricing table first, then the capability table, and spend the most energy on availability. The first two tables answer “Is it worth trying?”; the third answers “Can I trust it in production?” Below is a detailed walkthrough of all three tables, with evaluation methods and code you can take straight to your next review.

Table 1: The Economics of Pricing — Money Before Scores

1.1 Pricing Is Three Numbers, Not One

Most mainstream LLM API pricing today is split into three tiers: cache-hit input price, cache-miss input price, and output price. Focusing on a single “input X per million tokens” number will badly misestimate real cost, for three reasons:

• Prompt caching is now standard practice. Repeated prefixes—system prompts, long documents, few-shot examples—often cost a fraction of the miss price when cached (typically 2× to 10× cheaper). In production, repeated prefixes frequently account for more than half of input tokens, so actual bills can diverge sharply from the list price.
• Output is significantly more expensive than input. Each autoregressively generated token requires a full forward pass with no caching benefit. For leading models, output price is typically 3× to 6× the miss-input price.
• Output length is the model’s own choice. Two models tackling the same task can differ in verbosity, and the chatty one’s bill may be double the concise one’s. You must pair the price table with an estimate of average output tokens per task.

A quick estimation formula:

Cost per call ≈ [hit_tokens × P_hit + miss_tokens × P_miss + output_tokens × P_out] / 1e6

1.2 Long Context: Capability and Cost Amplifier

Million-token context windows sound like a capability spec; in practice they are a billing lever. Take a ~300k-token document (a multi-hundred-page bid, contract set, or codebase slice) with two rounds of Q&A. Using illustrative unit prices (hit 1.5, miss 15, output 60, per million tokens):

• Round 1 (miss): 300,000 × 15 / 1e6 ≈ 4.5
• Round 2 (hit, within cache TTL): 300,000 × 1.5 / 1e6 ≈ 0.45
• Two rounds of output, 10k tokens total: 10,000 × 60 / 1e6 ≈ 0.6

Total per task: roughly 5.5. For “read-the-whole-document-then-cross-check” workloads, price gaps between flagship models can be several times. When capabilities converge but prices diverge, the pricing table becomes the first thing to study.

1.3 Reading Generational Price Drops

Recent flagship releases share a pattern: parameter counts climb (trillion-scale), context windows stretch (million-token), yet unit prices drop roughly 30%. This usually stems from mature inference batching and KV-cache reuse, larger models spreading unit serving costs, and competitive pressure. For architects, the generational price gap is more than a savings signal—it reshapes architecture choices:

• When “full-context direct read” shifts from expensive to affordable, the traditional “chunk → retrieve → stitch” pipeline can simplify to “long-context direct read + citation grounding,” halving the pipeline and halving the failure surface.
• When output prices fall, “ask the model to reason one more step and emit a structured intermediate result” becomes affordable, raising the cost-effectiveness of chain-of-thought externalization and self-check rounds.

1.4 A Copy-Paste Cost Estimator

def estimate_call_cost(input_tokens: int, output_tokens: int, hit_rate: float,
                       p_hit: float = 1.5, p_miss: float = 15.0,
                       p_out: float = 60.0) -> float:
    """Rough cost per call (in yuan). Prices are per million tokens."""
    hit = input_tokens * hit_rate
    miss = input_tokens * (1 - hit_rate)
    return (hit * p_hit + miss * p_miss + output_tokens * p_out) / 1e6

During selection reviews, run this with real task token distributions (input length, repeated-prefix ratio, expected output length) to get a per-task expected cost rather than comparing list prices.

Table 2: Capability Benchmarks — Rankings Are a Screening Tool, Not a Verdict

2.1 The Three-Layer Benchmark Structure

• General leaderboards (aggregated scores from authoritative third-party evaluators, arena-style blind tests): good for a 30-second check of “Is this model in the top tier?”—and nothing more.
• Vertical task boards: code repair, math reasoning, tool-use/terminal tasks (multi-step agent scenarios), long-context retrieval. Your business usually lands in only one or two of these cells.
• Custom evaluation sets: 20–50 samples drawn from your real task distribution. This is the only gold standard—and the only table in this article nobody can fill in for you.

2.2 When the Gap Is Under 1 Percentage Point, Treat It as No Gap

A clear trend in recent releases: top-tier flagships have converged to within 1 percentage point on terminal-task benchmarks. At that scale, sampling temperature, random seeds, and dataset contamination (training data leakage) can each swing scores by ±1 point. In other words, when the gap falls inside the benchmark noise band, the ranking between two models is essentially meaningless, and any “Model X beats Model Y” narrative is marketing, not a conclusion. Accept the “capability convergence” premise and move evaluation down to your custom set: track first-attempt accuracy, format compliance rate, and the share of outputs needing human correction.

2.3 Generational Improvements Matter More Than Absolute Rank

On the same leaderboard, a vendor’s new flagship scoring 10+ points above its predecessor is far more informative than “who ranked where.” Leading models have long passed the “more parameters = higher score” stage; generational leaps typically come from post-training pipeline upgrades: verifiable-reward RL, large-scale synthetic data, tool-use curricula, and multimodal alignment. Treat “trillion-parameter” as the vessel and post-training quality as the substance.

2.4 The Practical Value of Native Multimodality

“Native vision understanding” deserves its own line item. In document scenarios, critical information lives in charts, scanned pages, and layout structures. Historically this required chaining “OCR → layout analysis → text model”—three stages, each an information-loss and failure point. Native multimodality collapses three stages into one, raising both the accuracy ceiling and engineering simplicity.

Table 3: Availability — Most Overlooked, Most Likely to Cause Trouble

3.1 Availability Is Not Uptime; It’s a Bundle of Engineering Parameters

Production availability hinges on: RPM/TPM limits, quota window type (fixed vs. rolling), concurrency caps, max context length, timeout behavior, and the structure of 429/5xx error codes. These parameters make or break long-running, high-concurrency, and multi-agent scenarios.

3.2 Rolling Windows and Long Tasks Are Natural Enemies

A rolling window (e.g., “tokens consumed in the past 5 hours count toward the current quota”) is friendly for short requests and hostile for long tasks. A job expected to run 40 minutes may hit the quota wall at minute 25—the first 25 minutes of tokens are wasted, the quota stays occupied, and retries must wait for the window to slide. Community complaints about “tasks dying mid-run, retries still fail” often trace back to this.

3.3 The Quota Multiplier Effect in Multi-Agent Systems

What looks like “one task” can internally fan out into N sub-agents, each carrying its own system prompt and each re-reading shared context—multiplying token consumption by tens of times. Sub-agents often launch in millisecond-scale bursts, instantly hitting TPM limits and triggering cascading 429s. Engineering defense checklist:

• Pre-flight budget check: estimate token budget before submission, apply a 1.2×–1.5× safety factor, and queue rather than let tasks die mid-stream.
• Client-side token bucket: don’t rely on server-side rate-limit signals; throttle yourself.
• Exponential backoff with jitter: prevent all workers from retrying in sync and creating periodic spikes.
• Checkpoint-based resumption: split long tasks into staged checkpoints so failures recover from the last good point.
• Quota monitoring: alert when window-remaining quota drops below 80%.

import time, threading
class TokenBucket:
    """Client-side rate limiter that gates requests by estimated consumption."""
    def __init__(self, capacity: float, refill_per_sec: float):
        self.capacity, self.refill = capacity, refill_per_sec
        self.tokens, self.ts = capacity, time.monotonic()
        self.lock = threading.Lock()
    def acquire(self, n: float):
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity,
                                  self.tokens + (now - self.ts) * self.refill)
                self.ts = now
                if self.tokens >= n:
                    self.tokens -= n
                    return
                wait = (n - self.tokens) / self.refill
            time.sleep(min(wait, 0.5))

3.4 What Compensation Policies Reveal About Engineering Maturity

After a rate-limit incident, responsible vendors typically reset affected quotas and offer pro-rata compensation. This reflects a post-incident attitude, but production users should be clear: compensation fixes sentiment, not SLA breaches. Before onboarding, audit: Are rate-limit docs public and unambiguous? Is there burst headroom? Is there an enterprise quota channel? Are error codes structured for programmatic handling? Whether you connect via a public cloud API or through a private/managed deployment on a platform like Baidu AI Cloud, GPU capacity planning, elastic scaling, and monitoring all belong on the same availability table and should be validated during load testing.

Merging the Three Tables: Switch to “Task Economics”

Four actionable metrics:

1. Expected cost per task = direct call cost ÷ task success rate (with retry costs amortized).
2. First-attempt accuracy: the share of outputs that pass without manual correction.
3. Average rework rounds: average interactions from first output to sign-off.
4. End-to-end latency distribution: P50/P95, and it must include queuing time from rate limits.

Suggested review flow: Day 1—define 20–50 real-task samples. Day 2—run each under cold and hot (cache-hit) conditions, logging cost, accuracy, and rework rounds. Day 3—inject faults by throttling the client to simulate rate limits, observe checkpoint recovery and retry behavior. Then fill the decision table:

Candidate Model	Cost per Task	First-Attempt Accuracy	Rework Rounds	P95 Latency	Failure Rate Under Throttling
A	5.5	82%	1.4	6 min	3%
B	21.0	84%	1.3	4 min	1%

This is what a procurement decision should actually look like. Candidate B is more accurate and faster but roughly 4× the price; candidate A delivers comparable volume at a fraction of the cost under the “capabilities have converged” premise. Teams will weight differently, but at least the decision happens in a shared coordinate system.

Selection Checklist

Pricing Table

• Are all three unit prices (hit-input / miss-input / output) publicly documented?
• Is cache TTL and hit-detection logic (prefix matching) clearly specified?
• Have you computed “expected cost per task” using real task token distributions?

Capability Table

• Does the gap on your target benchmark exceed the noise band (≥3 percentage points for meaningful ranking)?
• Does the improvement over the previous generation signal a generational leap?
• What are first-attempt accuracy and rework rounds on your custom evaluation set?
• Is multimodality native (eliminating the need for an external OCR pipeline)?

Availability Table

• Are RPM/TPM, concurrency, and max-context limits publicly documented?
• What are the quota window type, capacity, and recovery cadence?
• Are 429/timeout error codes structured for programmatic handling?
• Have checkpoint resumption and idempotency been validated under load?
• Is there burst headroom, an enterprise quota channel, and defined SLA terms?
• Are historical incidents and compensation policies publicly recorded?

Closing Thoughts

Benchmarks answer “Can it do it?” Pricing answers “Can you afford it?” Availability answers “Can you stake production on it?” The intersection of all three tables is the selection conclusion. When flagship capabilities converge inside benchmark noise but prices diverge by multiples, the evaluation framework must shift—from a “who’s leading” narrative to “task economics.” Pin this to the first page of your selection review: check the pricing table first, then the capability table, and finally—stare down the availability table until you’re satisfied.