Evaluating LLMs: The Three Tables of Capability, Pricing, and Availability

Whenever a new flagship Large Language Model (LLM) is released, official materials and media coverage tend to focus on the same metrics: parameter scale, context length, and benchmark rankings. However, for developers actually integrating these models into production systems, the priority order for evaluating a release should be reversed: look at the pricing table first, then the capability table, and finally scrutinize the part that providers often acknowledge needs improvement—availability. Together, these three tables form a complete evaluation coordinate system. Let’s break them down in this order.

1. Pricing Table: From “Cost per Million Tokens” to “Total Cost per Task”

1.1 Three-Tier Pricing is the Industry Standard

Currently, the list prices for mainstream model services are typically split into three tiers: input (cache hit), input (cache miss), and output. Cache hits are significantly cheaper because the provider reuses the previously computed KV Cache from earlier requests, bypassing most of the compute overhead in the prefill phase and passing those savings on to the user. The ratio between hit and miss prices typically ranges from one to two orders of magnitude. This means: the higher the context reuse rate, the better the bill looks.

1.2 The Economics of Long Context

Million-token-level context brings a direct change: “reading the entire document at once” becomes viable. A hundreds-of-pages proposal or a medium-sized code repository can be placed entirely into the context without chunking and retrieval. However, input costs scale linearly with the number of tokens, so whether this approach is cost-effective depends entirely on the pricing structure. The engineering best practice is to place long documents into the cache, allowing subsequent multi-turn Q&A and cross-checking to hit the cache price. The more questions asked against the same long context, the lower the amortized cost. Of course, long context does not equal long memory; information recall from the middle of ultra-long documents still requires structured summaries or targeted prompts as a fallback.

1.3 The Effective Cost Formula

Unit price is just the baseline of cost. The actual bill must be multiplied by several amplifying factors:

Effective Cost ≈ Single Call Cost × Call Rounds × (1 + Retry Rate + Rework Rate)

This is especially evident in Agentic workflows: a single task might be broken down into dozens of model calls, each carrying the system prompt and historical trajectory. Failed retries and reworks due to substandard results can make the actual expenditure several times higher than the “ideal single-pass” estimate. Therefore, when selecting a model, the real question isn’t “how much per million tokens,” but rather: “to complete a real task, how much does it cost, how many times does it fail, and how many rework rounds are needed?”

Here is a directly applicable cost estimation function:

def task_cost(input_tokens, cached_tokens, output_tokens,
              price_hit, price_miss, price_out,
              rounds=1, retry_rate=0.0):
    """Single task cost estimation under three-tier list pricing (price in currency/million tokens)"""
    uncached = max(input_tokens - cached_tokens, 0)
    once = (cached_tokens / 1e6 * price_hit
            + uncached  / 1e6 * price_miss
            + output_tokens / 1e6 * price_out)
    return once * rounds * (1 + retry_rate)
# Example: 300k token proposal (250k cacheable) + 2 review rounds + 10% retry rate
# task_cost(300_000, 250_000, 20_000,
#           price_hit=1, price_miss=10, price_out=60,
#           rounds=2, retry_rate=0.1)

Feeding the real tasks of various candidate models into this function yields a “cost per task” metric that is truly comparable horizontally. When the price gap between top-tier models reaches a multiple of several, the evaluation coordinate system should shift: stop asking “how far behind is it,” and start asking “for the same task, whose bill is thinner?”

2. Capability Table: Read the Benchmarks, but Read Them Correctly

2.1 When Score Differences are Smaller Than Noise, Rankings are Meaningless

Evaluation results inherently carry randomness: sampling temperature, tool-calling success rates, and execution environment differences can all cause the same model’s score to fluctuate between two evaluations. As a rule of thumb, when the score difference between two models on a similar benchmark narrows to within 1 percentage point, they are statistically indistinguishable and insufficient for making a selection decision. The correct approach is to run the same batch of tasks multiple times and report the mean and standard deviation, rather than citing a single run’s score.

2.2 Benchmark Capability ≠ Your Task Capability

Aggregate leaderboards reflect “average capability,” but business tasks often follow a long-tail distribution: perhaps 90% of requests are simple, and the remaining 10% determine success or failure. General leaderboards also have two known limitations—high scores might stem from overlap between the evaluation set and training data (contamination), as well as model overfitting to common question types. Therefore, it is essential to build a custom evaluation set (Golden Set): extract 50 to 200 representative tasks from real business operations, label the expected results, and use the pass rate from this set to gauge relevance to your system.

A minimal viable evaluation framework:

import statistics
def run_eval(tasks, runner, n_runs=5):
    """Run evaluation n times and report mean and variance to avoid being misled by a single run"""
    rates = []
    for _ in range(n_runs):
        rates.append(sum(runner(t).ok for t in tasks) / len(tasks))
    return {"mean": statistics.mean(rates),
            "stdev": statistics.stdev(rates),
            "worst": min(rates)}

2.3 Shifting the Evaluation Coordinate System

When multiple flagship models enter the “good enough” threshold in capability but their prices differ by multiples, the primary question for evaluation shifts from “how far has it caught up” to “where do its success rate, rework rounds, and comprehensive cost stand on my specific tasks?” The former is news; the latter is what selection is all about.

3. Availability Table: The Easiest to Crash, and the One to Discuss Upfront

3.1 Rate Limiting and Quotas are Part of Product Capability

Availability constraints for LLM services typically include concurrency limits, Requests/Threads Per Minute (RPM/TPM), and rolling window quotas by the hour or multi-hour. In the early stages of a new model release, capacity is tight, leading to paid users being rate-limited or long tasks hitting window limits halfway through—a recurring scenario in the industry. For the caller, these constraints are as critical as accuracy and price. A model with high benchmark scores but rate-limiting at critical moments is effectively unavailable to a production system.

3.2 Multi-Agents are Quota Consumption Amplifiers

In a multi-sub-agent architecture, each sub-task carries its own independent context and is billed independently. Parallelism instantly amplifies the request volume. If the client also stacks a naive “retry on failure” logic without proper backoff, it will form a retry storm, further amplifying abnormal consumption. Engineering requires four types of guardrails:

• Budget Circuit Breaker: Set a token budget for each task; abort and alert if exceeded.
• Exponential Backoff + Jitter: Increase retry intervals by 2^n with random jitter to avoid synchronized storms.
• Degraded Routing: Abstract a model gateway layer to automatically switch to a backup model when the primary one is rate-limited.
• Idempotency and Caching: Merge and reuse identical requests, and cache results to reduce redundant calls.

import time, random
class BudgetExceeded(Exception): ...
def call_with_guard(client, messages, *, max_retries=4, budget=None):
    spent = 0
    for attempt in range(max_retries):
        resp = client.chat(messages)          # Pseudo-code: generic model service interface
        spent += resp.usage.total_tokens
        if budget and spent > budget:
            raise BudgetExceeded(f"{spent} > {budget}")
        if resp.ok:
            return resp
        time.sleep(2 ** attempt + random.random())  # Exponential backoff + jitter
    raise RuntimeError("Retry quota exhausted")

3.3 Write Availability into Contracts and Scorecards

During selection, proactively confirm: whether rate limit thresholds can be increased, whether a service status page and semantic documentation for error codes are provided (handling strategies for 429 vs. 5xx are completely different), and what the compensation mechanism is during incidents (common practices include quota resets or proportional subscription credits). The completeness of these terms often reflects the engineering maturity of the provider better than launch presentation materials.

4. Synthesizing the Three Tables into One Decision Matrix

It is recommended to compress the three dimensions into a total score using the following weights (adjustable based on business needs):

Dimension	Example Weight	Observation Metrics
Task Success Rate	40%	Custom eval set pass rate, rework rounds
Cost per Task	30%	Currency/task (including retries and rework)
Availability	20%	Rate limit trigger frequency, error rate, P99 latency
Migration Cost	10%	API compatibility, toolchain adaptation effort

For implementation, it is recommended to first run a 5% canary traffic for one to two weeks, concurrently logging error rates, rework rounds, and billing curves, before deciding on a full rollout. Whether the candidate is an open-weight model self-deployed, or an LLM service hosted on cloud platforms like Baidu AI Cloud, this workflow of “calculating costs first, benchmarking second, and stress-testing availability last” applies equally.

Conclusion

No matter how information-dense a model launch event is, it is merely the starting point for evaluation. The mature team’s approach is to deconstruct marketing materials into three engineering tables: the pricing table answers “is it worth it,” the capability table answers “can it do it,” and the availability table answers “is it stable.” Once all three are in place, “how to evaluate a new model” is no longer a question of impression, but an engineering problem that can be quantified, reproduced, and written into a decision document.