How to Evaluate New-Generation LLMs: Capability, Cost, and Availability

Why the Old Yardstick No Longer Works

The release cadence of large language models has accelerated noticeably. Every time a new flagship model launches, community discussion is dominated by the same narrative: parameter count, context window length, position on some leaderboard, and how many points behind the strongest closed-source model it is. That narrative had real value in the early days — capability gaps between models were orders of magnitude apart, and ranking essentially determined usability.

Today, however, the public benchmark gaps between top models have generally narrowed to single-digit percentage points, sometimes falling within evaluation error margins. At the same time, price competition has stretched the unit-price gap between models in different tiers to several-fold. This implies two things:

1. Rankings carry less information. When the gap is smaller than the evaluation error margin, “fourth place” and “second place” are statistically indistinguishable.
2. Cost and supply stability carry more weight. When one model completes the same task at a fraction of the total cost of another, “how close is it to the top” is no longer the question engineering teams care about most.

So when a new model launches, adjust your reading order: check the price list first, then the capability table, and finally the service status announcements a few days after launch. The first two tables determine whether it is worth trying; the third determines whether you dare run it in production. The rest of this article follows that order.

The First Table: Capability Benchmarks — for Screening, Not Final Judgment

How to Use Public Benchmarks Properly

Comprehensive leaderboards and specialized benchmarks published by third-party evaluators (covering areas such as terminal tasks, code generation, and long-context understanding) still have value, but treat them as a screening tool, not a final verdict. Keep three points in mind:

• Look at gaps, not ranks. A 0.7-percentage-point difference and a 7-percentage-point difference are entirely different conclusions. The former should be read as “same tier”; only the latter constitutes a generational gap.
• Look at sub-scores, not totals. Aggregate leaderboards smooth out structural differences. If your workload is code-heavy, look only at the code sub-scores; if it centers on long documents, look only at the long-text sub-scores. A model with a high total score may well lag on your core scenario.
• Watch out for data contamination. Questions from mainstream benchmarks may already be in training corpora, so high scores may reflect memorization rather than ability. One way to check: see whether the model’s performance on newer, more niche benchmarks is consistent with its scores on the older ones.

Build Your Own Evaluation Set: Turn “My Tasks” into a Benchmark

There is always a gap between public benchmarks and real business needs: benchmarks measure “what models are generally good at,” while the business asks “can my tasks actually get done.” The only way to close this gap is to build your own evaluation set. Key practices include:

1. Align the task profile: the input length, output format constraints, and domain terminology distribution of your samples should match production traffic;
2. Stratified sampling: stratify by difficulty and task type so the evaluation set is not diluted by easy samples;
3. Reproducible scoring: turn anything that can be written as a unit test into assertions; for the rest, use structured rubrics or multi-judge voting, and fix random seeds;
4. Refresh regularly: the evaluation set itself needs protection against contamination — core samples should be checked for similarity against public data.

The Second Table: Pricing Structure — Convert Unit Prices into Task Bills

Break Token Pricing into Four Dimensions

A model service’s list price is usually not a single number but a set of parameters. Before selecting a model, break out at least these four:

• Separate input/output pricing: output tokens typically cost several times more than input tokens, because generation is an autoregressive process computed token by token;
• Cache-hit discounts: repeated prompt prefixes (system prompts, few-shot examples, long documents) can be billed at the cache-hit rate, which is often an order of magnitude lower than the miss rate;
• Context-length tiers: some services charge higher unit prices for extra-long context requests;
• Batch and off-peak discounts: offline batch jobs can usually get significantly lower unit prices.

Long-Context Tasks Are Highly Sensitive to Input Pricing

Consider a typical task: read a long document of several hundred thousand tokens, cross-check the data once, and produce about 20,000 tokens of conclusions. For tasks like this, input tokens often exceed output tokens by more than 10x, so the bill is dominated by input. Discounts on the input price are therefore “amplified” to the task level, and engineering optimizations that raise the cache-hit rate (fixing the prompt prefix, placing document content up front) deliver savings well beyond expectations.

Run the numbers with a set of purely illustrative figures (not real pricing from any product): suppose the miss price is ¥10 per million input tokens, the cache-hit price is ¥2 per million input tokens, and output costs ¥60 per million tokens. The task has 300,000 input tokens and 20,000 output tokens:

• All misses: 0.3 × 10 + 0.02 × 60 = ¥4.2 per round
• 80% prefix hits: 0.06 × 10 + 0.24 × 2 + 0.02 × 60 = ¥2.4 per round, a reduction of about 43%

Then there is a multiplier that is often overlooked: rework rounds. Real tasks rarely pass on the first try. If a task takes 2 rework rounds on average, the actual bill is 3x the single-round cost. So when evaluating a model, the real cost formula is:

Task cost = per-round token bill × average rounds × number of concurrent tasks

A Sample Snippet: Turn List Prices into a Cost Estimator

M = 1_000_000
def task_cost(input_tokens, output_tokens, rounds=1,
              p_in=10, p_out=60, p_cache=2, cache_ratio=0.0):
    """Estimate the token bill of a multi-round task (in CNY; prices are illustrative)"""
    hit = input_tokens * cache_ratio
    miss = input_tokens - hit
    per_round = (miss / M * p_in            # missed (uncached) input tokens
                 + hit / M * p_cache        # cache-hit input tokens
                 + output_tokens / M * p_out)  # output tokens
    return per_round * rounds
# Example: 300k input tokens (80% cache hit) + 20k output tokens, 2 rounds on average
print(task_cost(300_000, 20_000, rounds=2, cache_ratio=0.8))

Plug each candidate model’s list prices into this function and run it against the same batch of real task samples. The comparison you get is far closer to what the decision actually needs than any leaderboard ranking.

The Third Table: Availability — Problems That Only Surface After Launch Day

Typical Failure Modes

In the early days after a new model launches, the capability table and the price table are already fixed, but the third table is only beginning to be written. Three failure modes are common:

1. Rate limits and quota windows: some subscriptions or plans impose rolling-window caps on request frequency or token usage. A long job that runs for hours can hit a window reset point, invalidating all the computation done so far;
2. Runaway consumption: in multi-agent or looped-call scenarios, each subtask carries the full context, creating a multiplier effect on token consumption. A task split across 8 concurrent sub-agents can consume 5–10x a single task (illustrative magnitude), amplifying both the bill and quota usage;
3. Capacity crunches: traffic surges during the launch period cause latency jitter and rising timeout rates. This has nothing to do with model capability, yet it directly determines whether tasks are delivered on time.

An Engineering Checklist

These problems cannot be solved by “switching to another model” — they require engineering safeguards:

• Usage observability: major model service platforms (including Baidu AI Cloud and other cloud providers) generally offer usage dashboards and quota management. When integrating a platform, “can it track consumption per project and per task, with alerting” should be a hard checklist item;
• Retries with exponential backoff: back off and retry on rate-limit errors, adding random jitter so that multiple clients retrying in sync do not trigger an avalanche:

import time, random
def call_with_backoff(invoke, max_retries=5, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return invoke()
        except RateLimitError:
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)
    raise RuntimeError("Max retries exceeded; moving task to the fallback queue")

• Checkpoint and resume: persist intermediate results stage by stage for long jobs, so that hitting a quota window means resuming from a checkpoint instead of starting over;
• Fallback routing: when the primary model is rate-limited, automatically switch to a secondary model for the remaining steps, trading some quality to protect delivery deadlines;
• Off-peak scheduling: run batch jobs during traffic troughs to avoid competing for quota at peak times.

Evaluate Launch Day and the First 30 Days Separately

A pragmatic approach: on launch day, run only small-scale validation, and observe rate-limit trigger frequency, error rates, and latency percentiles for at least one to two weeks before scaling up. Benchmark scores are a one-day result; supply stability is a one-quarter result.

Combining the Three Tables into One Framework

Putting it all together, teams are advised to adopt a three-step evaluation process:

1. Define the task profile: measure the input/output token distribution, concurrency peaks, and fault-tolerance requirements of production traffic, and clarify “what counts as completing a task”;
2. Establish a cost baseline: pick a set of canary tasks with known correct answers, run them on each candidate model at its list prices, and record per-task cost, first-pass rate, and average rework rounds;
3. Small-traffic canary rollout: route 5%–10% of real traffic and observe rate-limit hits, timeout rates, and manual correction volume for one to two weeks before scaling up.

What goes into the final selection report should not be “ranked Nth on a leaderboard” but four numbers: per-task cost, first-pass rate, average rework rounds, and rate-limit trigger count. These four numbers answer the four questions business stakeholders care about most: how much it costs, how often it is right, how many rounds of rework it takes, and whether it can run to completion reliably.

Conclusion

The right way to follow a new model launch comes down to one sentence: check the price first, then the capability, and wait a few days to check service status. When capability gaps between models have shrunk to within error margins while price gaps have widened to several-fold, the center of gravity of evaluation inevitably shifts from “chasing capability” to “unit task cost and delivery quality.” Leaderboards tell you where a model’s ceiling is; bills and alerts tell you whether it can enter your production environment. For engineering teams, the latter is where decision-making starts.