How to Evaluate New LLMs: A Practical Guide to Cost, Speed, and Capability Testing

1. Background: Look Beyond Benchmark Scores

New model releases are coming faster than ever—new versions appear almost every quarter. The industry has largely settled on a three-tier structure: flagship models handle complex reasoning, advanced coding, and deep research; lightweight models take on simple tasks like classification and rewriting at rock-bottom prices; and the “mid-tier” models in between carry the bulk of everyday production workloads.

So when a new model ships, the questions engineering teams really need to answer aren’t “how much did the benchmark scores improve?” but three far more practical ones:

• For the same tasks, how much lower is the cost?
• How much faster is the response time?
• Does quality stay above the usable threshold?

All three can be quantified. This article lays out a reusable evaluation workflow: first, understand the product tiering and pricing structure, and translate official specs into a cost model for your own business; second, build a minimal evaluation set to measure latency, throughput, and quality; finally, turn the results into tiered routing and gradual rollout strategies. Whether you’re calling an LLM API from any provider—such as Baidu AI Cloud or other cloud platforms—this method applies universally.

2. Understanding Product Tiers: What Mid-Tier Models Are For

A typical three-tier structure divides the work like this:

• Lightweight: lowest price, fastest response—ideal for “per-second, high-volume” tasks such as text classification, field extraction, and short-text rewriting;
• Mid-tier: usually priced at about half of the flagship, noticeably faster, aimed at high-volume production tasks where each call carries little individual value;
• Flagship: highest price, highest capability ceiling—reserved for complex reasoning, complex code, and deep research.

Two pricing conventions are worth knowing: input and output tokens are billed separately, with output typically more expensive than input; and mid-tier pricing is often set at around 50% of flagship pricing as the volume tier.

The vendor’s positioning language for mid-tier models is also worth reading closely. When official materials emphasize “visual understanding, precise operations, and naturally fluent writing,” they’re essentially targeting high-throughput scenarios: reading receipts, entering data into business systems, interpreting diagrams and tables, generating drafts, and batch summarization. These tasks share a common trait—they’re worth doing with AI, but too expensive to run on a flagship model. A new mid-tier model opens up exactly the gap that was previously stuck on cost.

3. Translating the Spec Sheet into a Cost Model

When you get a new model’s spec sheet, focus on these items: context window, maximum output tokens, input/output price per million tokens, rate limits, default reasoning effort, and whether streaming output and function calling are supported.

The formula for per-call cost is straightforward:

cost_per_call = (input_tokens / 10^6) × input_price
              + (output_tokens / 10^6) × output_price

Let’s run through an example with hypothetical numbers (for illustration only). Assume the flagship is priced at 3 for input and 15 for output (per million tokens), and the mid-tier at 1.5 and 7.5. A typical task uses about 4,000 input tokens and 1,000 output tokens:

• Flagship per-call cost: 4000/10⁶ × 3 + 1000/10⁶ × 15 = 0.012 + 0.015 = 0.027
• Mid-tier per-call cost: 4000/10⁶ × 1.5 + 1000/10⁶ × 7.5 = 0.006 + 0.0075 = 0.0135

Exactly half. Multiply by your business volume—say, 10,000 calls per day across 22 working days per month—and the annualized difference becomes a meaningful budget line.

Two common pitfalls:

1. Verify the baseline behind “50% cheaper.” The advertised discount may be relative to the previous mid-tier model or the current flagship—two very different claims. Always plug in the real prices and calculate it yourself.
2. Output-heavy tasks are more sensitive to output pricing. In long-form generation, output tokens are usually the biggest cost driver; judging by input price reductions alone will seriously mislead you.

4. The Three Must-Test Metrics: Latency, Throughput, Quality

Paper calculations aren’t enough—you need real measurements. Three core metrics:

• TTFT (time to first token): how long users wait before the response starts; the first-experience metric for interactive scenarios;
• Throughput (tokens/s): the dominant factor in wall-clock time for batch jobs;
• Quality: sample 20–50 real cases from your business as a golden evaluation set, define scoring rubrics and key-field accuracy; use LLM-as-judge for initial screening and human spot checks for calibration.

A minimal sampling script (pseudocode):

import time, statistics
def sample(model_id, prompts, rounds=5):
    ttft_list, tps_list = [], []
    for prompt in prompts:
        for _ in range(rounds):
            t0 = time.perf_counter()
            first_at, n_tokens = None, 0
            for chunk in llm_client.stream(
                model=model_id,
                messages=[{"role": "user", "content": prompt}],
                temperature=0,
            ):
                if first_at is None:
                    first_at = time.perf_counter() - t0
                n_tokens += 1
            total = time.perf_counter() - t0
            ttft_list.append(first_at)
            tps_list.append(n_tokens / total)
    return {
        "ttft_p50": statistics.median(ttft_list),
        "tps_p50": statistics.median(tps_list),
    }

Experimental discipline matters too: fix the prompts and sampling parameters, run multiple times and take the median rather than the best single run, record the exact model version, and test streaming and non-streaming separately. Pay special attention to differences in default reasoning effort—if the old and new models ship with different default levels, a direct comparison isn’t fair.

5. Two Ledgers: Cost and Time

The cost ledger answers “how much can we save”: multiply the real token distribution from your own call logs (not official examples) by the new prices to get a conservative estimate of monthly savings.

The time ledger answers “how much faster can we get”: a 3x throughput gain means the same batch job finishes in a third of the wall-clock time; lower TTFT means a better experience in interactive scenarios. For batch work, you can manage inputs and outputs with generic cloud capabilities like message queues and object storage when running long jobs; for interactive work, always enable streaming output.

Other general levers for cutting cost and latency include: prompt caching for repeated prefixes, batch APIs for non-real-time tasks, explicit caps on maximum output length, and adjusting reasoning effort according to task difficulty.

6. Tiered Routing: Match Each Task to the Right Tier

Picking a single model is a false choice; the right approach is routing:

def route(task):
    if task.type in {"structured extraction", "format conversion", "document summarization", "copy drafting"}:
        return MODEL_MID_TIER          # low risk, high repetition → mid-tier
    if task.type in {"complex coding", "multi-step reasoning", "deep research"}:
        return MODEL_FLAGSHIP          # high difficulty → flagship
    return MODEL_MID_TIER              # default to mid-tier
def run_with_fallback(task):
    resp = call(route(task), task)
    if quality_score(resp) < THRESHOLD:
        resp = call(MODEL_FLAGSHIP, task)   # quality fallback: escalate and retry
    return resp

Alongside this, do three things: track cost and quality pass rates by task type; set a threshold for the escalation rate (a sudden spike means your routing rules need fixing); pin specific model versions in production for reproducibility, then periodically evaluate new versions to decide whether to upgrade.

7. Four Limitations to Know Upfront

1. Mid-tier models have a capability ceiling. For multi-step long-horizon reasoning, complex architecture design, and deep research, the risk of quality degradation is significant—don’t force a budget model onto your core pipeline.
2. Benchmarks don’t equal business performance. Third-party leaderboards have task distributions very different from your business; always rely on your own evaluation set.
3. Default settings hide costs. A high default reasoning effort can significantly inflate output token counts and latency—configure it explicitly on the calling side.
4. Long context doesn’t mean long context is used well. It’s a well-known phenomenon that recall degrades when key information sits in the middle of the context; important content should be placed at the beginning or end, or processed in chunks.

8. A Rollout Checklist

• Map out your task inventory and call volume distribution; identify the top token-consuming scenarios;
• Estimate average input/output tokens from real logs and calculate the old-vs-new cost difference;
• Build a golden evaluation set and complete the scoring;
• Measure baselines for TTFT, throughput, and quality;
• Switch low-risk, high-repetition tasks first, and run a canary release for a week;
• Set up cost and quality alerts, with escalation and rollback plans in place.

To sum it up in one sentence: “cheaper” and “faster” only matter if the model can still do the job. Turn this evaluation into a standard process, and every time a new model ships, your team can produce a quantified “switch or not” verdict within a day or two—worth far more in the long run than any single selection decision.