How to Evaluate New LLMs: Look at Total Cost, Not Just API Pricing

    Every time a new-generation large language model ships, the debate splits between capability and API price. This guide argues that comparing unit prices alone leads to misleading conclusions, because the number that actually matters is total cost of ownership. It introduces a four-dimensional evaluation framework—capability, performance, cost, and engineering/compliance—and shows how to build a private evaluation set, score it reproducibly, compute per-call and unit-effective-delivery costs from real token usage, and run production-like load tests. It also covers engineering practices that cut cost without sacrificing quality, such as tiered routing, caching, context compression, and canary rollouts. The goal is to help developers and teams decide whether a new model is worth upgrading by measuring quality-adjusted cost rather than headline pricing.

    Every time a new-generation large language model is released, the technical community splits into two camps. One asks whether the model is closing the gap with the top tier; the other watches the API price and debates whether it “got more expensive again.” Both concerns are legitimate: model capability shapes the product experience, and inference cost determines whether a business can scale. But when the argument stays at the level of “which unit price is bigger,” it tends to produce one-sided—sometimes even opposite—conclusions.

    With the common billing model, mainstream LLM services price input tokens and output tokens separately, and output is usually priced higher than input. So even when a new version lowers its unit price, actual spend can still rise because contexts grow longer and reasoning traces produce more tokens; conversely, a higher-looking unit price can still lower the cost per unit of effective delivery if better quality reduces rework.

    This article does not evaluate any specific product. Instead, it lays out a practical method for evaluating and selecting LLMs that helps developers and teams answer three questions: Is the new model worth upgrading to? Where does the extra cost come from, and where is it justified? How do you keep inference cost at a sustainable level?

    1. First, Fix the Mindset: Unit Price Is Not Cost

    The first principle is to separate price from cost. Price is the rate published by the provider; cost is the total price the business actually pays. A simplified formula connects them:

    1. total cost = Σ (input tokens × input price + output tokens × output price) × (1 + retry factor) + hidden costs

    The retry factor captures duplicate calls caused by timeouts, rate limiting, and format-validation failures. Hidden costs include evaluation maintenance, prompt tuning, human fallback, log storage, monitoring and alerting, and version migration.

    Consider a hypothetical example. In a long-document summarization scenario, each request sends about 300K input tokens and produces about 1,500 output tokens. In a short-Q&A scenario, each request sends about 500 input tokens and produces about 500 output tokens. Assume the two model versions have the same output price, the input price has dropped, and we compare only the cost of the model calls themselves:

    • Short-Q&A scenario: input tokens are a small fraction of the bill, so the new version’s price change barely moves total cost.
    • Long-document scenario: input tokens dominate the bill, so a lower input price directly thins out the largest chunk.

    If the new model generates a more detailed intermediate reasoning trace during long-document inference, output volume can rise noticeably—and total cost may go up rather than down. The conclusion is clear: judging unit price in isolation from the scenario easily leads to the wrong call.

    2. A Four-Dimensional Evaluation Framework

    To make evaluation results reproducible and comparable, break the assessment into four dimensions. Set a metric, a measurement method, and a weight for each, then roll them up with weights that reflect business priority:

    Dimension Core question Key metrics Measurement
    Capability Does it get it right? Task pass rate, key-point coverage, structured-output compliance Automated scoring on a private eval set + human spot-checks
    Performance Does it run smoothly? Time to first token, per-token output latency, end-to-end latency, throughput, error rate Production-like concurrency load testing, focused on P95/P99
    Cost Can we afford it? Cost per request, unit effective delivery cost Reconciled from token usage in call logs
    Engineering & compliance Can we govern it? API stability, observability, rate limits and quotas, content safety, data protection Trial run, architecture review, compliance review

    Capability

    Public leaderboards are useful for a first-pass screen, but they cannot replace business validation, for three reasons:

    1. Scenario mismatch: benchmark questions differ in data distribution from real business traffic.
    2. Scoring mismatch: multiple-choice auto-grading is not the same as judging the quality of open-ended generation.
    3. Data contamination risk: a model may have seen some public questions, inflating its score.

    Key points for evaluating capability:

    • Core abilities: knowledge Q&A, long-document understanding, logical reasoning, code generation, instruction following, and multi-turn dialogue. Instruction following and structured output (e.g., fixed-format JSON) matter especially, because they determine whether downstream systems can parse results reliably.
    • Long-context effectiveness: don’t just look at the advertised context length. Embed key information near the upper limit of the text and then ask about it, to check whether the model can accurately retrieve and use it.
    • Robustness: test with samples containing typos, polysemous words, ambiguous references, and adversarial prompts, and watch whether the model gets led astray or hallucinates.

    Performance

    In interactive scenarios that users can perceive, time to first token (TTFT) determines “how long until something happens after I click,” while per-token output latency (TPOT) determines “how fast the text streams.” In batch scenarios, throughput and cost per token matter more. Many teams measure only average latency, yet timeouts are frequent in production—because tail latency (P95/P99) under high concurrency is far higher than the mean.

    Cost

    Beyond explicit call fees, account for hidden costs:

    • Extra calls from failed retries;
    • Repeated retries when input truncation is handled poorly;
    • Maintenance of post-processing code caused by unstable output formats;
    • Added human review to close a quality gap.

    At low volume these costs are negligible; at a million calls a day, they often matter more than the difference in unit price.

    Engineering & compliance

    • Engineering: is the API style compatible with existing code, are error codes clear, is the rate-limiting policy explicit, and can it emit usage and latency metrics?
    • Compliance: does it let you opt out of training-data use, is data encrypted in transit, can sensitive information in logs be masked, and is the content-safety filtering robust?
    • Stability: after a version update, does output behavior drift noticeably?

    3. Step 1: Build Your Own Evaluation Set

    1. Sampling and labeling

    Sample production logs in strata that follow the business distribution, covering high-frequency, long-tail, and edge-case scenarios. Aim for at least 50–100 cases per core scenario, each containing the input, a reference answer or scoring points, and hard constraints (content that must appear or must not appear).

    2. Structured storage

    1. {
    2. "id": "kb-001",
    3. "scenario": "knowledge base Q&A",
    4. "input": "(user question, anonymized)",
    5. "reference": "reference answer or scoring points",
    6. "must_include": ["keyword A", "keyword B"],
    7. "forbid": ["prohibited content"],
    8. "output_format": "json",
    9. "max_chars": 300
    10. }

    3. A sample scorer

    1. import json
    2. from typing import Callable
    3. # call_llm: your own wrapper that calls an LLM (takes a prompt, returns text)
    4. CallLLM = Callable[[str], str]
    5. def case_score(case: dict, output: str) -> float:
    6. """Score a single case against hard constraints (0-100)"""
    7. score = 0.0
    8. # Key-point coverage: each matched required item earns 60/n points
    9. must = case.get("must_include", [])
    10. if must:
    11. score += 60 * sum(k in output for k in must) / len(must)
    12. # Forbidden content: any hit fails the case outright
    13. forbid = case.get("forbid", [])
    14. if any(k in output for k in forbid):
    15. return 0.0
    16. # Output format: validate JSON parses when required
    17. if case.get("output_format") == "json":
    18. try:
    19. json.loads(output)
    20. score += 20
    21. except json.JSONDecodeError:
    22. pass
    23. else:
    24. score += 20
    25. # Length constraint
    26. if len(output) <= case.get("max_chars", 1_000_000):
    27. score += 20
    28. return score
    29. def evaluate(dataset: list[dict], call_llm: CallLLM, pass_line: float = 80) -> dict:
    30. """Dataset-level evaluation: returns average score, pass rate, and failure list"""
    31. scores, failures = [], []
    32. for case in dataset:
    33. score = case_score(case, call_llm(case["input"]))
    34. scores.append(score)
    35. if score < pass_line:
    36. failures.append({"id": case["id"], "score": round(score, 1)})
    37. return {
    38. "avg_score": round(sum(scores) / len(scores), 2),
    39. "pass_rate": round(sum(s >= pass_line for s in scores) / len(scores), 3),
    40. "failures": failures,
    41. }

    4. Cross-model comparison

    Run the same cases, the same prompt template, and the same generation parameters (temperature, etc.) against each candidate model, and record the raw outputs. Break scoring into four sub-dimensions, each with its own weight:

    • Correctness: against the reference answer or scoring points;
    • Faithfulness: whether it fabricates content;
    • Completeness: whether it covers all sub-questions;
    • Format compliance: whether it meets the structured-output requirement.

    5. Sample size must be large enough to support a decision

    Small differences on a small sample may be just noise. Let the pass rate be p; over n repeated trials, the half-width of the 95% confidence interval is about 1.96 * sqrt(p(1-p)/n). For example, with n=100 and p=0.90, the half-width is about 0.059—so the true pass rate is, with 95% confidence, within about 6 percentage points of the observed value.

    So a model that scores 92% versus 90% on 100 cases is not necessarily stronger. Either increase the sample size or skip manual scoring. Structured tasks can be validated directly in code; run them concurrently via a container platform or batch jobs, and reaching thousands of samples is straightforward.

    6. Human spot-checks and consistency

    Automated scoring can only cover quantifiable hard constraints. For subjective dimensions such as fluency and professional accuracy, spot-check at a fixed ratio by hand. With multiple reviewers, compute inter-rater agreement to make sure the results are reliable.

    4. Step 2: Compute Per-Call Cost and Unit Effective Delivery Cost

    1. Measure real usage from logs

    More important than unit price is the measured average input and output token volume. Record each request’s token usage, latency, and result status at the API gateway or call-wrapper layer, then aggregate weekly percentiles—only then do you get representative cost inputs.

    2. Cost calculation code

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class PriceCard:
    4. input_per_m: float # input price per 1M tokens
    5. output_per_m: float # output price per 1M tokens
    6. @dataclass(frozen=True)
    7. class Usage:
    8. input_tokens: int
    9. output_tokens: int
    10. def request_cost(price: PriceCard, usage: Usage, retry_rate: float = 0.0) -> float:
    11. """Cost of a single call, including retries"""
    12. base = (usage.input_tokens * price.input_per_m
    13. + usage.output_tokens * price.output_per_m) / 1_000_000
    14. return base * (1 + retry_rate)
    15. def effective_cost(price: PriceCard, usage: Usage,
    16. pass_rate: float, retry_rate: float = 0.0,
    17. human_cost: float = 0.0) -> float:
    18. """Unit effective delivery cost: total cost paid per acceptable output"""
    19. if not 0 < pass_rate <= 1:
    20. raise ValueError("pass_rate must be in (0, 1]")
    21. # Default handling cost for failed cases is 0; add human review cost per your business
    22. return (request_cost(price, usage, retry_rate) + (1 - pass_rate) * human_cost) / pass_rate

    3. Worked example: how a quality gap changes the conclusion

    Assume the following (illustrative, to show the method):

    • Model A: 0.006 per call, 90% pass rate;
    • Model B: 0.010 per call, 98% pass rate.

    Without human fallback:

    • Model A’s unit effective delivery cost is about 0.0067;
    • Model B’s is about 0.0102.

    Here Model A is cheaper. But if the business requires a pass rate of at least 95%, Model A’s failed cases need human handling at about 0.5 each. Then Model A’s unit effective delivery cost is about (0.006 + 0.1 × 0.5) / 0.9 ≈ 0.062—roughly six times Model B’s.

    That is the point of “unit effective delivery cost”: it folds the quality gap into cost, so you are not misled by the headline unit price.

    4. Don’t let hidden costs slip through

    Retry cost is easy to underestimate. Let the per-call failure rate be f, with independent retries and immediate retry after each failure until success. The expected number of calls is 1/(1-f), so cost inflates by about 1/(1-f):

    • 5% failure → about +5.3%;
    • 10% failure → about +11.1%;
    • 30% failure → about +42.9%.

    If some failed cases can never pass no matter how many retries, you must add fallback cost on top—human review or a downgrade to a backup model.

    5. Step 3: Quantify Performance with Production-Like Load Testing

    1. import time
    2. from concurrent.futures import ThreadPoolExecutor
    3. def percentile(values: list[float], p: float) -> float:
    4. """Compute a percentile using linear interpolation"""
    5. values = sorted(values)
    6. pos = (len(values) - 1) * p
    7. lo = int(pos)
    8. hi = min(lo + 1, len(values) - 1)
    9. return values[lo] + (values[hi] - values[lo]) * (pos - lo)
    10. def benchmark(call_llm: CallLLM, prompts: list[str], concurrency: int) -> dict:
    11. """Stepped-concurrency load test: returns throughput and latency percentiles"""
    12. latencies = []
    13. def run(prompt: str) -> None:
    14. start = time.perf_counter()
    15. call_llm(prompt)
    16. latencies.append(time.perf_counter() - start)
    17. wall_start = time.perf_counter()
    18. with ThreadPoolExecutor(max_workers=concurrency) as pool:
    19. # map returns results in input order and waits for all requests to finish
    20. list(pool.map(run, prompts))
    21. elapsed = time.perf_counter() - wall_start
    22. return {
    23. "concurrency": concurrency,
    24. "throughput(QPS)": round(len(prompts) / elapsed, 2),
    25. "P50_latency": round(percentile(latencies, 0.50), 3),
    26. "P95_latency": round(percentile(latencies, 0.95), 3),
    27. "P99_latency": round(percentile(latencies, 0.99), 3),
    28. }
    29. # Ramp concurrency and watch for the throughput knee and tail-latency growth
    30. # for c in (1, 5, 10, 20, 50):
    31. # print(benchmark(call_llm, prompts, c))

    Additional notes:

    1. Run load tests in a network environment close to production, and cover rate-limiting and timeout handling. If the function has no internal retry or exception handling, failed requests will break the test outright—so add both before a formal run.
    2. Match input lengths to the real distribution, and test requests near the context-length limit separately.
    3. When possible, record time to first token and total latency separately at the call-wrapper layer, to distinguish “slow because queued” from “slow because generating.”

    6. Engineering Practices That Cut Cost Without Cutting Quality

    1. Tiered routing

    Let rules, a lightweight classifier, or a small model handle simple requests, and send complex requests to the large model.

    1. from typing import Literal
    2. ModelTier = Literal["small", "large"]
    3. def route(text: str, meta: dict) -> ModelTier:
    4. # Long input or a high-difficulty task goes to the large model
    5. if len(text) > meta.get("long_threshold", 8_000):
    6. return "large"
    7. # Explicitly declared complex task types
    8. if meta.get("task_type") in meta.get("hard_tasks", {"complex_reasoning", "code_generation"}):
    9. return "large"
    10. # Low classifier confidence: conservatively hand off to the large model
    11. if meta.get("intent_confidence", 1.0) < meta.get("confidence_threshold", 0.8):
    12. return "large"
    13. # Everything else (simple requests) is handled by the small model
    14. return "small"

    Run offline replays against the routing rules and measure the misroute rate. Misrouting a simple request as complex only adds cost; misrouting a complex request as simple directly hurts the experience.

    2. Result caching and semantic caching

    Build a cache for high-frequency questions with relatively stable answers (product docs, FAQs). Beyond exact matches, use semantic similarity to hit similar questions. The cache key should include the model version and key generation parameters; after a model upgrade, invalidate the cache wholesale to prevent mixing old and new outputs.

    3. Context engineering and prompt compression

    • Inject only relevant fragments via retrieval augmentation instead of the whole document.
    • Summarize long material up front, then hand it to the model to reason over.
    • Constrain output format with structured templates to reduce redundant explanations and the retries caused by parse failures.
    • Periodically measure prompt token usage and prune long, unchanging background blocks.

    4. Batching and async

    For non-real-time tasks—offline summarization, batch labeling, content classification—use batch APIs or an async task queue. This uses throughput better and more easily qualifies for batch-pricing discounts (subject to the provider’s actual policy).

    5. Observability and cost governance

    • Wire up logging and monitoring/alerting; break down token usage and spend by business line, scenario, and tenant, and set budget alerts.
    • Cap input length, output length, and per-user call frequency on the caller side to prevent abnormal calls from inflating cost.

    6. Data security and deployment form

    For sensitive data, prefer services that don’t use data for training and that encrypt data in transit and at rest; when necessary, evaluate private or dedicated-instance deployment. When self-hosting inference, beyond hardware depreciation, count operations headcount and resource utilization. If average GPU utilization is only 30%, the effective unit cost multiplies by about 3.3x.

    7. Small-traffic canary and version rollback

    Before a new model goes live, canary a small share of traffic and run a shadow comparison of the old and new versions in parallel: call both for the same request and compare pass rate and cost. Keep a one-click rollback to the old version so the business isn’t stuck when a new version regresses.

    7. Implementing Evaluation in a Platform Environment

    You can implement the methods above efficiently by leveraging general cloud capabilities on a platform—for example, the relevant product categories offered by Baidu AI Cloud:

    • Elastic compute and container platform: run evaluation scripts and load-test programs, scaling on demand;
    • Batch computing: run large evaluation sets in parallel;
    • Object storage: hold evaluation sets, raw outputs, and annotation results;
    • Message queue: decouple async inference from result write-back;
    • Logging, monitoring/alerting, and data visualization: power token-usage and cost dashboards;
    • Key management, access control, and content safety: protect credentials and ensure output compliance.

    These are general capability categories; see each product’s official documentation for specific features. The goal of building this toolchain is to standardize and automate evaluation, moving model selection from “gut feeling” to “data.”

    8. Conclusion: Back to “Does It Fit?”

    The community debate over whether a new model is “expensive” is really a mismatch between expectation and scenario. A price change may be a minor shift for a short-conversation app but a multiple-fold difference for long-document or deep-reasoning apps. Rather than argue over which number is more expensive, answer three questions:

    1. Quality: how much did the new model’s pass rate improve on my core scenario?
    2. Cost: after conversion to unit effective delivery cost, did it go up or down?
    3. Experience and risk: can latency, stability, and compliance meet the business’s requirements?

    Whether you call a general-purpose LLM service or self-host inference on a platform such as Baidu AI Cloud, what ultimately decides success is not the number on the price sheet but “unit effective delivery cost.” Teams should accumulate three assets: a private evaluation set that keeps iterating, a cost model built on real token usage, and an inference architecture that covers tiered routing and caching. Then, whatever new model appears, you can give a rational, data-backed judgment quickly.