LLM Evaluation Guide: Beyond Benchmark Scores and Unit-Price Myths
When a major LLM ships a new version, teams face a deceptively simple question: is it better, and is it cheaper? This guide shows why neither benchmark rankings nor headline token prices answer that question reliably, then presents a reusable evaluation framework built on four pillars—model architecture and scale, business-oriented capability testing, total cost of ownership, and engineering deployment. You will learn to separate total from activated parameters, weigh context-length trade-offs, build and version your own evaluation set with error attribution, script cost estimation that accounts for caching, batching, and retries, and design layered model routing with retries, fallback, and observability. It closes with an upgrade decision checklist and the common pitfalls that lead teams to contradictory conclusions.
Every time a leading LLM releases a new version, developer communities circle the same three questions: Is the new version stronger than the last one? Is the API price going up or down? And where do Chinese models stand overall? These discussions reflect real engineering anxiety—models iterate fast, selection is costly, and mistakes are expensive to recover from. Once the debate stays at the level of “who is stronger, who is pricier,” you end up with contradictory conclusions: some reject an upgrade because the unit price moved, while others switch everything based on a single leaderboard score.
From an engineering perspective, the more rigorous question is: Can the new version reliably complete my tasks at a lower cost, within my business constraints (response latency, cost budget, compliance)? Answering it requires weighing four things at once—model architecture, evaluation methodology, cost structure, and engineering deployment. This article builds a reusable evaluation framework around these four points and gives practical methods for each.
1. Architecture and Scale: Parameter Count Is Not Capability, and Capability Is Not Low Cost
Distinguishing “total parameters” from “activated parameters”
Mainstream LLMs now largely adopt the Mixture-of-Experts (MoE) architecture: the model contains many “expert” sub-networks, and for each token only the few experts selected by a routing mechanism take part in computation. This creates two different measures of parameter scale:
- Total parameters determine the model’s capacity ceiling and also determine the weight memory footprint during inference. The more experts there are, the more memory the model must keep resident.
- Activated parameters determine the compute cost of a single forward pass. An MoE model can have a very large total parameter count while keeping per-token computation relatively manageable.
So marketing numbers like “trillion-parameter models” cannot directly imply “more expensive inference” or “cheaper inference.” To judge inference cost, look at the activated parameter scale, context length, output length, and the provider’s pricing strategy. Likewise, parameter scale only sets a capacity ceiling; whether that capacity turns into real capability depends on training data quality and mix, post-training alignment methods, and inference strategy (such as chain-of-thought or multi-sample voting). Many parameters without sufficient training does not guarantee strength; moderate parameters with good data and alignment can also perform reliably in real workloads.
The trade-off between dense and sparse models
Dense models are structurally simple with predictable inference behavior, and under the same activated-compute budget they usually achieve stable performance more easily. Sparse MoE models often achieve stronger overall performance under the same training compute, but engineering-wise you face load imbalance, cross-device communication overhead, and memory pressure. For private deployment, these differences show up directly in hardware selection and cluster utilization; for hosted inference, they show up more in latency stability and price fluctuation.
Context length: longer is not always better
Long context has been a key battleground across the last two model generations, but for developers it is a double-edged sword:
- Compute grows super-linearly with length. The compute cost of the self-attention part grows quadratically with sequence length, while the memory footprint of the KV cache grows linearly. Stuffing dozens of pages of documents into a prompt at once raises both time-to-first-token and cost noticeably.
- Effective utilization is lower than the nominal window. Industry research commonly finds that models use information in the middle of a long context less effectively than information at the beginning and end; key information placed in the middle risks being ignored.
- Retrieval is often more cost-effective. For scenarios like knowledge-base Q&A, first retrieving relevant fragments from documents and then handing them to the model to generate is usually faster, cheaper, and more controllable than “stuffing everything in.”
Open weights vs. hosted services: control vs. convenience
The value of open weights is not that they are “free,” but that they are controllable: you can deploy privately, fine-tune for a vertical domain, and fully control data flow. The cost is that you take on GPU resources, inference optimization, and operations yourself. A hosted API removes the infrastructure burden and keeps model upgrades continuous on the provider’s side, but data passes through an external path and behavioral details are not visible. For most organizations, a reasonable choice is a hybrid architecture: keep sensitive data and high-frequency stable workloads in a private environment, and use hosted services to cover complex long-tail tasks.
A version upgrade is a multi-dimensional change, not a single improvement
A single version update often bundles capability gains, price adjustments, context-window changes, and improved tool-calling stability all at once. A change in any single dimension does not represent the overall direction; you must break it down and verify item by item.
2. Capability Evaluation: Leaderboards Have Boundaries; Your Own Evaluation Set Is the Basis for Decisions
Three limitations of benchmark leaderboards
- Data contamination. Public evaluation sets and their variants may already be in a model’s training data, so performance on these questions is systematically overestimated. Rewriting questions, using private test sets, and focusing on questions published after a data cutoff can partially mitigate this.
- Saturation. Top scores on mainstream benchmarks gradually approach the ceiling, so subtle gaps between models are hard to show on leaderboards—yet they may be magnified in real workloads.
- Distribution mismatch. Leaderboards test general capability, but the business cares about accuracy in a specific domain, compliance with a specific format, and stability at a specific length. The leaderboard winner and the business-optimal model are often not the same model.
Business-oriented evaluation dimensions
It is recommended to cover at least the following dimensions, with independent samples prepared for each:
- Instruction following: whether it strictly obeys format, length, tone, and prohibitions;
- Structured output: the validity rate and field accuracy of formats like JSON and tables;
- Long-context information extraction: locating details in long documents and handling mutually contradictory information across multiple places;
- Tool calling: whether parameter extraction is accurate, whether multi-step call chaining is correct, and whether it can self-correct after an error response;
- Code generation and repair: the proportion passing unit tests in an executable environment;
- Multilingual and dialect: stability across Chinese variants and other languages;
- Safety and refusal: the ability to block harmful requests and the false-refusal rate on legitimate requests.
Building an evaluation set
- Sampling: sample from real online requests stratified by intent, difficulty, input length, and language, ensuring coverage of all kinds of scenarios.
- Labeling: write a reference answer, scoring criteria, and executable checks (such as unit tests and field assertions) for each sample, forming a “gold-standard set.”
- Blind testing: hide the identity of candidate models during scoring, to avoid brand impressions biasing judgment.
- Versioning: automatically re-run after each model upgrade, outputting a per-dimension comparison between old and new versions to form a traceable evaluation report.
An automated evaluation skeleton
The following example implements a minimal evaluation framework using only the standard library: candidate models are plugged in through a unified interface, the scorer selects “execution-based” or “match-based” checks by case type, and finally accuracy, average latency, and token consumption are aggregated by category.
import timeimport unicodedatafrom collections import defaultdictdef normalize(text: str) -> str:"""Unify case, whitespace, and full/half-width forms to reduce misjudgments from formatting differences."""text = unicodedata.normalize("NFKC", text).strip().lower()return " ".join(text.split())def make_judge(case):"""Return a scoring function by case type; it returns 0 or 1."""if case.get("tests"): # code case: run unit tests in a sandboxreturn lambda answer: run_in_sandbox(answer, case["tests"])if case.get("must_include"): # keyword case: all key fragments must be presentkeys = [normalize(k) for k in case["must_include"]]return lambda answer: all(k in normalize(answer) for k in keys)return lambda answer: normalize(answer) == normalize(case["reference"])def evaluate(cases, generate):"""generate(prompt) is a unified model-call interface returning (text, input_tokens, output_tokens)."""stats = defaultdict(lambda: {"total": 0, "passed": 0, "in": 0, "out": 0, "ms": 0})errors = []for case in cases:start = time.perf_counter()answer, in_tok, out_tok = generate(case["prompt"])elapsed = (time.perf_counter() - start) * 1000ok = bool(make_judge(case)(answer))s = stats[case["category"]]s["total"] += 1s["passed"] += oks["in"] += in_toks["out"] += out_toks["ms"] += elapsedif not ok:errors.append({"id": case["id"], "answer": answer})report = {}for name, s in stats.items():report[name] = {"accuracy": round(s["passed"] / s["total"], 4),"avg_latency_ms": round(s["ms"] / s["total"], 1),"avg_in_tokens": round(s["in"] / s["total"], 1),"avg_out_tokens": round(s["out"] / s["total"], 1),}return report, errors # errors are used for manual review and error attribution
After getting the report and error samples, you should perform error attribution rather than staring only at the total score. Common errors fall into five categories:
- Factual errors: answers that contradict the facts;
- Instruction violations: format, length, or prohibitions not obeyed;
- Context omission: details in long documents not used;
- Tool misuse: wrong parameter extraction or confused call ordering;
- Expression issues: vague, verbose, or inappropriate tone.
Different attributions call for different responses: factual problems go to retrieval augmentation, instruction problems go to prompt engineering and post-processing validation, tool problems require redesigning interface descriptions, and expression problems can be targeted with a few examples. Even if the new version’s total score improves, if a critical category of error increases, it may not be suitable for launch.
3. Cost Evaluation: The Unit Price Is Just One Variable
The structure of total cost
Reducing “is it expensive” to “how much per million tokens” ignores several factors that amplify or shrink actual spending:
- Input and output pricing differ, with output tokens usually priced higher per unit; long-text generation applications are especially sensitive to this;
- Context caching: placing fixed prefixes like the system prompt and knowledge background at the start of a request and keeping them stable lets the provider reuse the prefix computation; the hit portion is usually billed at a lower price, and time-to-first-token drops;
- Batch inference: asynchronous batch APIs usually trade a lower price for longer response times, suitable for offline summarization, data cleaning, and similar tasks;
- Context inflation: carrying full history in every dialogue turn makes the context grow rapidly with turns, and cost often grows roughly quadratically;
- Retries and failures: timeouts, rate limits, and content blocks all produce extra calls.
A cost estimation script
from dataclasses import dataclass@dataclassclass PricePlan:input_per_mtok: float # input price, CNY per million tokensoutput_per_mtok: float # output price, CNY per million tokenscached_per_mtok: float = 0.0 # input price for the cache-hit portiondef estimate_cost(plan, calls, in_tokens, out_tokens, *,cache_hit=0.0, cache_price=None, batch_ratio=0.0, retry_rate=0.0):"""Estimate total cost. Prices and cache discounts follow the actual service documentation; retry_rate is the extra retry fraction."""total_in = calls * in_tokens * (1 + retry_rate)total_out = calls * out_tokens * (1 + retry_rate)cached = total_in * cache_hitfresh = total_in - cachedcached_price = plan.cached_per_mtok if cache_price is None else cache_pricereturn (fresh * plan.input_per_mtok + cached * cached_price+ total_out * plan.output_per_mtok) / 1_000_000 * (1 - batch_ratio)
A worked example: the unit price dropped, but total spending rose
All values below are hypothetical and used only to demonstrate the calculation method. Suppose a Q&A application makes 1 million calls per month, averaging 1,000 input tokens and 400 output tokens per call:
- Old version: input at 20 CNY per million tokens, output at 60 CNY per million tokens, 0.044 CNY per call, 44,000 CNY per month, of which output accounts for about 55%;
- New version: input drops to 14 CNY per million tokens (about 70%), output rises to 84 CNY per million tokens, 0.0476 CNY per call, about 47,600 CNY per month—about 8% higher than the old version;
- New version with caching and batching added: the fixed prefix accounts for about one-third of the input and hits the cache entirely (cache price at 20% of the input price), and half the requests go through the batch API (at 50%), bringing monthly spending down to about 35,800 CNY.
Clearly, a “price cut” conclusion drawn from the input unit price alone can be the exact opposite of the real bill. A metric closer to the business is cost per successful task: total cost divided by the number of completed tasks. If the new version raises the task success rate from 88% to 93%, then under a monthly spend of 47,600 CNY the cost per successful task is about 0.0511 CNY—already lower than the old version’s 0.05 ÷ 0.88 ≈ 0.0568 CNY. From a business standpoint, this upgrade is actually worthwhile.
The cost boundary of self-hosted inference
The cost of private deployment is made up of GPU depreciation, data-center and power, the labor for inference-framework tuning, and operations and on-call coverage, with clear economies of scale: when utilization is low, per-call cost is far higher than cloud services; when the load is stable and large enough, it may form an advantage. A reasonable path is to first use a hosted service to validate the business, and once call volume, load curves, and data sensitivity are all clear, migrate the stable, high-frequency portion to a private environment.
4. Engineering Deployment: Layered Routing and Fine-Grained Control
A unified model gateway: hide differences, support canary releases
Different models differ in API format, parameter naming, and error codes. It is recommended to add a unified gateway layer between the business and the models, so business code depends only on a unified interface and model selection is configuration-driven, enabling percentage-based canary rollout, fast rollback, and parallel comparison of two versions.
import randomclass ModelGateway:"""Unified model access layer: business code depends only on invoke(); models and routing policy are fully configurable."""def __init__(self, models, canary_ratio=0.0):# models: {"v1": client_a, "v2": client_b}; clients implement a unified chat() interfaceself.models = modelsself.canary_ratio = canary_ratiodef invoke(self, task, **kwargs):model = task.get("model")if model is None:model = "v2" if random.random() < self.canary_ratio else "v1"return self.models[model].chat(**kwargs)
Layered routing: let simple requests use lightweight models
Not every request needs the strongest model. Routing by input features can usually significantly reduce average cost with roughly unchanged quality:
def choose_model(req: dict) -> str:"""Route by difficulty and input size: 0 lightweight / 1 standard / 2 strong reasoning."""if req["has_tools"] or req["need_reasoning"] or len(req["messages"]) > 30:return "2"if req["input_tokens"] > 4000 or req["intent_confidence"] < 0.8:return "1"return "0"
Routing signals can come from a lightweight intent classifier placed up front, or from the business’s own features; deterministic tasks such as order-status lookups and fixed-template generation can be handed directly to a small model or even a rules engine.
Prompt and cache friendliness
To improve cache hit rate and output controllability, you can adopt these practices:
- Put invariant content such as the system instruction, tool descriptions, and a few examples at the very front of the prompt;
- Put dynamic content such as the user’s question and retrieval results at the tail of the prompt;
- Avoid inserting volatile content such as timestamps or random IDs into the fixed prefix, which would break the cache;
- Require structured output and validate the return value strictly; if validation fails, trigger a limited number of retries.
import jsondef parse_structured(text: str):"""Parse the JSON returned by the model; returns None on failure, and the caller decides whether to retry or fall back."""try:data = json.loads(text)except json.JSONDecodeError:return Nonerequired = {"answer", "confidence", "sources"}if not isinstance(data, dict) or not required.issubset(data):return Noneif not isinstance(data["sources"], list):return Nonereturn data
Stability governance: retries, fallback, and observability
Model services are external dependencies and must be designed as unstable dependencies:
- Idempotency and backoff: generate an idempotency key for each request; for transient errors such as rate limiting and timeouts, use exponential backoff with added random jitter; for deterministic errors such as parameter errors, fail directly to avoid ineffective retries that amplify cost;
- Timeout budget: set an end-to-end timeout for the whole chain and allocate an independent time budget to the model call, preventing a single point from dragging down the entire chain;
- Fallback strategy: when the primary model is unavailable, switch to a backup model or return a cached result to keep the business basically available;
- Observability: log input/output token counts, cache-hit status, latency distribution (P50/P95/P99), success rate, and task-completion rate by model and scenario, and aggregate cost by business dimension.
In a cloud environment, evaluation and offline tasks can be deployed on a container platform with on-demand scaling, call logs centralized to a logging service, key metrics connected to monitoring and alerting, and model weights and datasets stored in object storage. This way, without building your own data center, you get full elasticity and observability. For businesses with high data sensitivity, platforms such as Baidu AI Cloud also offer multiple deployment options and security-compliance capabilities worth evaluating; refer to each platform’s official documentation for specifics.
5. Decision Checklist and Common Pitfalls
Upgrade decision checklist
- Is the new version no worse than the current one on the core dimensions of your own evaluation set, with no new high-frequency error types?
- After estimating with the real input/output distribution, does the cost per successful task decrease?
- Do P95 and P99 latency meet the business constraints, and can rate-limit quotas support peak load?
- Are data flow, retention policy, and compliance requirements satisfied?
- Do you have canary, rollback, and dual-track parallel capabilities?
Only when all the above answers are “yes” should you proceed to a full rollout; any “no” answer should first be validated within a canary scope.
Common pitfalls
- Looking only at leaderboards: a high leaderboard score does not mean good business results;
- Looking only at unit price: ignoring output share, caching, and retries often makes the bill the opposite of intuition;
- Blind upgrading: a new version may enable a longer chain of thought by default, so output tokens can multiply and simple tasks become more expensive;
- One-shot full rollout: without canary and rollback, it is hard to stop losses quickly when problems arise;
- Ignoring evaluation-set maintenance: the business keeps changing, and the evaluation set must keep updating with online data, or it will gradually lose its reference value.
6. Conclusion
The rapid iteration of Chinese LLMs in parameter scale, open-source strategy, and inference cost gives developers ever-richer choices, and also makes “how to evaluate a new version of a model” a question every team faces. But the answer is not in leaderboards, nor in unit-price tables, but in your own business data and constraints. By centering on the scenario and combining capability evaluation, cost estimation, and engineering governance, you can keep making rational, verifiable, and rollback-able selection decisions throughout the fast-evolving LLM cycle.
