LLM Model Selection and Evaluation After Version Updates
New model versions can change quality, cost, and latency, so teams should re-evaluate candidates instead of assuming that newer or larger models are always better. This article describes a structured process for task-specific evaluation, reproducible test sets, unit effective cost comparison, tiered routing, price-change response, and canary validation with continuous monitoring.
1. Why Re-evaluate After a New Version Is Released
Model version updates often bring changes in capabilities, pricing, and service policies. A model that improves on one type of task is not necessarily better on every task. A lightweight tier may also outperform a higher-tier version in specific scenarios.
Therefore, the selection question should not be only “Is the new version stronger than the old one?” It should be “Does the new version meet my business requirements for quality, cost, and latency on my tasks?” That question requires a structured evaluation.
2. Common Pitfall: Subjective Impressions Instead of Data
Much of the discussion about models in technical communities is based on a small number of experiences. There are three common pitfalls.
The first is sample bias. The quality of answers to one or two questions cannot represent model performance across the full business distribution. The second is inconsistent comparison conditions. For example, one side uses default parameters while the other uses carefully tuned prompts, so the results are not comparable. The third is treating a single dimension as the overall conclusion. Looking only at price, only at leaderboard scores, or only at one generation result can easily lead to a one-sided judgment.
The way to avoid these pitfalls is to standardize evaluation conditions and record quality, cost, and latency metrics together.
3. Define Tasks and Constraints Before Evaluation
Before evaluation begins, break business requirements into measurable items. The following three steps are recommended.
- Task classification. Divide business requests by type, such as information extraction, text classification, code generation, multi-turn Q&A, and long-document summarization. Different task types place very different demands on model capabilities and should be evaluated separately.
- Quality criteria. Define judgment rules for each task type. Structured output can be judged through format validation and field matching. Open-ended generation can be judged through human scoring or model-assisted scoring, but the scoring rubric must be explicit.
- Hard constraints. Clarify the maximum context length, whether image or other modality inputs are required, the response time limit, and the per-call budget limit. A model that does not meet hard constraints should not enter the candidate list, regardless of its quality.
Constraints should be documented before evaluation. Adjusting standards afterward can easily bias conclusions toward existing preferences.
4. Build a Reproducible Evaluation Set
The evaluation set is the foundation of the entire process. A usable evaluation set usually has the following characteristics.
- Covers the real distribution. Samples should come from production logs or typical requests provided by business users, with stratified sampling by task type and difficulty.
- Includes boundary cases. Add cases such as non-standard input formats, missing information, and contradictory instructions to test model robustness.
- Has traceable annotations. Each sample should record the reference answer or judgment rule, along with the annotator and annotation time.
- Avoids data leakage. The evaluation set should not be included in prompt examples, and it should not overlap with model training or fine-tuning data. If the source cannot be confirmed, state this risk in the report.
Below is illustrative code for a general evaluation framework. call_fn and judge_fn need to be implemented by the user according to their own service interface and judgment rules.
import mathimport statisticsimport timedef evaluate(model_name, cases, call_fn, judge_fn):"""Run the evaluation set for a single model and return quality and latency metrics."""records = []for case in cases:start = time.perf_counter()output = call_fn(model_name, case["prompt"])latency = time.perf_counter() - startpassed = judge_fn(output, case["expected"])records.append({"passed": passed, "latency": latency})latencies = sorted(r["latency"] for r in records)p95_index = max(0, math.ceil(0.95 * len(latencies)) - 1)return {"pass_rate": sum(r["passed"] for r in records) / len(records),"median_latency": statistics.median(latencies),"p95_latency": latencies[p95_index],}
The same evaluation set must be used for all candidate models with the same prompt template, decoding parameters, and judgment function. Evaluation results should be archived together with the configuration so they can be rerun later.
5. Measure Price with Unit Effective Cost
Looking only at the listed price per million tokens can be misleading. A model with a lower unit price may end up costing more if its task success rate is low and multiple retries are required. A more reasonable metric is unit effective cost: the average cost of completing one qualified task.
The calculation method is as follows: first calculate the token cost of a single call, then divide it by the task success rate.
def unit_effective_cost(input_tokens, output_tokens,price_in_per_m, price_out_per_m, success_rate):"""Calculate the average cost of completing one qualified task. Prices are per million tokens."""if success_rate <= 0:raise ValueError("success_rate must be greater than 0")raw_cost = (input_tokens * price_in_per_m+ output_tokens * price_out_per_m) / 1_000_000return raw_cost / success_rate# The following parameters are only for demonstrating the calculation method and are not real prices for any product.cheap_model = unit_effective_cost(1200, 400, 1.0, 4.0, success_rate=0.78)strong_model = unit_effective_cost(1200, 600, 3.0, 12.0, success_rate=0.96)print(f"Model A unit effective cost: {cheap_model:.6f}")print(f"Model B unit effective cost: {strong_model:.6f}")
Note that the success rate should come from the evaluation set results above, not from an estimate. If the business also includes human review costs or losses caused by errors, those should also be factored in.
6. Tiered Selection: Match Model Tier to Task Requirements
Many service providers offer multiple tiers for the same model generation. Common dimensions include reasoning depth, response speed, and unit price. In general, lightweight tiers are suitable for high-throughput, low-latency tasks with relatively fixed structures, such as classification, extraction, and short rewrites. Flagship tiers are suitable for multi-step reasoning, complex code modifications, and long-context understanding.
A tiered selection approach is to use a lightweight tier for most requests first, then route low-confidence or failed requests to a stronger tier. This cascading approach often achieves a good balance between quality and cost. Routing thresholds should be determined based on the evaluation set rather than set by experience alone.
7. Responding to Price Changes
After a price change, teams should recalculate unit effective cost instead of directly reusing old selection conclusions. The specific steps are as follows.
- Use the new prices and existing success-rate data to recalculate the unit effective cost of each candidate model.
- Check whether the ranking changes after the price adjustment. If the ranking changes, additional evaluation is needed before launch.
- Set up a unified model-call abstraction at the business layer. Callers depend only on internal interfaces, and switching models requires only a configuration change. This reduces migration costs caused by a single provider’s price change.
- Set budget alerts for daily or monthly call costs, and record the cost share of each task type so that sources of cost growth can be identified.
8. Canary Rollout and Continuous Monitoring
Evaluation conclusions need to be validated online. The following process is recommended.
- Shadow traffic. Send a portion of real requests to both the old and new models, return only the old model’s results, and record the new model’s outputs for offline comparison.
- Small-percentage canary rollout. After confirming that shadow traffic results are stable, switch a small amount of real traffic to the new model and keep a fast rollback switch.
- Metric monitoring. Continuously track task success rate, human adoption rate, P95 latency, cost per request, and error-type distribution. If metrics become abnormal, pause the rollout and investigate the cause.
Monitoring data should be compared with evaluation set results regularly. If online metrics differ significantly from offline evaluation, it usually means the evaluation set no longer represents the real request distribution and needs updated samples.
9. Selection Checklist
- Are models evaluated separately by task type instead of using only one set of general questions?
- Does the evaluation set come from real requests and include boundary cases?
- Are quality, cost, and latency metrics recorded together?
- Does unit cost account for success rate and retry overhead?
- Has the ranking been recalculated after price changes?
- Has shadow traffic or a small-percentage canary rollout been completed before launch, with a rollback plan in place?
If your team uses model services from Baidu AI Cloud, you can include their APIs in the same evaluation framework and compare them with other candidates under the same conditions. The final selection should be based on your own business data rather than external reputation or a single leaderboard.
