Evaluating and Selecting New LLMs: A Practical Framework for Developers

Every so often, a new large language model is officially released, and with it comes a flood of evaluation charts: the new model sets records across benchmarks, closes the gap with the strongest model to a razor-thin margin, or dramatically surpasses previous results on some capability. For ordinary users, these numbers are conversation fodder; for developers, they are merely the starting point of model selection. The real question is not “where does this model rank” but “how does it perform in my scenario, how high is the migration cost, and will it remain reliable over the long term.” This article attempts to provide a systematic evaluation framework to help teams turn “chasing hype” into “making decisions.”

1. Reading Benchmark Scores: How to Interpret Leaderboard Numbers

1. The meaning of a score depends on the composition of the evaluation set

Public benchmarks typically cover knowledge QA, mathematical reasoning, code generation, multilingual understanding, and similar directions, but the question distribution and scoring methods (exact match, unit test pass rate, human or model grading) vary widely across benchmarks. When you see a number like “65.7%,” the first question to ask is: what does it measure? Is it the pass rate on a single capability, or a weighted average across multiple subsets? Two models with similar overall scores may have completely different capability profiles—one scoring through reasoning, the other through memorization—and their performance on a specific business task can differ dramatically.

2. Beware of data contamination and overfitting

Most mainstream evaluation sets are public, and training corpora likely cover some of their questions, inflating scores—the model has “memorized the answers” rather than genuinely possessing reasoning ability. Common countermeasures in the industry include using private, non-public evaluation sets, periodically updating questions, and constructing perturbed versions (changing numbers, rephrasing questions, swapping scenarios). As an external observer, be cautious about scores that consistently “hit” specific benchmarks over time, and prioritize reports that clearly describe evaluation protocols and support reproducibility.

3. Score gaps must be judged by statistical significance

Suppose in one evaluation, Model A scores 65.7%, Model B scores 58.2%, and Model C scores 66.4%. The 0.7 percentage point gap between A and C, given a limited number of questions and a single evaluation run, likely falls within the confidence interval and is insufficient to assert superiority; the 7.5 percentage point gap between A and B is more likely to reflect a real capability difference. In addition, evaluation settings (number of few-shot examples, whether chain-of-thought is enabled, sampling temperature, number of evaluation rounds) all affect results. When reading evaluation reports, pay attention to these details rather than just the conclusion line.

2. A Checklist of Capability Dimensions More Important Than Benchmark Scores

Beyond benchmark scores, it is advisable to build a three-dimensional understanding of a model across the following dimensions, each corresponding to a real engineering risk:

1. Reasoning and task decomposition: When facing multi-step problems (complex calculations, process planning, multi-constraint conditions), can the model correctly decompose and solve them step by step, or does it “skip steps,” forget preconditions, or contradict itself midway?
2. Code capability: Not just “can it write it,” but also the runnability rate of generated code, cross-file understanding of existing codebases, defect repair success rate, and whether output conforms to the team’s engineering standards and style constraints.
3. Long context: The nominal context window length and “effective context” are two different things. You can verify with a “needle in a haystack” style test: insert key information at different positions in a very long document and check whether the model can reliably recall and correctly cite it, rather than showing obvious quality degradation in the middle-to-late sections.
4. Instruction following and format stability: Engineering integration relies heavily on structured output. Whether the model can stably output valid JSON and strictly follow field constraints across hundreds or thousands of calls directly determines the robustness of the parsing layer and retry costs.
5. Tool calling and agent capabilities: Whether function call parameters are accurate, planning and error-correction ability in multi-step tasks, and efficiency in using tool return results—these determine the ceiling of agent-based applications.
6. Chinese and domain corpora: Depth of understanding of Chinese semantics, idioms, and industry terminology, as well as factual accuracy on vertical domain knowledge QA, often only reveals differences on real corpora.
7. Safety and refusal boundaries: Whether handling of sensitive requests meets compliance requirements, and whether there is excessive refusal that blocks normal business requests.

3. From “Reading Leaderboards” to “Real Testing”: Building Your Own Evaluation Set

External leaderboards can only provide reference; what truly determines selection should be your own evaluation. A minimal viable process has three steps:

Step 1: Collect real business samples. Extract desensitized typical requests from online logs, covering high-frequency scenarios, edge cases, and historical defect cases. A starting size of one or two hundred samples is sufficient. Sample quality matters far more than quantity.

Step 2: Define evaluation dimensions and scoring criteria. For example, how much weight each of “factual accuracy,” “completeness,” and “format compliance” carries, and what judgment standard corresponds to each score level—write these down in advance to avoid criteria drift during review.

Step 3: Build an automated pipeline. Use rule-based validation (format, keywords, unit test pass rate) for quantifiable items, use LLM-as-a-Judge for open-ended QA, and supplement with a small proportion of manual spot checks to calibrate the scorer. Illustrative code:

# Minimal viable model evaluation pipeline (pseudocode)
import json
test_set = load_jsonl("business_eval_set.jsonl")   # desensitized business samples
def run_case(client, case):
    prompt = build_prompt(case["instruction"], case["input"])
    return client.generate(prompt, temperature=0.2, max_tokens=1024)
def judge(resp, case):
    rule_score = rule_check(resp, case)            # hard checks: format/keywords/unit tests
    llm_score  = llm_judge(resp, case,
                    dims=["accuracy", "completeness", "format"])  # model judge scores by dimension
    return {"rule": rule_score, "llm": llm_score}
report = {}
for name, client in candidate_clients.items():     # side-by-side comparison of candidate models
    scores = [judge(run_case(client, c), c) for c in test_set]
    report[name] = aggregate(scores)               # mean, pass rate, percentiles
print(json.dumps(report, ensure_ascii=False, indent=2))

A few notes: evaluation sets should be version-controlled and updated regularly like code, to prevent questions from leaking into prompts or fine-tuning data and causing “score gaming”; each evaluation should record the model version, parameter configuration, and timestamp to ensure reproducibility. If the team lacks the capacity to build evaluation infrastructure from scratch, you can also leverage model invocation, evaluation, and monitoring capabilities provided by LLM platforms such as Baidu AI Cloud, concentrating effort on constructing business samples.

4. Engineering and Cost Perspective: Factors Beyond Capability That Cannot Be Ignored

1. Latency and throughput: First-token latency affects interactive experience, while generation speed and concurrency limits determine whether peak traffic can be handled. Two models with similar capability but a 2x difference in latency may deliver completely different product experiences.
2. Cost structure: Beyond token unit price, also consider how long contexts are billed, whether context caching is supported (repeated prefixes in multi-turn conversations or long documents can significantly reduce costs), whether batch processing discounts exist, and estimate the real bill based on your own invocation patterns.
3. Stability and rate limiting: Service availability commitments, rate limiting and retry policies, and model version update notification mechanisms all affect the reliability of online systems and should be included in the evaluation.
4. Deployment form and data compliance: Public cloud API, private deployment, and hybrid deployment each have trade-offs and should be decided based on data sensitivity level, operational capability, and cost budget.

5. Canary Rollout and Smooth Migration

Even if evaluation results are ideal, it is not advisable to switch all traffic at once. A safer approach:

1. Canary rollout: First route 5%–10% of online traffic to the new model, compare key business metrics such as task completion rate, manual intervention rate, and user feedback, and gradually expand the proportion only after confirming there is no regression.
2. Dual-model fallback: Automatically degrade to a backup model when the primary model fails or times out, ensuring service continuity. Illustrative configuration:

# Illustrative: canary and fallback routing configuration
routing:
  primary:  { model: model-a, traffic: 0.1 }   # route 10% of traffic to the new model first
  fallback: { model: model-b }                  # automatic fallback on error or timeout
  timeout_ms: 30000
  retry: 1

1. Prompt regression testing: Different models have different sensitivities to the same prompt. Before switching, run the existing prompts fully through regression on the candidate model, and make targeted adjustments if necessary, rather than assuming “prompts can be reused as-is.”
2. Version locking and change tracking: Lock the validated model version, subscribe to official change notifications, and verify major version updates in a staging environment before they reach production, avoiding “upstream silently upgrades, downstream fails in production.”

Conclusion

Every new model release means the industry’s overall water level is rising, which is a good thing; but for a specific team, the “optimal model” is always a relative concept defined by business scenario, cost budget, and engineering constraints. Rather than chasing every leaderboard, it is better to build your own evaluation set, evaluation pipeline, and canary mechanism as early as possible—this infrastructure will not become obsolete, and it will let you capture returns faster and more reliably than others in every model iteration.