Multi-Model, Multi-Dimensional LLM Evaluation: A Practical Framework for Model Selection
Public leaderboards answer "which model scores higher on average across general tasks," but engineering selection needs a different question: "which model is more reliable for my specific scenario?" This article explains why that gap exists and presents a self-built, reproducible evaluation pipeline for comparing LLMs across multiple vendors. It covers seven evaluation dimensions, a unified access layer for batch inference, scoring rubrics with clear anchors, automated LLM-as-judge combined with calibrated human review, and a method for interpreting results that avoids the "aggregate score trap." Practical Python code, dimension and result tables, and engineering practices such as baseline tracking, repeated sampling, gating thresholds, and regression alerts are included throughout. The goal is to move model selection from a one-off guess to a sustainable, repeatable engineering capability.
1. Why You Need a Self-Built Evaluation Process
Selection decisions from different teams often contradict each other. Some people look only at overall rankings, others draw conclusions from a few trial runs, and problems only surface after deployment. Public leaderboards have reference value, but they answer “which model has the higher average score on general tasks,” whereas engineering selection needs to answer “which model is more reliable in my scenario.” The gap between the two comes from three main sources.
First, data contamination and practice effects. Public test sets have circulated online for a long time and carry the risk of being absorbed into pretraining corpora, which can systematically inflate scores. This is not an accusation that any specific model cheats; it simply means that measuring real capability with public questions weakens the signal.
Second, uneven capability. Model capability looks more like a radar chart than a single ranking line. A model that excels at math may frequently “confidently make things up” on factual questions. An aggregate score averages away these weaknesses and hides the real risk.
Third, leaderboards are not isomorphic to your business. Leaderboard questions are mostly general knowledge or academic problems, while enterprise scenarios are often customer-service Q&A, ticket classification, document summarization, or code completion, with completely different distributions. A two- or three-point lead on a leaderboard may be imperceptible on a specific task.
So instead of asking “which domestic models have decent quality,” it is better to build your own evaluation process: connect to models from multiple vendors through one interface, score across multiple dimensions, weigh them according to your own business scenarios, and make every step reproducible and regression-testable. This article presents an evaluation design with seven dimensions, an automated batch evaluation script, and a selection method that guards against the “aggregate score trap.”
2. Before Evaluation: Use Engineering to Guarantee Comparability
The most overlooked part of model evaluation is not scoring itself, but whether the experimental conditions are consistent. If the following conditions are fixed before evaluation, the conclusions become comparable.
- Unified access protocol. Most mainstream model services in China offer invocation methods compatible with a common chat-style API specification, typically called with parameters such as a model name and a message list. This means one codebase can cover multiple vendors without writing a separate protocol adapter for each model. Route them all through a single gateway (a self-hosted proxy or a third-party aggregation service both work) and identify candidate models by alias inside the gateway. That way, business code depends on a single entry point, and switching candidate models requires no code changes.
- Fixed decoding parameters. Set temperature to 0 or close to 0 and fix the random seed so sampling is as deterministic as possible. If production uses a higher temperature, keep a corresponding set of high-temperature results in the evaluation so that “evaluation performance” and “production performance” do not diverge.
- Guaranteed complete output. Set the maximum output length to 2–3 times the expected answer length so long answers are not truncated and misjudged as “low quality.” If a model supports parameters like reasoning effort or thinking budget, keep every model at the same tier; otherwise you are comparing configurations rather than models.
- Unified prompts. All candidate models use the same system prompt and user prompts, and the prompts describe only the task requirements with no brand-biased wording.
- Masked model identity. During scoring, give raters only anonymous IDs (such as model-a, model-b), and reveal the mapping only after scoring is done, to rule out prior bias from brand impressions.
- Complete metadata logging. Every evaluation saves the model name, model version (if available), evaluation time, prompt version, decoding parameters, and rubric version, so any conclusion can be traced.
Among these, “fixed decoding parameters, unified prompts, and masked model identity” directly determine whether scoring is fair, while “complete metadata logging” determines whether conclusions can be reproduced and regression-tested. You should also fix the candidate list before evaluation: the pool usually contains 5–10 models of similar scale and positioning. Too many dilutes the cases per model, while too few makes a meaningful comparison difficult.
3. The Unified Access Layer: Write Once, Call in Batch
The access layer has a single goal: reduce “swap the model and evaluate” to “change one variable.” The following is an illustrative wrapper covering base URL and key loading, timeout retries, concurrency control, and result persistence.
import hashlibimport jsonimport osimport timefrom concurrent.futures import ThreadPoolExecutor, as_completedfrom dataclasses import dataclass# 1. Read secrets only from environment variables; never hardcode themGATEWAY_BASE_URL = os.environ["LLM_GATEWAY_BASE_URL"]GATEWAY_API_KEY = os.environ["LLM_GATEWAY_API_KEY"]JUDGE_API_KEY = os.environ["JUDGE_API_KEY"] # separate key for the judge model, to isolate usage# 2. Logical alias -> real model name inside the gateway; switching models requires no business-code changesMODEL_ALIASES = json.loads(os.environ["MODEL_ALIASES"])# e.g., {"model-a": "alias for candidate model A", "model-b": "...", "judge": "judge model alias"}@dataclassclass ChatClient:base_url: strapi_key: strtimeout: int = 120retries: int = 3def chat(self, model_alias: str, messages: list[dict], **params) -> str:payload = {"model": MODEL_ALIASES[model_alias],"messages": messages,"temperature": params.get("temperature", 0.0),"max_tokens": params.get("max_tokens", 2048),}for attempt in range(self.retries):try:# Illustrative code: connect to any HTTP client compatible with the chat-style APIresp = self.http_post(f"{self.base_url}/chat", payload, self.api_key, self.timeout)return resp.json()["choices"][0]["message"]["content"]except (TimeoutError, ConnectionError):if attempt == self.retries - 1:raisetime.sleep(2 ** attempt) # exponential backoff for rate limits and transient failuresraise RuntimeError("unreachable")runner = ChatClient(GATEWAY_BASE_URL, GATEWAY_API_KEY)judge = ChatClient(GATEWAY_BASE_URL, JUDGE_API_KEY)# 3. Concurrent evaluation: constrained by gateway rate limits; don't blindly increase concurrencydef evaluate_all(cases: list[dict], model_alias: str, workers: int = 8) -> dict:results = []with ThreadPoolExecutor(max_workers=workers) as pool:futures = {pool.submit(runner.chat,model_alias,[{"role": "user", "content": case["prompt"]}],): casefor case in cases}for fut in as_completed(futures):case = futures[fut]try:results.append({"id": case["id"], "answer": fut.result()})except Exception as exc: # record failed cases separately; they don't participate in scoringresults.append({"id": case["id"], "error": repr(exc)})return {"model": model_alias, "results": results}
There are three key design points in this wrapper.
Decoupling logical aliases from real model names. Business code only knows model-a, model-b, while the underlying layer connects to different vendors through an alias mapping table. Keep the mapping in environment variables or a configuration center, not hardcoded in the code.
Timeout retries and exponential backoff. When calling large models in batch, rate limiting and network jitter are the norm. If a retry still fails, record that case as an “evaluation failure” and count it separately rather than scoring it 0. Otherwise, a model that is merely “slow on the network” gets judged as “unable to answer.”
Concurrency and persistence. The concurrency level is determined by the gateway rate limit; start at 4–8 and increase gradually. Append each result to a JSONL file, one per line, which makes resume-from-checkpoint easy and per-case post-mortem convenient.
Turning Evaluation into a Pipeline
Chaining the above wrapper into fixed steps forms a repeatable evaluation pipeline:
- Load the dataset: read the evaluation set and validate field completeness for every case.
- Batch model inference: call
evaluate_alland write raw answers plus invocation metadata to JSONL. - Automated scoring: run the scoring stage to get a 0–1 score per dimension.
- Sampled human review: measure rater agreement and route disagreements to arbitration.
- Aggregate the profile: aggregate by dimension, combine with scenario weights, and produce a candidate list.
- Regression comparison: compare against the previous baseline and output a diff list.
If an evaluation task needs to run for a long time, hand it to a container platform as a batch job; evaluation data and results are well suited to archiving in object storage for version management and cross-team sharing; and gateway keys are best managed in a dedicated secrets-management service.
4. Seven Dimensions: From a Single Score to a Capability Profile
The dimension design follows three principles:
- Keep dimensions as orthogonal as possible to reduce double counting;
- Give every dimension actionable scoring anchors so raters do not score by gut feeling;
- Cover the four risk categories of “correctness, controllability, usability, and safety.”
Based on this, here are seven dimensions.
| Dimension | Question it addresses | Typical case form | Main scoring method |
|---|---|---|---|
| Instruction following | Whether it strictly follows format and constraints | Specified JSON structure, word limits, forbidden words | Rule validation (automated) |
| Factuality & anti-hallucination | Whether statements are grounded or fabricated | QA over given material, verifiable common-sense questions | Compare with reference answers (automated or judge) |
| Logical reasoning | Whether multi-step derivation is correct | Math word problems, conditional reasoning, logic puzzles | Final answer check + process review |
| Code ability | Whether code is correct and robust | Function implementation, defect localization, unit test writing | Unit tests + static analysis |
| Long context | Whether it locates and integrates distant information | Cross-section retrieval, multi-document comparison, long-text summarization | Key-point coverage + judge |
| Chinese semantics & culture | Whether it understands Chinese context and formal style | Idiom disambiguation, ambiguity resolution, official-document rewriting | Judge + human sampling |
| Safety & compliance | Whether risky requests are handled appropriately | Jailbreak inducement, prompt injection, privacy extraction | Rule hits + human review |
4.1 Instruction Following: The Baseline of Controllability
Business systems usually depend on model output being directly parseable text, and small flaws in instruction following trigger cascading failures downstream. For example, if you ask the model to return JSON with specified fields but it outputs an extra explanation, parsing fails.
Instruction following is best validated first with automated rules: whether JSON parses, whether fields are complete, whether word count is within range, and whether forbidden words appear. Clear and deterministic rules make it a good “gate” for the other dimensions.
4.2 Factuality and Anti-Hallucination: The Risk Most Easily Hidden by Aggregate Scores
Factuality contains at least two sub-questions:
- Are the facts correct? Compare item by item against the reference answer.
- Are knowledge boundaries clear? For questions the material does not cover, the model should explicitly say “the material contains no relevant information” rather than fabricating one.
This is the most dangerous of the seven dimensions. Some models perform excellently on common-sense Q&A, but for relatively obscure facts they may still produce fluent yet wrong answers. The evaluation must specifically design “out-of-material” questions to test whether the model knows what it does not know.
4.3 Logical Reasoning: Look at the Process, Not Just the Conclusion
The key to logic problems is distinguishing “guessed right” from “reasoned right.” Scoring should verify both the final answer and a sample of the reasoning process: are there skipped steps, circular arguments, or swapped conditions? For multi-step problems, have the model give the conclusion first and the justification second, and let raters judge whether the conclusion actually follows from the justification.
4.4 Code Ability: Let the Tests Do the Talking
The code dimension should “run,” not “look right.” Prepare a set of unit tests for each problem and use the pass rate as the score; then use static analysis to check for common defects such as unhandled exceptions or unvalidated inputs. Some models generate code with implicit dependencies or version incompatibilities, so the evaluation environment should standardize dependency versions.
4.5 Long Context: A Dual Test of Location and Integration
Long-context evaluation is often reduced to “needle in a haystack,” embedding a fact in a very long text to see if the model can find it. This only tests information retrieval. Real business needs cross-paragraph integration, such as summarizing risk clauses from a several-dozen-page report or comparing differing clauses between two contracts. The evaluation should therefore include both single-point retrieval and multi-point integration cases.
4.6 Chinese Semantics and Culture: The Key Difference in Local Scenarios
Chinese demands on idiom allusions, ambiguity resolution, official-document formatting, and classical-Chinese understanding often go beyond what general corpora cover. Design this dimension around three case types—rewriting, imitation writing, and disambiguation—for example, rewriting a colloquial passage into standard official-document style, then checking whether its format and wording follow industry conventions.
4.7 Safety and Compliance: Both “Refuse Correctly” and “Answer Correctly”
Safety evaluation has two common errors: failing to refuse what should be refused, where the model is induced into inappropriate output; and refusing what should not be refused, where legitimate business requests are wrongly blocked. Scoring must count these two error types separately. For prompt-injection cases, test whether the model can recognize that instructions come from an untrusted source rather than being “hijacked” by the data content.
5. Scoring Rules: Making 0–1 Scores Truly Comparable
Once you have dimensions, you need unified scoring rules. Without clear rules, scores from different raters are not comparable.
5.1 Write Anchors First, Then Score
Every dimension should provide tier descriptions, for example for factuality:
- 1.0: all key facts match the reference, no fabrication, clear boundaries;
- 0.75: mostly correct, with minor deviations that do not affect the conclusion;
- 0.5: partially correct, but a key conclusion is wrong or vague;
- 0.25: mostly wrong, with only a few correct fragments;
- 0.0: the conclusion contradicts the facts, or is completely off-topic.
The purpose of scoring anchors is to give subjective questions an objective reference. Only allow scores at anchor tiers or between adjacent tiers, to avoid “giving 0.8 by gut feeling.”
5.2 Separate “Automatically Determinable” from “Requires Subjective Judgment”
- Use rule-based judgment for dimensions like instruction following and code ability.
- Use a judge model on some questions for dimensions like factuality and logic.
- Use humans primarily for dimensions like Chinese semantics and safety.
The hybrid approach significantly reduces evaluation cost, but automated judging needs calibration, as covered in Section 6.
5.3 Control Sampling Variance
Even with temperature set to 0, some models still vary slightly between batches. It is recommended to repeat the same case 3 times and take the median, while recording the standard deviation. If a model’s standard deviation is clearly higher than others, it means its performance is unstable on similar questions, which is itself important selection information.
5.4 Provide Confidence Intervals
The size of the evaluation set directly determines the credibility of conclusions. For example, if one model scores 0.85 on factuality and another 0.82, but each has only 30 factuality questions, that 0.03 gap likely comes from sampling variance rather than a real capability difference. It is recommended to use no fewer than 30–50 cases per dimension, no fewer than 100 for critical dimensions, and to report both the mean and the confidence interval.
6. Automated Judging and Human Review: How to Keep Scoring Trustworthy
Using a model to evaluate models is a common way to improve efficiency, but it can introduce bias. If the judge model itself took part in the evaluation, it may unconsciously favor its own or same-family models’ answers. A more robust approach is as follows.
6.1 Judge Selection
- Choose a model that did not participate in this evaluation as the judge;
- The judge model’s context length and reasoning ability should be at least no weaker than the models under test;
- Ideally, the judge model should come from a different developer than the candidates.
6.2 Avoiding Common Scoring Biases
- Position bias. Score each pair of candidate answers twice, once in A, B order and once in B, A order, and take the average of the two runs.
- Length bias. In the scoring prompt, explicitly require “judge content correctness first, then evaluate completeness of expression,” so longer answers do not automatically win.
- Self-preference. Cross-score with different model families and regularly compare against human scores.
6.3 Calibrate with Agreement Metrics
Sample a set where humans and the judge model score the same answers independently, then compute rater agreement, commonly with a coefficient such as Cohen’s Kappa. As a rule of thumb, an agreement coefficient above 0.6 indicates moderate-or-better agreement; below 0.6, revise the scoring anchors or adjust the judge prompt. You can also have humans review 5%–10% of samples per batch to continuously monitor judge drift.
6.4 Illustrative Code
JUDGE_PROMPT = """You are a strict rater. Evaluate the [MODEL ANSWER] based only on the given reference answer and scoring anchors.[Question] {question}[Reference] {reference}[Model answer A] {answer_a}[Model answer B] {answer_b}[Anchors] 1.0=fully correct; 0.5=partially correct with a key omission; 0.0=wrong or off-topic.Note: judge content correctness first, then consider completeness of expression; a longer answer is not necessarily better.Output strictly in the following JSON format:{{"a": {{"score": 0.0, "reason": ""}}, "b": {{"score": 0.0, "reason": ""}}}}"""def dual_judge(case: dict, ans_a: str, ans_b: str) -> tuple[float, float]:"""Score in both A/B orders and average, canceling out position bias."""scores = {"a": [], "b": []}for a, b in [(ans_a, ans_b), (ans_b, ans_a)]:raw = judge.chat("judge",[{"role": "user", "content": JUDGE_PROMPT.format(question=case["prompt"], reference=case["reference"],answer_a=a, answer_b=b)}],)# Recommended: retry on JSON parse failure of the judge output, up to 2 timesparsed = json.loads(raw)scores["a"].append(parsed["a"]["score"])scores["b"].append(parsed["b"]["score"])return sum(scores["a"]) / 2, sum(scores["b"]) / 2
7. Interpreting Results: Beware the “Aggregate Score Trap”
Consider three illustrative candidate models with the following scores across the seven dimensions (weights are examples and must be adjusted per business):
| Dimension | Weight | Model A | Model B | Model C |
|---|---|---|---|---|
| Instruction following | 0.15 | 0.93 | 0.88 | 0.85 |
| Factuality & anti-hallucination | 0.20 | 0.88 | 0.62 | 0.90 |
| Logical reasoning | 0.15 | 0.82 | 0.85 | 0.78 |
| Code ability | 0.10 | 0.85 | 0.86 | 0.80 |
| Long context | 0.10 | 0.80 | 0.75 | 0.82 |
| Chinese semantics & culture | 0.15 | 0.87 | 0.86 | 0.88 |
| Safety & compliance | 0.15 | 0.90 | 0.82 | 0.85 |
| Weighted total | 1.00 | 0.865 | 0.785 | 0.851 |
By total score, Model A is first, Model C is close behind, and Model B clearly trails. But Model B’s factuality is only 0.62, clearly below the other two. If the scenario demands high factual accuracy, such as customer-service Q&A or knowledge-base retrieval, Model B’s risk is masked by the total. The reasonable conclusions should be:
- Model A is the first choice. Balanced across dimensions, with no obvious weakness.
- Model C is a backup or standby. Its gap to Model A is only 0.014; if their confidence intervals overlap, they are statistically indistinguishable, and you can decide based on cost, latency, or supply stability.
- Consider Model B only when factuality is not on the critical path. For example, formatted generation or text polishing.
From this, three selection principles emerge.
Principle 1: Set weights by scenario. Knowledge-base Q&A weights factuality, a code assistant weights code ability, and content creation weights Chinese semantics. Weights should be jointly set by the business team and the algorithm team and kept on record, to avoid “whoever decides sets the weights by guesswork.”
Principle 2: Set gating thresholds. Define a minimum passing score for critical dimensions such as factuality and safety, for example no less than 0.75. Any model below that line does not enter the candidate list regardless of how high its total is. A single veto controls risk better than a weighted average.
Principle 3: Focus on stability, not a one-time ranking. Across repeated evaluations, models with less score variance are more suitable for production. The value of evaluation is not to produce an absolute ranking, but to identify “which models are usable in which scenarios, and where their boundaries lie.”
8. Putting the Evaluation Script into Practice: From Experiment to Routine
A one-off script can only answer “who is better this time.” To make evaluation continuously valuable, you need to consolidate it into a repeatable pipeline.
import statisticsREPORT_TEMPLATE = """# Model Evaluation ReportEvaluation time: {timestamp}Model version: {model_meta}Prompt version: {prompt_version}Rubric version: {rubric_version}## 1. Dimension scores (median of 3 runs; stdev in parentheses)| Dimension | Score | Std. dev. | Gate | Pass? ||---|---|---|---|---|{instruction}{factuality}{reasoning}{code}{long_context}{chinese}{safety}## 2. Weighted total{weighted_score}## 3. Key issues (5 cases with the largest rater disagreement){disputes}## 4. Conclusion and recommendations{conclusion}"""def summarize(model_name: str, dim_scores: dict[str, list[float]],weights: dict[str, float], gates: dict[str, float]) -> dict:"""Aggregate repeated dimension scores; output the profile and gate verdict."""summary = {}for dim, scores in dim_scores.items():summary[dim] = {"median": statistics.median(scores),"stdev": statistics.stdev(scores) if len(scores) > 1 else 0.0,"gate_passed": statistics.median(scores) >= gates[dim],}summary["weighted"] = sum(summary[d]["median"] * w for d, w in weights.items())summary["qualified"] = all(summary[d]["gate_passed"] for d in gates)return summary
This pipeline has four engineering characteristics.
Baselines are traceable. Every evaluation records the model version, prompt version, and rubric version. Any score change can be traced to whether it came from a model upgrade, a prompt change, or a rubric adjustment.
Repeated sampling and statistics. Each case is repeated three times, described by median and standard deviation to capture stability and avoid misjudgment from a single run.
Gates up front. Pass the critical-dimension gates first, then compute the weighted total, so that a high score cannot mask a weakness.
Regression to prevent degradation. After a model or prompt update, automatically compare against the previous baseline. Alert when any dimension drops beyond a set threshold (such as 0.03), rather than discovering the problem only after it surfaces in production.
Five Common Pitfalls to Avoid
- Evaluation set leakage. Evaluation questions and reference answers must not enter training or fine-tuning corpora, nor appear in a retrievable knowledge base. Refresh the questions periodically to keep the evaluation set fresh.
- Looking only at average scores. Averages hide weaknesses, so always read them together with the dimension profile and gates.
- Ignoring cost and latency. When quality is similar, response time, concurrency, and per-call cost are often the deciding factors. It is recommended to separately record each model’s average latency, throughput, and per-call cost alongside quality evaluation.
- Disconnect between evaluation and production. The prompts and input distributions used in evaluation should match production, and the evaluation set should be refreshed periodically with real production requests (after de-identification).
- One-time conclusions. Model capabilities change dynamically, and a single evaluation only reflects the version at the time of evaluation. It is recommended to rerun the evaluation on a fixed schedule to form a trend curve.
9. Conclusion
For the question “how do you choose an LLM,” a more reliable approach is not to chase a single ranking, but to build an evaluation capability of your own:
- On the technical side, a unified access layer solves “how to compare,” seven dimensions solve “what to compare,” 0–1 scores with scoring anchors solve “how to score,” and an uncompeting judge model plus human review solve “whether the scores are trustworthy.”
- On the engineering side, baseline management, repeated sampling, gating rules, and regression alerts turn evaluation from a one-off experiment into a sustainable, routine capability.
As intelligent application development deepens, model selection is no longer a one-time technical decision but an engineering process that needs continuous calibration. Once the evaluation pipeline runs stably, whenever a new model is released or a version is updated, the team only needs to rerun the established process to quickly get answers to “can we use it, where should we use it, and how much risk is involved.” That is the core value of systematic evaluation over experience-based judgment.
