How to Build a Multi-Dimensional LLM Evaluation Pipeline (and Rank Models Reliably)
Ad-hoc "just try a few prompts" testing is not enough for model selection or quality regression: results are not reproducible, different capabilities get blended into one vague impression, and manual review does not scale. This article explains how to build an evaluation pipeline instead — standardized test cases, concurrent calls across multiple models, raw outputs persisted to disk, independent per-dimension scoring by a judge model, and a final aggregated ranking that pinpoints each model's weak dimensions. It covers orthogonal evaluation dimensions (compliance, conversion, decision, facts, format, persona), how to design constraints and traps that make cases discriminating, Python code for concurrent execution with resumable output files, how the judge model should be configured, the aggregation logic, and common pitfalls such as benchmark contamination, judge bias, and uncontrolled randomness.
1. Why “just try a few prompts” evaluation is unreliable
In real projects, many teams evaluate LLMs “by feel”: they send a handful of prompts, eyeball the outputs, and assign scores from intuition. That is acceptable during early exploration, but as soon as you need the results to support a model-selection decision or a quality regression check, three serious problems surface:
- Not reproducible: the same model produces widely varying outputs across runs and prompts. Without a fixed benchmark, there is no basis for side-by-side comparison.
- Confused dimensions: capabilities as different as compliance, factual correctness, format adherence, and persona consistency get blended into one vague impression, so a failure in a single dimension is masked by strong overall performance.
- Not scalable: manual case-by-case comparison cannot cover large test sets, and it is hard to track results continuously across many models and versions.
That is why the professional approach is to build an evaluation pipeline: standardized test-case inputs, concurrent generation across multiple models, complete persistence of raw outputs, independent per-dimension scoring by a judge model, and finally an aggregated ranking that localizes the weak dimensions. The rest of this article follows that pipeline.
2. Decide what you are evaluating: breaking down the dimensions
The starting point of an evaluation is not the model — it is dimension design. A practical multi-dimensional evaluation system usually consists of several orthogonal capability dimensions:
- Compliance: does the output cross a safety red line, or contain prohibited or inappropriate content?
- Conversion / structuring: can it reliably turn a natural-language input into the required structured result, such as JSON or a table?
- Decision: given multiple constraints, can it make a reasonable and self-consistent judgment and trade-off?
- Facts: are the statements consistent with the provided knowledge or with objective facts, or are there hallucinations?
- Format: does it strictly respect the output constraints, including fields, length, encoding, and punctuation?
- Persona: does it hold the specified role consistently, without drifting or breaking character?
These dimensions are orthogonal: a model may have excellent compliance but poor factual accuracy, or flawless format adherence while its persona collapses repeatedly. Only by scoring them separately does a ranking become diagnostic.
3. Constraints and traps: making test cases discriminate
If test cases are too easy, every model passes and the benchmark loses its discriminating power. A high-quality evaluation set needs deliberately designed constraints and traps:
- Stacked constraints: present several mutually constraining rules at once and check whether the model can satisfy all of them rather than trading one off against another.
- Misleading traps: embed plausible-looking but incorrect guidance in the input to see whether the model follows it blindly.
- Boundaries and conflicts: construct scenarios whose requirements contradict each other, and observe whether the model executes mechanically or recognizes the contradiction.
- Format pressure: require strict JSON output with no fields added or removed, which deliberately amplifies differences in format adherence.
Freeze the case set and put it under version control so that every evaluation run uses identical inputs — that is the precondition for reproducible results.
4. Pipeline architecture: from task definition to ranked output
The whole evaluation flow can be abstracted as a clear chain:
Task setup (constraints and traps) -> concurrent calls to multiple models -> raw outputs persisted to disk(one file per model, resumable) -> judge model scores each dimension independently -> aggregate ranking,localize failing dimensions
4.1 Concurrent calls and persisting raw outputs
When you target multiple models with a large number of cases, sequential calls are extremely inefficient. Concurrent calls are standard practice, and every raw output must be persisted in full — it is the only basis for later recomputation, review, and dispute investigation.
The engineering key to persistence is one file per model plus resumability: each model writes to its own file (for example JSONL), so after an interruption the program can skip the completed records and rerun only the unfinished cases. A sketch:
import jsonfrom pathlib import Pathfrom concurrent.futures import ThreadPoolExecutor, as_completeddef load_done(model_name, out_dir):path = Path(out_dir) / f"{model_name}.jsonl"done = set()if path.exists():with path.open(encoding="utf-8") as f:for line in f:try:done.add(json.loads(line)["case_id"])except json.JSONDecodeError:continue # skip corrupted lines to keep resumption robustreturn path, donedef run_case(client, model_name, case):resp = client.chat(case["prompt"], temperature=0) # fixed temperature for comparabilityreturn {"case_id": case["id"], "model": model_name, "raw": resp}def dispatch(client, model_name, cases, out_dir, workers=8):path, done = load_done(model_name, out_dir)todo = [c for c in cases if c["id"] not in done]with path.open("a", encoding="utf-8") as f:with ThreadPoolExecutor(max_workers=workers) as pool:futures = {pool.submit(run_case, client, model_name, c): c for c in todo}for fut in as_completed(futures):f.write(json.dumps(fut.result(), ensure_ascii=False) + "\n")f.flush() # flush promptly so no data is lost on interruption
A few points worth noting: fix the randomness parameters such as temperature, otherwise the same case will fluctuate between runs; choose the concurrency level carefully in light of downstream rate limits; persist with append writes and flush promptly so that resumption stays reliable. If the data volume is large, archive raw outputs in object storage and use a logging service to retain call audit information.
4.2 The judge model: independent per-dimension scoring
Manual scoring does not scale, so a judge model replaces human review. The judge receives the task constraints plus the raw output of the model under test, and scores it dimension by dimension. Design considerations:
- Score each dimension independently: give a separate score and rationale per dimension instead of one aggregate number, so when something fails you know which dimension failed.
- Provide scoring anchors: state explicit criteria, such as the judgment description for each of the 0 / 0.5 / 1 levels, to reduce subjective drift.
- Blind review: do not reveal which model produced the output, which avoids anchoring bias.
- Auditable: keep the judge’s rationale text so you can spot-check it with human reviewers and assess how trustworthy the judge itself is.
4.3 Score aggregation and ranking
Once you have per-dimension scores, you need to aggregate them into averages and produce a ranking. A sketch:
def aggregate(scores):# scores: {model: {case_id: {dim1: 0.9, dim2: 0.7, ...}}}dims = ["compliance", "conversion", "decision", "facts", "format", "persona"]ranking = []for model, cases in scores.items():per_dim = {d: [] for d in dims}for case_scores in cases.values():for d in dims:per_dim[d].append(case_scores.get(d, 0))dim_avg = {d: round(sum(v) / len(v), 2) for d, v in per_dim.items() if v}overall = round(sum(dim_avg.values()) / len(dim_avg), 2)ranking.append({"model": model, "overall": overall, "dims": dim_avg})ranking.sort(key=lambda x: x["overall"], reverse=True)return ranking
The value of a ranking is not “who is in first place” but the distribution of each model’s weak dimensions. A model with a leading average but a very low score in one dimension may be unusable in compliance-sensitive scenarios; in contrast, a model with a middling average but balanced dimensions is often the safer choice for production.
5. Common pitfalls and engineering best practices
- Contaminated benchmark: if cases have been “seen” by the model or leaked into public corpora, scores will be inflated. Keep the evaluation set private and rotate it regularly.
- Single-metric decision making: looking only at the average ranking while ignoring dimension-level weaknesses is the most common source of misjudgment.
- Judge bias: the judge model may have its own preferences, such as favoring longer answers, so calibrate it with anchor criteria and human spot checks.
- Uncontrolled randomness: not fixing parameters such as temperature makes results non-reproducible.
- No regression: after a model is updated, rerun the same case set and use historical scores as the regression baseline.
A mature evaluation pipeline should support one-click triggering, automatic persistence, automatic scoring, and automatic ranking, and it should feed key metrics into a monitoring and alerting system so that model quality becomes visible and continuously regression-tested. Whether you evaluate in-house models or general-purpose capabilities accessed externally, the methodology applies — the value of an evaluation system depends not on who is being evaluated but on whether the system is rigorous, reproducible, and diagnostic.
6. Conclusion
At its core, LLM evaluation turns “does this feel good” into an engineering problem that is reproducible, attributable, and regression-testable. Four things matter: first, decompose the evaluation dimensions so that they are orthogonal; second, design cases with real discriminating power through mind traps; third, use concurrent calls and resumable execution to gain scale and reliability; and finally, score each dimension with a judge model and aggregate a ranking to localize weaknesses. Master this pipeline, and you can make defensible decisions in both model selection and quality monitoring.
