Engineering Approaches for Evaluating Output Quality Across Multiple LLMs
This article presents a structured, engineering-focused methodology for evaluating output quality across multiple large language models. It covers a four-stage pipeline including task definition, concurrent model invocation, raw output persistence with resumable execution, and dimension-wise scoring by a judge model. The seven-dimension scoring framework enables teams to pinpoint specific weaknesses—such as format compliance or factual accuracy—rather than relying on composite averages that obscure failure patterns. Practical code examples, output archival strategies, and ranking and analysis techniques are included.
When the same task is assigned to different large language models, output quality can vary significantly. Relying solely on an overall average score often masks underlying issues. One model may demonstrate strong factual accuracy yet frequently violate formatting requirements, while another may produce well-structured outputs but often deviate from its assigned role. Both models may achieve similar average scores, but their failure modes are entirely different, and so are the appropriate directions for improvement.
An effective evaluation system must therefore satisfy three goals. First, raw outputs from each model must be traceable for later review. Second, scoring should be broken down by dimension so that specific weaknesses can be identified. Third, the entire pipeline should be repeatable, allowing results from different runs to be compared.
Overall Pipeline
The evaluation process typically consists of four stages:
Task definition (constraints and known pitfalls)-> Concurrent invocation of candidate models-> Persistence of raw outputs (per-model files, with resume support)-> Dimension-wise scoring by a judge model-> Aggregate ranking and weakness localization
The task definition stage should clearly specify three elements: input samples, explicit constraints, and known pitfalls. Pitfalls refer to deliberately designed traps that may induce errors in models, such as requiring the model to cite non-existent data or to ignore system rules. Test sets lacking such designs tend to measure only basic capabilities and offer poor discrimination between models.
Concurrent Invocation and Timeout Control
When the number of models under evaluation is large and single-response latency is high, serial invocation becomes inefficient. A common approach is to use asynchronous concurrency while capping the maximum concurrency level to avoid triggering server-side rate limits.
import asyncioMAX_CONCURRENCY = 5TIMEOUT_SECONDS = 60async def run_one(sem, client, model_name, prompt):async with sem:try:resp = await asyncio.wait_for(client.generate(model=model_name, prompt=prompt), # illustrative interfacetimeout=TIMEOUT_SECONDS,)return {"model": model_name, "ok": True, "output": resp}except asyncio.TimeoutError:return {"model": model_name, "ok": False, "error": "timeout"}except Exception as exc:return {"model": model_name, "ok": False, "error": str(exc)}async def run_all(client, models, prompt):sem = asyncio.Semaphore(MAX_CONCURRENCY)tasks = [run_one(sem, client, m, prompt) for m in models]return await asyncio.gather(*tasks)
This code is illustrative only; client represents any call wrapper that conforms to the expected contract. Two points are critical: the semaphore controls the concurrency level, and each call has an explicit timeout. Failed results must also be returned and recorded rather than discarded. Otherwise, the aggregation stage may incorrectly assume a model was skipped, leading to skewed statistics.
Persisting Raw Outputs with Resume Support
Interruptions during evaluation are common—caused by network fluctuations, exhausted quotas, or terminated processes. If results are kept only in memory, rerunning the entire process becomes costly. The recommended approach is as follows.
- Create a separate directory per model, with one file per sample:
outputs/<model_id>/<sample_id>.json. - Write to a temporary file first, then atomically rename it after completion, preventing partial files from being treated as valid results.
- On startup, scan existing files, skip completed samples, and run only the missing ones.
import jsonfrom pathlib import Pathdef save_output(root: Path, model: str, sample_id: str, record: dict) -> None:target = root / model / f"{sample_id}.json"target.parent.mkdir(parents=True, exist_ok=True)tmp = target.with_suffix(".tmp")tmp.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding="utf-8")tmp.replace(target) # rename is atomic within the same filesystemdef pending(root: Path, model: str, sample_ids: list[str]) -> list[str]:model_dir = root / modeldone = {p.stem for p in model_dir.glob("*.json")} if model_dir.exists() else set()return [s for s in sample_ids if s not in done]
This persistence strategy offers another advantage: the evaluation stage can be rerun independently without re-invoking the candidate models. This saves cost and ensures that when the same outputs are scored multiple times, the input is consistent and results remain comparable.
A Note on Output Archive Field Design
Each record should include the sample ID, model identifier, prompt version, invocation timestamp, latency, and the raw text. The prompt version is especially important. Once the prompt changes, historical results can no longer be directly compared with new ones; recording the version prevents such misinterpretations.
Seven-Dimension Scoring by a Judge Model
Asking a judge model to produce a single overall score makes results difficult to interpret and problems hard to pinpoint. A more reliable approach is to decompose scoring into multiple dimensions, each scored with a separate prompt and a unified scale from 0 to 1.
The following seven dimensions are recommended.
- Compliance: Whether constraints and safety rules are violated.
- Contextual Fit: Whether the tone and style match the intended use case.
- Framing: Whether inappropriate leading or suggestive phrasing appears.
- Decision Support: Whether judgments are substantiated and recommendations are actionable.
- Factual Accuracy: Whether key statements align with the provided material or established facts.
- Format Adherence: Whether specified structure, fields, and length are respected.
- Role Consistency: Whether the required persona, tone, and boundaries are maintained.
Each dimension should have explicitly defined scoring criteria. For example, 0 indicates clear violation, 0.5 indicates partial compliance, and 1 indicates full compliance. The more concrete the criteria, the smaller the disagreement between different judges.
DIMENSIONS = {"compliance": "Whether constraints and safety rules are violated. 0 = clear violation, 1 = full compliance.","format": "Whether specified structure, fields, and length are met. 0 = major deviation, 1 = full compliance.",# other dimensions defined in the same manner}def build_judge_prompt(dim_key: str, task: str, output: str) -> str:return (f"Task requirements:\n{task}\n\n"f"Output to evaluate:\n{output}\n\n"f"Evaluate only the following dimension: {DIMENSIONS[dim_key]}\n""Provide a brief rationale in 2-3 sentences, then return a JSON object "'in the form {"reason": "...", "score": a number between 0 and 1}.')
Scoring only one dimension at a time reduces interference between different criteria. Requiring the judge to provide reasoning before assigning a score also allows for manual spot-checking of the decision process.
Aggregation, Ranking, and Weakness Localization
For each model, the score for each dimension is averaged across all samples. The overall score can be computed as the arithmetic mean of the seven dimension averages. If certain dimensions carry more business weight, they can be assigned greater weight in the aggregate.
Beyond ranking, the more informative output is identifying each model’s weakest dimension. Two models with identical overall scores may have entirely different weak dimensions, pointing to different areas of improvement.
from statistics import meandef summarize(scores: dict[str, dict[str, list[float]]]) -> list[dict]:rows = []for model, dims in scores.items():dim_means = {d: mean(v) for d, v in dims.items()}overall = mean(dim_means.values())weakest = min(dim_means, key=dim_means.get)rows.append({"model": model,"overall": round(overall, 2),"weakest": weakest,"weakest_score": round(dim_means[weakest], 2),})return sorted(rows, key=lambda r: r["overall"], reverse=True)
This per-dimension aggregation makes it possible to compare models not just on a single composite score but on specific capabilities. It also supports targeted improvements: knowing that a model’s weakest dimension is format adherence leads to a different optimization strategy than discovering its weakest dimension is factual accuracy.
