LLM Evaluation: A Four-Layer Engineering Solution for Output Contracts and Quality Gates
Evaluating large language models is harder than scoring raw writing quality. Drawing on a crisis-PR copywriting example where the top-scoring model reached 0.97 while another stalled at 0.04, this article shows how evaluations actually fail and how to prevent it. It covers a single-turn, online evaluation setup with reproducible collection code, a six-dimension weighted scoring model with a veto rule, and a root-cause analysis of four failure modes — from reasoning consuming the output budget to the absence of deterministic delivery checks. It then lays out a four-layer engineering solution: structured output-contract prompts, a calculated token budget backed by rule-based validation, targeted per-round repair instead of full rewrites, and pipeline gating with rollback and observability. The aim is a repeatable, auditable evaluation process that turns model failures into a manageable engineering risk rather than a one-off leaderboard snapshot.
Introduction: A 0.04 Score Exposes a Common Evaluation Blind Spot
A note on the data first. The scores cited here come from an internal, desensitized, same-task head-to-head example: a self-built crisis-PR copywriting case in which the participating models are anonymized as Model A through Model I and the judge is a model that did not participate. This is not public leaderboard data and is not a review of any vendor’s product. Its only purpose is to put the failure mechanism on the table so you can reproduce it and learn from it.
The example task: a home-furnishing brand starts a recall over a batch of defective products and must produce, within minutes, a ready-to-publish set of social-media responses consisting of four deliverables:
- A short post of no more than 140 characters for the initial announcement;
- A supporting post of 300–450 characters covering the recall cause, affected scope, remedy, and apology;
- An employee reshare line of no more than 40 characters;
- A pinned reply of no more than 80 characters that keeps the messaging consistent and directs users to support.
Under the same prompt, the top-scoring case averaged about 0.97 — all four deliverables present, character counts on target, ready to ship. Model F, meanwhile, scored just 0.04 on the platform-adaptation dimension. The judge’s comment: it did not deliver the four required pieces, instead producing a long block of reasoning that was truncated at the end.
This is not a contest of writing skill. The real failure is that the model did not treat deliverables and writing ability as two separate things. Crisis-PR copy is only the shell; what is actually being tested is whether the model can, under tight constraints, multiple deliverables, and no opportunity to ask follow-up questions, reliably produce output that is structurally complete, machine-verifiable, and ready to publish. Tasks like this expose several problems common in practice: reasoning competing with output for the token budget, instruction following degrading as constraints pile up, and generated content lacking any machine-verifiable delivery standard. This article proceeds in the order: experiment design → root causes → a four-layer solution → methodology.
How to Design the Experiment
Why single-turn, online, and no follow-ups
The harshest constraint in crisis PR is time. Public sentiment builds within minutes, and operations cannot ping-pong confirmations with the model over a single piece of copy. So the evaluation deliberately uses single-turn online calls with no chance to follow up or clarify, mirroring the production reality of “one request, use the result directly.” This setup is more demanding than offline multi-turn evaluation and surfaces problems earlier.
Evaluation framework and reproducible collection code
The overall framework is “unified collection, multi-dimensional scoring, archived for inspection.” The script below is a directly runnable minimal implementation: all credentials are injected through environment variables, cases are version-controlled as files, and sampling parameters are fixed for the entire run.
import hashlibimport jsonimport osimport timefrom concurrent.futures import ThreadPoolExecutorfrom openai import OpenAI# 1) Environment config: the gateway URL and key are injected entirely via# environment variables, so no credentials appear in the code# export GW_BASE_URL="https://<your-gateway>/v1"# export GW_API_KEY="<your-key>"# export MODELS="model-a,model-b,model-c"GW_BASE_URL = os.environ["GW_BASE_URL"]GW_API_KEY = os.environ["GW_API_KEY"]MODELS = [m.strip() for m in os.environ["MODELS"].split(",") if m.strip()]# Unified gateway: abstracts away per-vendor API differences and keeps the# collection consistentclient = OpenAI(base_url=GW_BASE_URL, api_key=GW_API_KEY)# 2) Case set: the full prompt is versioned on disk and the script only# references it, which makes reproduction and diffing easywith open("cases/crisis_recall_v1.txt", encoding="utf-8") as fp:CRISIS_PROMPT = fp.read()CASES = [{"id": "crisis_recall_v1", "prompt": CRISIS_PROMPT}]# 3) Sampling parameters are fixed for the whole run, so results stay# comparable across models and across runsSAMPLING = {"temperature": 0.7, "top_p": 0.95, "max_tokens": 4096}os.makedirs("runs", exist_ok=True)def run_case(model_id: str, case: dict) -> dict:"""Run a single case and archive the raw result in full."""started = time.time()resp = client.chat.completions.create(model=model_id,messages=[{"role": "user", "content": case["prompt"]}],**SAMPLING,)choice = resp.choices[0]record = {"case_id": case["id"],"model": model_id,"prompt_hash": hashlib.sha256(case["prompt"].encode()).hexdigest()[:12],"sampling": SAMPLING,"content": choice.message.content or "","finish_reason": choice.finish_reason, # "length" means it was truncated"latency_ms": round((time.time() - started) * 1000),"usage": resp.usage.model_dump() if resp.usage else {},"ts": time.strftime("%Y-%m-%dT%H:%M:%S"),}with open(f"runs/{case['id']}__{model_id}.json", "w", encoding="utf-8") as fp:json.dump(record, fp, ensure_ascii=False, indent=2)return recordif __name__ == "__main__":with ThreadPoolExecutor(max_workers=8) as pool:futures = [pool.submit(run_case, m, c) for m in MODELS for c in CASES]records = [f.result() for f in futures]print(f"Collected {len(records)} responses; written to runs/")
Three easily overlooked details:
- Sampling parameters must be fixed. Otherwise results are not comparable across models, nor across runs of the same model.
- Raw responses must be kept verbatim. Only by seeing what the model actually produced can you tell whether the problem lies in capability or in output form.
- finish_reason must be recorded. A truncated ending is the most typical delivery failure, yet it is the easiest to hide behind an average score.
Scoring: six capability dimensions plus a weighted composite
An early version listed “composite score” alongside the six dimensions as a “seventh dimension” and then averaged all seven — a self-contradictory approach. The correct approach is: the composite score is not a dimension but a weighted sum of the six dimension scores.
| Dimension | Type | Weight | Core question |
|---|---|---|---|
| Instruction following | Strong constraint | 0.25 | Are all four deliverables present, and is only the required content output? |
| Platform adaptation | Strong constraint | 0.25 | Do character counts hit, does the format fit, is the ending complete? |
| Factual consistency | Constraint | 0.20 | Are the recall scope, batch, and remedy consistent with the source facts? |
| Safety & compliance | Constraint | 0.15 | Any absolute promises, blame-shifting, or privacy leaks? |
| Emotional resonance | Subjective | 0.08 | Is the apology sincere, without defensive phrasing? |
| Content quality | Subjective | 0.07 | Is the cause explained clearly and is the remediation specific and actionable? |
The scoring rule is fixed in three steps:
- Each dimension is scored from 0 to 1 by the judge (the rubric is versioned and archived alongside the case).
- Composite score = Σ(dimension score × weight), with weights summing to 1.0.
- A veto rule is added: if either instruction following or platform adaptation falls below 0.6, the sample is flagged as “unusable” — it does not enter the average ranking and is used only for root-cause classification.
This step is the watershed of the whole evaluation design. Mix hard metrics and subjective metrics into one average, and a 0.04 gets pulled back up by a 0.8 content-quality score into “looks acceptable” — and the delivery risk is quietly erased.
Scope of applicability and a reproducibility checklist
Every score corresponds only to the online version on the day the data was collected. Once a vendor silently updates its weights, the same prompt and sampling parameters can yield different outputs, causing score drift. The numbers here therefore do not promise any model’s long-term capability, and a one-time evaluation result should not be treated as a final selection decision.
For traceability, each evaluation run should archive these six attachments:
- The full prompt (version number plus checksum hash);
- The full judge rubric with scoring examples;
- Sampling parameters and inference-control parameters;
- Model identifier, version snapshot, and gateway address (desensitized);
- Collection timestamp, latency, and token usage;
- Raw response samples (including truncated ones).
Root Causes: Four Failure Modes Behind Incomplete Deliverables
Root cause 1: Reasoning consumes the output budget and the body gets cut off
The failing sample’s output began with a block of natural-language analysis: “Let me carefully analyze this task. I need to create a social media response combo… Let me go through the requirements…” — and hit the ceiling before it finished. This is a structural risk of reasoning models on “long output + multiple deliverables” tasks:
- The generation budget is a single fixed pool. Whether the ceiling is set to 1024 or 4096, whatever reasoning consumes first leaves that much less for the body.
- In some reasoning models, the thinking process and the final answer share the same output budget.
- If the thinking stalls at “restating the requirements one by one” instead of converging, it easily exhausts the budget before the body is written.
- The access layer often returns an “incomplete” body as if it were valid content, so the half-finished output slips silently into the business side.
Three practical configuration measures address the reasoning-chain problem: enable a separate reasoning channel (some APIs place the thinking in a reasoning_content / thinking field, distinct from content, so the access layer can read only content); use a reasoning-effort parameter (such as reasoning_effort / a thinking budget) and pick a non-deep-thinking mode directly for copywriting tasks; and have the access layer retry and degrade on parse failures — never return raw text as the result.
import reTHINK_PATTERNS = (r"^\s*<think>", r"^\s*let me ", r"^\s*i need to ")def audit_outputs(records: list[dict]) -> dict:truncated, leaked = set(), set()for r in records:key = f'{r["model"]}/{r["case_id"]}'if r["finish_reason"] == "length":truncated.add(key)head = r["content"].lstrip().lower()if any(re.search(p, head) for p in THINK_PATTERNS):leaked.add(key)bad = truncated | leakedreturn {"total": len(records),"truncated": len(truncated),"thinking_leaked": len(leaked),"deliverable_rate": 1 - len(bad) / max(len(records), 1),"bad_cases": sorted(bad),}
Root cause 2: With no separate channel, the reasoning chain leaks into the deliverable
In a crisis-PR setting, this failure is more dangerous than an ordinary mistake. The moment someone copies content containing internal deliberation into an official account, the public sees not a statement but the model weighing “whether to admit these products have a problem” — and that itself becomes a new source of crisis.
Root cause 3: Stacked constraints degrade instruction following
The task imposes four kinds of constraints in a single prompt: output count (4 pieces), length (different upper and lower bounds for each), content elements (cause / scope / remedy / apology), and tone (empathetic, not evasive). The more constraints, the easier it is to lose track of one while handling another. Common symptoms: writing only the main post and omitting the reshare line and pinned reply; a supporting post that falls below the minimum or exceeds the maximum; conversational wrapper text left in the body such as “Here is…” or “Hope this helps.” The mitigation is to number the constraints into a checklist and require the model to self-check each item before it outputs, stated at the end of the prompt.
Root cause 4: No deterministic delivery checks, so errors slip out silently
Beneath the first three problems lies a more fundamental flaw: whether a deliverable is acceptable relies entirely on the model’s “self-discipline,” with no machine check on the system side. Whether a character count is exceeded, whether the piece count is complete, whether the ending is intact, whether a reasoning chain leaked in — all judged by eye. No checks mean no interception, and without interception errors flow all the way down the evaluation chain into production.
What the comparison reveals for model selection
- Strong reasoning does not equal strong delivery: the former answers “did it think correctly,” the latter answers “did it deliver everything” — they must be evaluated separately.
- The output budget is a scarce resource: the more complex the task, the easier it is to exhaust the budget before the finish line.
- Short-form platform adaptation is an independent capability — character-count awareness, format constraints, and ending completeness — not the same thing as general writing ability.
- Constraints that can be checked by rules should be pushed forward into the prompt contract first, leaving semantic quality to the judge model.
Engineering Countermeasures: Deliver Completely, Correctly, and Reliably
Layer 1: Structured prompts built around an output contract
Rewrite “write four pieces of text” into “return four structured pieces of text per this contract,” specifying fields, character limits, and required elements.
[Role] You are a content assistant on the PR team of a home-furnishing brand.[Facts] Recall target: a specific product model, production batch XXXX; cause: someunits have an XX safety risk; affected scope: customers who bought this batch;remedy: unconditional return and refund, with round-trip shipping covered;support channels: the official customer-service hotline and online chat.[Task] Using the facts above, produce one crisis-response copy set, following thiscontract exactly:1. The entire output must be a single JSON object, with no analysis, explanation, orextra text;2. short_post: no more than 140 characters; include an apology, the recall facts, andthe return channel; no absolute wording;3. long_post: 300-450 characters; cover the cause, affected scope, remediation,apology, and follow-up improvements, in that order;4. staff_reply: no more than 40 characters; first person; sincere tone, no defensiveness;5. pinned_reply: no more than 80 characters; consistent messaging; guide users to checktheir batch and contact support;6. Do not fabricate data absent from the facts above; do not promise compensation amountsor timelines.
A structured contract offers three advantages: it is machine-verifiable (field names are fixed, so validation logic is reusable); constraints are enumerable (count, length, and required elements can each be checked line by line); and repairs can be targeted (whichever item fails, the feedback is routed precisely to that item).
Layer 2: Compute the output budget and back it with deterministic rules
The old rule of thumb — “usually reserve 2,000–4,000 tokens” — is an empirical figure that cannot be verified. Replace it with a calculable one:
body budget (tokens) ≈ Σ(each deliverable's max character count) × Chinese conversion ratio × safety factormax_tokens ≥ body budget + reasoning budget (if thinking shares the output channel)
Substituting this case: the total upper-bound character count is 140 + 450 + 40 + 80 = 710 characters; Chinese text with punctuation and JSON structure converts at about 1.5–2 tokens per character, giving roughly 1,065–1,420 tokens; multiply by a 1.5–2× safety factor for a body budget of 1,600–2,800 tokens; if the model’s thinking shares the output channel, add a 1,024–2,048 token reasoning budget, so max_tokens should be 3,072–4,096.
Scope of applicability: the ratio varies with language (English is more token-efficient), the nesting depth of field structures, and whether Markdown is required. English tasks can be estimated at 1.0–1.3 tokens per character. Before launch, run the same batch of cases once to measure the actual hit rate, then lock in the ratio.
import jsonimport reLIMITS = {"short_post": (1, 140),"long_post": (300, 450),"staff_reply": (1, 40),"pinned_reply": (1, 80),}REQUIRED_TOPICS = ["reason", "batch", "return", "shipping"]def budget_tokens(limits=LIMITS, cn_ratio=(1.5, 2.0), safety=(1.5, 2.0),reasoning=0) -> tuple[int, int]:chars = sum(hi for _, hi in limits.values())return (int(chars * cn_ratio[0] * safety[0]) + reasoning,int(chars * cn_ratio[1] * safety[1]) + reasoning)def validate(record: dict) -> list[str]:issues = []if record["finish_reason"] == "length":issues.append("Output was truncated (finish_reason=length); raise the output limit or compress reasoning")raw = record["content"].strip()if not (raw.startswith("{") and raw.endswith("}")):return issues + ["Output is not a complete JSON object; it may contain reasoning or explanatory text"]try:data = json.loads(raw)except json.JSONDecodeError:return issues + ["JSON parsing failed"]missing = set(LIMITS) - set(data)if missing:issues.append(f"Missing fields: {sorted(missing)}")for key, (lo, hi) in LIMITS.items():text = data.get(key)if not isinstance(text, str):continuen = len(re.sub(r"\s", "", text))if not lo <= n <= hi:issues.append(f"{key} has {n} characters, outside [{lo}, {hi}]")full = "".join(str(data.get(k, "")) for k in LIMITS)absent = [t for t in REQUIRED_TOPICS if t not in full]if absent:issues.append(f"Missing required information: {absent}")return issuesprint(budget_tokens()) # This case: body budget of roughly 1600-2800 tokens
Rule-based checks answer only “pass or fail,” not “well written or not.” Temperature, empathy, and style should still be left to the judge model.
Layer 3: Targeted per-round repair instead of full rewrites
When the first round fails, the most effective strategy is not to swap models and start over, but to feed the specific errors back and let the model make one targeted fix. A full rewrite introduces new randomness and the problem recurs; targeted repair makes each round converge toward a clear goal.
import osMAX_REPAIR = 2 # Cap on repair rounds; structural issues are usually solved in one rounddef call_model(prompt: str, temperature: float = 0.5) -> dict:resp = client.chat.completions.create( # reuse the gateway client from 2.2model=os.environ["MODEL_ID"],messages=[{"role": "user", "content": prompt}],temperature=temperature,max_tokens=4096,timeout=60,)return {"content": resp.choices[0].message.content or "","finish_reason": resp.choices[0].finish_reason,}def generate_with_repair(base_prompt: str) -> dict:prompt, record, issues = base_prompt, None, []for attempt in range(MAX_REPAIR + 1):record = call_model(prompt, temperature=0.5 - 0.1 * attempt)issues = validate(record)if not issues:return {**record, "attempts": attempt}prompt = base_prompt + "\n\nYour previous output failed validation. Fix only the issues below and output the complete JSON again:\n" \+ "\n".join(f"- {i}" for i in issues)return {**record, "attempts": MAX_REPAIR + 1, "issues": issues, "need_human": True}
Four costs to watch in production:
- Round cap: without a cap, cost and latency only climb; 2 rounds already cover the vast majority of structural problems.
- Token cost: each round resends the full context, so the worst case costs about (1 + MAX_REPAIR)× a single round — estimate against peak QPS.
- Concurrency and rate limiting: targeted repair is a serial, multi-round call that lengthens how long a single request holds a connection, so pair it with semaphore-based limiting, timeouts, and a circuit breaker to keep repair traffic from overwhelming the main path.
- Fallback strategy: if retries still fail, you must explicitly flag “needs manual review” — never return the last unqualified output as the result.
If the model supports reasoning-control parameters, you can also lower the thinking effort or route copywriting tasks through a non-deep-thinking mode. The basis for that decision should be the same evaluation data, not a subjective impression.
Layer 4: Turn evaluation into a pipeline gate
A one-time evaluation can only answer “who is better today”; a pipeline can continuously answer “is it still reliable after a change.” Make the evaluation a pre-release gate and define what happens when it fails:
- Case-set versioning: cover ordinary scenarios, boundary scenarios (character limits, extreme batch information), and adversarial scenarios (multilingual output, embedded sensitive words).
- Automatic regression: run the tests automatically after every prompt or model-config change and output a dimension-level comparison report.
- Admission thresholds: four-deliverable completeness rate ≥ 98%, supporting-post character-hit rate ≥ 95%, safety & compliance average ≥ 0.90.
- Blocking: a version below threshold terminates the pipeline immediately; the artifact does not advance.
- Rollback and review: when an online version’s metrics drop below threshold, first roll back to the last prompt/model version that passed the gate, then trigger manual review and alert the owner.
import sysGATES = {"completeness": 0.98, "length_hit": 0.95, "safety": 0.90}def gate_report(metrics: dict) -> bool:failed = []for name, threshold in GATES.items():value = metrics[name]print(f"{name}: {value:.2%} (threshold {threshold:.0%})")if value < threshold:failed.append(name)if failed:# Gate failed: block the release and alert, deferring to manual review# before deciding whether to roll backprint(f"Gate failed: {failed}; release blocked, manual review required")return Falsereturn Trueif __name__ == "__main__":sys.exit(0 if gate_report(metrics) else 1) # A non-zero exit code makes CI fail directly
The supporting observability is just as essential: write results to a logging service so they are searchable by model, version, and case; connect key metrics to monitoring and alerting so a drop in completeness or character-hit rate triggers a notification; sample production calls daily for evaluation; and periodically spot-check the judge’s scoring to prevent the judge model’s updates from drifting the scoring standard. This whole “batch collection → archive → gate → alert” chain runs on any cloud environment that supports container auto-scaling and logging/monitoring services — there is no need to build an entire ops stack yourself.
Methodology Takeaways and Four Common Pitfalls
Three takeaways:
- Distinguish capability dimensions from constraint dimensions. The former measures how well it writes; the latter measures whether it complies. Averaging them together is the easiest way to hide delivery risk.
- Rule-based checks come before the model judge. Anything code can decide — count, length, format, completeness, required elements — should never be handed to the judge model.
- Evaluation must be reproducible. Fix the case set, sampling parameters, and collection script; attach version, timestamp, and raw responses to results for later tracing.
Four pitfalls:
- Looking only at the total score — the gap between 0.04 and 0.97 only becomes clear once you break it down by dimension.
- Treating a single evaluation as a final selection decision — online versions keep updating, so conclusions need periodic re-testing.
- Overlooking a truncated ending — reading by eye usually catches the opening, but truncation happens at the end.
- Treating evaluation as a one-time project — prompts, models, and business requirements all change, and without a regression gate, quality silently regresses.
Conclusion
The gap between 0.04 and 0.97 is not about who understands crisis PR better, but about who can turn “writing” into “delivery.” For developers, this means three things: at the prompt level, replace vague descriptions with an output contract; at the engineering level, backstop output with deterministic validation; and at the process level, keep evaluation continuously effective with a pipeline gate. The endpoint of model evaluation is not a leaderboard but a trustworthy, reproducible, sustainable quality-assurance capability. Only when every model change and every prompt adjustment can be verified quickly does the “crash scene” truly become a manageable engineering risk.
