Failure Analysis of a Long-Constraint Generation Benchmark
This article explains how to design and audit a controlled benchmark that tests whether large language models can follow many hard constraints when writing short community-style restaurant reviews. It covers why total scores can mislead, how to score seven dimensions separately, and why removing reasoning blocks before scoring can change the ranking. It also describes a reproducible pipeline and a way to diagnose failures by type.
The Task Tests Constraint Memory, Not Writing Ability
A short review looks simple: a few hundred words, a casual tone, and a clear opinion. The difficulty lies in constraint density. A more reliable design gives the model a fixed set of facts and a list of hard rules that can conflict with each other. This forces the model to keep the format, the factual boundaries, and the value judgment in mind at the same time.
Consider sample material about a long-established neighborhood eatery. The years of operation must follow the year given in the material. The main dish is cheap handmade wontons, and the portion size and price are fixed. The queue is long. The owner has a bad temper but remembers regulars’ tastes. The rival shop next door uses semi-finished ingredients and charges more. Payment options are limited. The role is a reviewer who is “picky but fair,” and the text must avoid advertising tone.
Limit the hard constraints to about six. Each one should be verifiable automatically or semi-automatically:
- A maximum title length, and the title must include a specific number from the material. This prevents vague headlines.
- Drawbacks must appear in the text, such as the owner’s temper and the queue length. Praise-only reviews are not allowed.
- Years, prices, and portion sizes must come from the material. The model may not invent a “more reasonable” number.
- Comparisons with competitors may only state facts such as price, ingredients, and wait times. Insulting characterizations are prohibited.
- The ending must give a clear verdict on whether the place is worth it. It cannot stop at “each has its strengths.”
- The full text must fall within a specified word count range, and emoji use has an upper limit.
A review that reads well but misses one of these constraints is unusable in real publication. The scoring goal should therefore be: under a long constraint list, does the model still remember every item? Fluent writing alone does not answer that question.
Why Use Seven Equally Weighted Dimensions, and Why Not Trust the Total Alone
A single total score mixes failures of very different kinds. The judge should score at least the following seven dimensions. Normalize each to a 0 to 1 range, average them with equal weights first, then reweight them according to business needs:
- Compliance: title length, body word count, emoji count, and whether all required facts are present.
- Consistency: prices, years, and wait times do not contradict each other, and no numbers appear that are not in the material.
- Disclosure: drawbacks are reported as written, and negative reviews are not rewritten as “having character.”
- Evidence: judgments are anchored in facts from the material, not vague adjectives.
- Persona: the tone matches “picky but fair,” and the text does not slide into advertising or attacks.
- Platform fit: the text reads as a publishable review, not a report, an outline, or a draft of reasoning.
- Value judgment: the ending gives an actionable verdict on whether the place is worth visiting.
Equal weights make results comparable and easy to review, but they hide differences in importance. For real publication, platform fit usually matters far more than whether the text includes one extra piece of background. Readers see only the published body. Leaked reasoning destroys platform fit even when the other six dimensions look strong. A drop in compliance often comes from attention decay in a long checklist. The model remembers to write about drawbacks but forgets the number in the title, or the word count goes outside the range. These two kinds of failure cannot be fixed by polishing the prose. They require different strategies for prompts, post-processing, and model selection.
The Evaluation Pipeline: Removing Reasoning Blocks Can Change the Conclusion
The general workflow is sequential calls, raw responses saved to disk, and offline scoring. The difference from ordinary concurrent load testing is the preprocessing step in the middle. Some models place their reasoning inside tagged blocks before or after the answer. If you do not remove these blocks, the judge will score the reasoning chain as if it were the review, and platform fit and word count will both collapse. If you remove them in every case, the word count changes meaning. Some models reason at length and produce a short answer, so after removal the text lands at 400 to 500 words. Other models rarely output tags, so removal has no effect on them. The two readings answer different questions.
A more auditable approach is to keep both sets of scores and declare one of them as the primary result in advance:
- Raw-output view: the judge sees the complete string the API returns. This simulates a user or downstream system that publishes without cleaning.
- Cleaned view: explicit rules remove the reasoning tags and their contents, and then word count and platform fit are measured. This simulates a publishing gateway that cleans the output.
The main leaderboard must use one fixed view. If the goal is “connect to a compatible chat interface and publish directly,” the raw-output view is more honest. Reasoning leakage then counts as a platform-fit failure, which reflects a real fault in the product path, not an unfair judge. If the goal is to measure a model’s internal capability ceiling, you can also report cleaned scores in parallel. However, you must not use cleaned scores alone to claim that the output is “ready to publish.”
Write the preprocessing rules down in advance so that no one deletes text by hand based on mood:
- Remove only the paired tags defined in advance, along with the text inside them.
- If a tag is unclosed, mark the entire output as contaminated instead of guessing where the body starts.
- Save both the raw and cleaned outputs to disk, and record the word count and emoji count for each.
- State in the judge prompt whether text inside tags counts as body text, and keep this rule consistent across the run.
Keep the Same Gateway and Run for Calls and Scoring
Do not mix protocols, system prompt wrappers, or temperature settings within one experiment when you reproduce it. Keep the calling side as consistent as possible: use the same compatible chat completion endpoint, the same timeout, and the same sampling parameters. Save each raw JSON response to object storage or a local directory, and name each file with the task ID, candidate ID, and collection time. Run scoring offline from the saved files. Scoring online can compete with generation for the same quota, and the judge might see text that differs from the version that was saved.
The following sketch shows only the pipeline skeleton and does not represent any vendor’s proprietary interface:
import jsonfrom pathlib import Pathdef run_one(client, model, messages, out_path):resp = client.chat.completions.create(model=model,messages=messages,temperature=0,)raw = resp.model_dump()Path(out_path).write_text(json.dumps(raw, ensure_ascii=False),encoding="utf-8",)return raw["choices"][0]["message"]["content"]def visible_text(text, strip_think):if not strip_think:return text# Remove only the paired reasoning tags defined by the experiment protocol;# if a tag is unclosed, treat the whole text as contaminatedstart, end = text.find("<think>"), text.find("</think>")if start != -1 and end != -1 and end > start:return (text[:start] + text[end + len("</think>"):]).strip()if start != -1 or end != -1:return ""return text
The judge side must also be fixed: one judge, one scoring guide, and seven dimensions, each with a score from 0 to 1 and a one-sentence rationale. Do not let a generation model score its own output. The judge prompt should require a check of each of the six hard constraints before the dimension scores are assigned. This reduces the chance that a text is rated highly simply because it reads smoothly.
Declare the Reproduction Conditions Before the Results
These numbers are very sensitive to silent weight updates from providers. A responsible report should declare the collection date, the script language and major dependency versions, whether the same gateway was used, whether the run was single or multiple, the temperature, whether reasoning tags were removed, who the judge was, and whether the seven dimensions were equally weighted. All scores correspond only to the online versions available on the collection day. After a provider update, results often fail to match when the benchmark is rerun a few weeks later. A single run should not be presented as a final ranking of capabilities.
A single run also carries variance. Running the same constraints again may shift compliance and word count scores. If you run only once, limit your conclusions to “the failure modes observed in this run.” Prioritize qualitative phenomena that others can check, such as reasoning leakage, extrapolated facts, and vague judgments. Do not interpret a 0.01 difference in average scores as a stable difference in quality.
From Scores to Model Selection: Identify the Failure Type First
When average scores are close, examine the dimension profile before comparing rankings:
- Platform fit drops sharply while the other dimensions look acceptable: suspect reasoning tags, leftover outline text, or a persona that breaks into a manual. The publishing path should either clean the output or use a model with cleaner output habits. Editing the prompt wording is not a reliable fix.
- Compliance is average while persona and evidence are acceptable: this is typical long-constraint decay. Split the hard constraints into post-generation checks. If a title lacks a number, the text exceeds the word limit, or the value verdict is missing, trigger an automatic retry or fall back to a safer option. Do not expect a single sample to remember everything.
- Consistency is poor: the model chose specificity over the rule against inventing facts. Number the facts in the material as a fact table, instruct the model to cite only numbers in that table, and programmatically compare the set of numbers in the text against the table.
- The value judgment is empty: keep the ending template outside the prompt and let a program check whether the text contains a clear verdict sentence. This is more reliable than lengthening the prompt’s description of what a verdict should look like.
For advertising use, the weighting intuition is that platform fit and compliance matter more than disclosure, and disclosure matters more than subtle differences in persona style. Equal-weight rankings help detect anomalies. Weighted rankings should decide whether a model can enter the content pipeline.
The validation can be lightweight. For example, extract the title length, count emojis, use regular expressions to find four-digit years and price patterns, and compare them against the fact table as sets. Leave the judge to handle semantic items such as persona, the evidence chain, and whether the text insults competitors. Rules should catch failures that rules can catch. Do not hand all checks to another model.
Common Pitfalls
First, announcing a winner based on cleaned text while publishing the raw response. Second, placing the constraints at the end of a long system prompt, where lengthy material drowns them out and the model forgets the later items. Third, listing seven dimensions in the rubric while the actual prompt emphasizes only overall quality, so reasoning leakage gets rewarded as “deep analysis.” Fourth, presenting the ranking from one experiment as a purchasing decision without recording the gateway, temperature, or version date. Fifth, allowing models to add “common industry knowledge” to increase word count, which rewrites years and prices and breaks consistency.
A reliable experiment record should include at least the fact table, the scripts that check the six constraints, the raw responses, the cleaned text, both sets of scores, the judge’s rationales, and the collection time. With these records, others can verify the result without relying on the narrative. They can check whether a model lost points on writing quality or because it leaked its reasoning into the body.
Summary
Short content generation with the same topic works well as a probe for constraint adherence. It is not suitable as a ranking of writing quality. Split failures into compliance decay and output contamination. Fix whether the score uses raw or cleaned output. Use rules to enforce numbers, word counts, and required drawbacks. Only then will the benchmark be useful for content and engineering decisions. When average scores are close, read the dimension profile and the raw responses first. Then decide whether to add a cleaning gateway, add retries, or change how the model is called to improve format compliance.
