Evaluating LLMs on Tightly Constrained Short Copy: Scoring Dimensions and Failure Modes

    This article uses a single, demanding task — writing an opening of no more than 180 characters for a community Q&A answer that must contain an identity, a counterintuitive conclusion, an admission of failure, and a closing hook — as a test bed for comparing large language models. It walks through a reproducible evaluation pipeline (single-day collection, a unified invocation protocol, a judge separated from the contestants, and explicit version risk), provides a minimal runnable Python skeleton for invoking, persisting, judging, and aggregating results, and breaks subjective quality into seven orthogonal 0–1 scoring dimensions. It then analyzes two dominant failure modes, fabricated evidence and chain-of-thought leakage, and argues that format compliance is converging across model families while factual grounding remains the real dividing line. It closes with three practical recommendations for model selection and deployment.

    Why Tightly Constrained Short Copy Makes a Good Test Case

    How you design the evaluation task shapes the conclusions more than the model you are testing. Ask any model to “write an article about learning piano” and nearly all of them will return something readable; the differences get diluted by the slack that long-form writing allows. What actually separates models are tasks that stack constraints and leave almost no room for error.

    This evaluation uses one target: write an opening of at most 180 characters for a community Q&A answer. This kind of short copy looks simple, but it has to fit four elements inside a very small character budget:

    • Identity framing: establish “who I am” in the first person, quickly;
    • Counterintuitive conclusion: lead with a judgment that runs against intuition;
    • Admission of failure: candidly disclose a failed or unflattering experience to earn credibility;
    • Closing hook: leave enough suspense to keep the reader reading.

    The key point is that a tight constraint turns over-delivery into a deduction. One extra self-congratulatory sentence, one invented research finding, or one fragment of internal reasoning leaking into the body text is enough to break the balance of the whole structure. Put differently, this scenario naturally amplifies differences in judgment about what should and should not be said, which makes it well suited to side-by-side model comparison.

    A Reproducible Evaluation Pipeline

    The most common source of trouble in model evaluation is not the model but the process. To make the conclusions hold up, the measurement protocol has to be pinned down:

    1. Single-day collection: run all candidate models on the same day in the same network environment, so that version drift across different points in time does not introduce bias;
    2. Uniform invocation protocol: issue requests concurrently through one standard interface, persist each response to disk as JSON, then hand it to the judging stage;
    3. Separate the judge from the contestants: an independent judge model scores by dimension and does not compete, which avoids a model grading itself;
    4. Record version risk explicitly: scores correspond only to the hosted version on the day of collection. If a vendor silently updates its weights, the numbers drift, and old figures represent only that moment in time.

    Below is a minimal runnable skeleton that chains invocation, persistence, judging and aggregation together:

    1. import json, time
    2. from concurrent.futures import ThreadPoolExecutor
    3. PROMPT_TEMPLATE = """Answer this community question in the first person: {question}
    4. Hard constraints:
    5. 1. The whole response must not exceed {max_chars} characters;
    6. 2. It must contain all of: identity framing / counterintuitive conclusion / one admission of failure / closing hook;
    7. 3. Do not invent data, cases or research conclusions; use generalized wording when you have no reliable basis;
    8. 4. Output the body text only. Do not output the reasoning process, and do not add subheadings or bullet labels.
    9. """
    10. def call_model(client, model_id, prompt):
    11. resp = client.chat.completions.create(
    12. model=model_id,
    13. messages=[{"role": "user", "content": prompt}],
    14. temperature=0.7,
    15. )
    16. return resp.choices[0].message.content
    17. def judge(judge_client, rubric, question, answer):
    18. sys = ("You are a scoring judge. Output a score from 0 to 1 for each given dimension "
    19. "and return strict JSON."
    20. f"Dimension definitions: {json.dumps(rubric, ensure_ascii=False)}")
    21. resp = judge_client.chat.completions.create(
    22. model="judge-model",
    23. messages=[{"role": "system", "content": sys},
    24. {"role": "user",
    25. "content": f"Question: {question}\nAnswer: {answer}"}],
    26. temperature=0.0,
    27. )
    28. return json.loads(resp.choices[0].message.content)
    29. def run(candidates, judge_client, question, max_chars):
    30. prompt = PROMPT_TEMPLATE.format(question=question, max_chars=max_chars)
    31. records = []
    32. with ThreadPoolExecutor(max_workers=8) as pool:
    33. for m in candidates:
    34. ans = call_model(m["client"], m["id"], prompt)
    35. s = judge(judge_client, RUBRIC, question, ans)
    36. s["avg"] = round(sum(s.values()) / len(RUBRIC), 3)
    37. records.append({"model": m["label"], "answer": ans, "scores": s})
    38. with open(f"eval_{int(time.time())}.json", "w", encoding="utf-8") as f:
    39. json.dump(records, f, ensure_ascii=False, indent=2)
    40. return records

    Keeping the raw responses on disk is deliberate: beyond looking at scores, you need to go back and inspect where things went wrong. Many conclusions only surface when you compare the original outputs line by line.

    Seven Scoring Dimensions: Turning “Written Well” into Measurable Items

    Whether copy is “written well” is subjective, but evaluation has to be objective. A workable approach is to break overall quality into seven orthogonal dimensions, score each from 0 to 1, then average them. These dimensions are not tailored to a single task; they are a general abstraction for tightly constrained persona copy:

    Dimension What it measures Typical low-score behavior
    Format compliance Hard constraints such as length, structure and forbidden output Exceeding the character limit, adding subheadings on its own
    Consistency Whether identity, tone and timeline stay self-consistent Claiming to be a beginner at the start and a lifelong practitioner later
    Information disclosure Whether limits and failures are disclosed appropriately Over-polishing, avoiding the real situation
    Factual grounding (no fabrication) Whether data, citations or cases are invented Making up “one study shows that…”
    Persona Whether the character is vivid and believable Generic, AI-sounding, no personality
    Platform tone Whether it fits the target community’s expression habits Bureaucratic register, marketing register
    Cost-effectiveness The ratio of quality to invocation cost Mediocre quality at an abnormally high cost

    The main value of decomposing the score is that it turns “who is better” into “who is stronger on which capability”. Two models whose averages differ only slightly can have very different dimension profiles, and that structural difference is more actionable than a single overall number.

    Two Typical Failure Modes

    On the same prompt, the models at the bottom are usually not failing because their writing is bland. They make errors of an entirely different nature, and telling those errors apart matters more than the ranking itself.

    Failure 1: Fabricated evidence (hallucination)
    To make a counterintuitive conclusion sound credible, the model casually invents external support: a study, a statistic, someone’s experience. The problem is that in short copy every word carries weight. A single fabricated “research shows” instantly punctures the credibility of the whole passage. This is exactly why the factual-grounding dimension exists: it penalizes not writing quality but the decision to fabricate when fabrication was not required.

    Failure 2: Chain-of-thought leakage
    Another class of models writes its internal reasoning straight into the output, for example first laying out “first I need to establish the persona, then I need to drop the conclusion…” before producing the actual content. This violates the hard constraint of body text only, and what the reader gets is a passage with the scaffolding exposed. The failure is not about whether the content is correct, but about failing to hold the boundary.

    These two failures point to two distinct capabilities. The first is factual-constraint capability: distinguishing what it knows from what it does not, and refusing to generate without grounding. The second is instruction-boundary capability: strictly following output format and role constraints. A model can be flawless on format and still collapse on factual restraint, and vice versa. This is why a near-perfect format compliance score can still end up separated on the total.

    Format Compliance Is Converging; the Dividing Line Is Not Fabricating

    Looking across Chinese-built and international model families, one trend stands out: format-following ability is converging fast. The visible hard constraints — character limits, complete structure, taking or dropping content as instructed — differ very little among mainstream models, and the leading candidates can be squeezed into an extremely narrow score band.

    What actually opens the gap is an invisible soft constraint like factual grounding. It does not ask the model to “write as if”; it asks the model to hold back when it should: knowing that inventing a data point would make the copy more compelling, yet still choosing to generalize, stay vague, or honestly express uncertainty. That restraint is closely tied to training alignment and value settings, and it is much harder to compensate for through engineering alone.

    This yields a general lesson for evaluation: when explicit capabilities converge, what separates models is whether they hold the line where it would be easy to err but not necessary to. Tightly constrained short copy is exactly such a test.

    Selection and Deployment Recommendations

    Back to the original practical question: for this kind of tightly constrained persona copy, is the most expensive tier necessary? The answer has three layers:

    1. Do not chase the top blindly. When the leading models are separated by a hair on overall score and format compliance is broadly in place, paying a significant premium purely to be “ranked first” is poor value. Mid-tier Chinese flagship models are already sufficient on most dimensions.
    2. Select by dimension weakness. If the business has zero tolerance for fabrication (healthcare, finance, legal explainers), prioritize the stability of each candidate on the factual-grounding dimension rather than the overall score. If format consistency is critical (for example, batch generation of structured card copy), focus on verifying format compliance and instruction boundaries.
    3. Use engineering as a safety net. Writing hard constraints explicitly into the evaluation prompt, adding a “generalize when there is no basis” rule, and running a second validation pass over outputs (character count, forbidden formats, suspicious-number detection) all significantly reduce the failure rate. These measures are independent of which model you choose, and they form the most reliable layer of insurance.

    To sum up: on tasks like tightly constrained short copy, the contest between models is no longer about whether they can write, but about whether they hold the constraints and whether they fabricate evidence. Decomposing the evaluation dimensions, fixing the pipeline, and writing boundary constraints into the prompt is the right way to keep generation quality stable and controllable.