LLM Text Generation Evaluation: Why Score Gaps Reflect Instruction-Following, Not Writing Quality

    This article presents a practical test of how leading large language models handle a brand-story generation task built to probe discipline rather than prose. It explains why judge scores ranging from roughly 0.92 to 0.35 reflect instruction-following and factual-constraint abilities—staying within the material and obeying explicit rules—rather than writing quality. It walks through the task design (fact traps, compliance landmines, material-selection tradeoffs, and hard constraints), a seven-dimension scoring rubric with anchored deductions, the failure modes that create score cliffs (factual hallucination, constraint breakdowns, and reasoning leakage into deliverables), a reproducible evaluation pipeline, and the method's own limits. It closes with engineering safeguards such as structured output, pre-delivery validation, and continuous re-evaluation.

    1. Score Gaps Are Not a Measure of Writing Quality: A Misread Conclusion

    Give several leading large language models the same raw, unedited interview transcript and ask each to generate a version of a brand story. Judge scores range from about 0.92 at the top to about 0.35 at the bottom—a gap of nearly 0.6. At first glance, many people read that curve as evidence that “some models write better.” That is a misread.

    What actually separates the models is not their command of phrasing, but their compliance with the generation boundary—whether they can resist writing information that isn’t in the source, and whether they break none of the explicit constraints. When the task is designed to test discipline rather than literary flair, the scores directly reflect a model’s instruction-following and factual-constraint abilities, not its language aesthetics. Understanding this is the prerequisite for building any generation-quality evaluation system: define the evaluation objective clearly first, then talk about scores.

    2. Task Design: Creating Separation with “Distractors”

    A meaningful evaluation is won at the level of task design. The raw material is a rough interview transcript, deliberately seeded with four kinds of distractors that map to four capabilities:

    • Fact traps: The old loom was actually used only to knit sample socks; mass production relied on newer machines. The task permits “each style’s first sample sock still came off grandmother’s old loom” but forbids implying that “all products are woven on the old machine.” This tests whether the model can faithfully reproduce the fact without over-extrapolating.
    • Compliance landmines: The source contains a sourcing claim that carries marketing-use restriction risk; the model must proactively avoid or neutralize it rather than copy it verbatim.
    • Material-selection tradeoffs: The interview includes a failed attempt (“abandoned after trying 2,000 numbered pairs”). The task explicitly requires leaving it out—including it would dilute the main thread. This tests whether the model understands that material selection is itself part of delivery quality.
    • Hard constraints: Explicitly banning three overused buzzwords—“craftsmanship,” “heritage,” and “warmth.” This tests the most basic form of constraint-following.

    In one sentence, the core of this task is: it tests not writing ability, but the discipline of “don’t write what isn’t in the material.” That is also what makes it good material—it breaks the abstract notion of “generation quality” into observable, judgeable behaviors.

    3. Evaluation Method: Seven-Dimension Breakdown with Anchored Deductions

    The most common mistake in evaluation is covering every problem with a single lumped-together overall score. When a sample has both a factual error and a format breakdown at the same time, one score blends them into a vague “poorly written,” which can neither attribute the cause nor guide optimization.

    A more sound approach is to break quality into seven orthogonal dimensions, each scored separately (0–1 per dimension):

    Dimension Focus
    Audience Fit Whether it matches the tone and reader expectations of the brand story at the top of the product detail page
    Compliance Whether it avoids restricted marketing terms and risky phrasing
    Factual Accuracy Whether it stays faithful to the source—no fabrication, no over-extrapolation
    Format Whether word count, structure, and output purity meet the requirements
    Language Whether the expression is fluent, natural, and free of grammatical errors
    Persuasiveness Whether the main thread is clear and able to move the target reader
    Material Selection Whether it makes appropriate tradeoffs, omitting the distractors that should be omitted

    The key design point is that every dimension must have an “anchored deduction” rather than a single vague score comment. For example, the format dimension can specify “exceeding the word limit deducts 0.3; mixing non-body content into the output deducts 0.5.” This way, a model with “compliance 0.45, format 0.05” can be clearly decomposed into two independent problems—one place where phrasing crosses the line, and one place where format has completely failed—rather than being blurred into a single judgment.

    The judging stage can also bring in a neutral, non-participating model as the judge, avoiding the conflict of interest that comes from “a contestant serving as its own judge.” A fixed prompt can require the judge to score dimension by dimension and attach deduction reasons, improving explainability and auditability.

    4. Dimension Review: How the Cliffs Appear

    Lay the seven dimensions side by side, and the scores do not slide down smoothly—they fall off a cliff on specific dimensions:

    • Factual Accuracy: from about 0.98 down to about 0.42. Top models can almost fully reproduce the detail that “the old loom only knit sample socks” without overstepping; bottom models make two typical errors—either writing the old machine as the mass-production workhorse (over-extrapolation), or dropping this key piece of information entirely (information loss).
    • Format: from about 0.99 down to about 0.05. The gap is even more extreme. Strong models strictly hold to the word limit and output only the body text; the worst tier is close to a complete format breakdown.

    The cliffs in these two dimensions point to the same root cause, which the next section unpacks.

    5. Root-Cause Analysis: Three Typical Failure Modes

    1. Factual hallucination and over-generation. Models tend to “complete” the material into a fuller, more appealing story, so they add causes, figures, or scenes that the source never provided. This is especially dangerous in marketing copy, where fabricated content can directly constitute a compliance risk.
    2. Constraint breakdown. Exceeding the word limit, misusing banned words, or writing in a failed attempt that was explicitly meant to be omitted all fall into this category. It reflects not a capability gap but a robustness problem in instruction-following—once constraints pile up, bottom-tier models lose track of one while attending to another.
    3. Reasoning-chain leakage into deliverables. This is the most alarming phenomenon from this test: some models dump their internal reasoning, tradeoffs, and “how should I write this” deliberation straight into the final deliverable. The model’s mutterings end up mixed into the brand-story body, and the format dimension drops to zero instantly. This shows the model has failed to distinguish the boundary between “internal thinking” and “external output”—and for production this is fatal, because the deliverable is presented directly to end users.

    6. A Reproducible Evaluation Pipeline

    Evaluation conclusions must be reproducible; otherwise, the numbers drift the moment a version updates. Below is an illustrative pipeline focused on “per-dimension structured scoring + traceable results,” with no dependency on any specific vendor interface:

    1. # Illustrative code: core structure of a multi-dimension evaluation pipeline
    2. # client is a unified model-invocation wrapper; the judge model and candidate models are decoupled
    3. DIMENSIONS = ["Audience Fit", "Compliance", "Factual Accuracy",
    4. "Format", "Language", "Persuasiveness", "Material Selection"]
    5. RUBRIC = {
    6. "Format": {"over word limit": -0.3, "mixed-in non-body content": -0.5, "missing structure": -0.2},
    7. "Factual Accuracy": {"information not in the source": -0.4, "key fact lost": -0.3},
    8. "Compliance": {"restricted marketing term": -0.5, "risky phrasing": -0.3},
    9. # Other dimensions follow the same pattern; each needs deduction anchors
    10. }
    11. def judge(sample, judge_prompt):
    12. scores, reasons = {}, {}
    13. for dim in DIMENSIONS:
    14. # Score each dimension independently; require the judge to give deduction points and reasons
    15. r = client.chat(judge_prompt, dimension=dim,
    16. rubric=RUBRIC.get(dim, {}), sample=sample)
    17. scores[dim] = r.score # 0-1
    18. reasons[dim] = r.reason # traceable
    19. return scores, reasons
    20. def run_eval(samples):
    21. report = []
    22. for s in samples:
    23. scores, reasons = judge(s, judge_prompt=BASE_PROMPT)
    24. report.append({
    25. "id": s.id,
    26. "scores": scores,
    27. "overall": sum(scores.values()) / len(DIMENSIONS),
    28. "reasons": reasons,
    29. "model_version": s.model_version, # record version to handle silent updates
    30. })
    31. return report

    Design highlights: scoring each dimension independently and keeping deduction reasons makes results attributable; recording the model version and timestamp guards against score drift caused by silent vendor updates; decoupling the judge from the contestants through a unified wrapper makes it easy to swap in and re-run.

    7. The Boundaries of the Method Itself

    Evaluation is not absolute truth, and its limitations need to be recognized clearly:

    • Judge bias. The model serving as judge has its own tendencies—for instance, favoring longer, more ornate answers. Left unconstrained, it will systematically add points to certain styles. Mitigations include forcing the judge to score strictly against the deduction anchors, looking only at facts rather than writing quality, and cross-validating across the same batch of samples.
    • Version drift. Every score corresponds to the online version captured on the collection date. After a vendor silently updates its weights, outputs under the same prompt change and the numbers drift with them. Therefore every score must be bound to a “collection date + model version” and taken from running the pipeline below yourself, not from citing an isolated number.
    • Single-round sampling variance. A single batch run is affected by sampling randomness; it is better to repeat multiple times, report the mean and variance, and avoid drawing conclusions from one run.

    8. Engineering Recommendations for Production

    Finding problems through evaluation is only the starting point; the real goal is to add safeguards on the production path:

    • Enforce structured output. Use explicit output-format constraints (fixed fields, a word limit, a ban on outputting the reasoning process) to compress the room for reasoning leakage and format breakdowns at the source.
    • Validate before and after delivery. Add a programmatic gate before text enters the deliverable: keyword blacklist filtering (blocking banned words and restricted marketing terms), word-count checks, and factual-consistency verification (aligning the generated content’s entities against the source material).
    • Dual fact and compliance review. On the factual side, check “whether each statement can be traced to evidence in the material”; on the compliance side, maintain and dynamically update a list of restricted terms and risky phrasing.
    • Make evaluation routine. Wire the seven-dimension evaluation into the release process and re-run it whenever the model or prompt changes, judging quality shifts by trend curves rather than a single-point score.

    Conclusion

    The gap from 0.92 to 0.35 is essentially a mirror, reflecting the difference between “generation ability” and “generation discipline.” For teams building applications, rather than chasing some vague high score, the better path is to define the evaluation objective clearly, break the dimensions down finely, write the deduction anchors explicitly, and turn results into reproducible, traceable reports. When disciplines such as “never write what isn’t in the material,” “never break a single constraint,” and “never let thinking leak into the deliverable” are reliably met, a high score is just a natural outcome—not a destination to chase.