Evaluating LLM Copywriting Generation: Where the Score Gaps Come From
This article examines how to evaluate large language models on constrained marketing copywriting tasks, where the hard part is discipline rather than prose. It presents a reproducible evaluation setup built on a single-round batch run through a unified model gateway, and a seven-dimension scoring rubric (audience fit, compliance, factuality, format, language, persuasion, and material selection) with explicit deduction anchors. It analyzes the score gap between the top and bottom models (0.92 vs. 0.35), traces where the gaps come from (factuality, format/channel leakage, compliance, and material selection), explains the biases of an LLM judge, and offers practical guidance for building a reproducible, regressible evaluation loop for content generation.
Why “Writing Copy” Deserves Its Own Evaluation
When a generative model writes a piece of marketing copy, it may look like a “prose quality” task, but in an engineering context it is really a constrained information-restructuring task: given a set of unorganized primary materials, the model must produce a finished piece that meets format and compliance requirements without fabricating, overstepping, or drifting off-topic.
The difficulty here lies not in the vocabulary but in the discipline. A typical evaluation task might look like this: write a brand story for the top of a product detail page for a handmade wool-sock brand, within 400 words. The founding material is a set of unedited interview quotes with four kinds of “distractors” buried inside:
- The old loom was actually only used to weave sample socks, while mass production used new machines — if the model treats “grandma wove socks on the old loom for decades” as a description of mass production, that is a factual error.
- A certain origin/raw-material claim carries marketing-restriction risk — writing it in directly creates compliance problems.
- A failed attempt (“tried 2,000 numbered pairs, then gave up”) that the task explicitly asks to leave out — including it dilutes the main thread and is a selection-control failure.
- A set of explicitly banned high-frequency marketing words (such as the common “craftsmanship,” “heritage,” and “warmth”) — any hit costs points.
The task allows only qualified phrasing such as “the first sample sock of each model still came from grandma’s old loom,” and does not permit implying that “everything is woven on the old machine.” In other words, what this task really tests is the baseline discipline of “don’t write what isn’t in the material,” not how ornate the model’s language is. Understanding this is the key to understanding why the scores gap out later.
Evaluation Design: Data Methodology and a Seven-Dimension Split
To make the evaluation credible, first make the methodology explicit. Here we use a single-round batch run: a unified model-calling interface, several candidate models launched concurrently, and the same prompt and materials used for generation. The runtime is Python 3.x, with concurrent calls made through a unified model gateway. Every score corresponds to the online version captured on the collection day — and this needs special emphasis: after a vendor silently updates the online model, the weights change, the scores drift, so reproduction must rely on your own pipeline rather than citing historical numbers.
More important than the batch run itself is how scoring is done. Instead of summarizing quality with a single aggregate score, we break quality into seven orthogonal dimensions, each scored independently on a 0–1 scale:
- Audience fit (whether it speaks to the target buyer)
- Compliance (whether it touches banned words or marketing-restricted claims)
- Factuality (whether it stays faithful to the material with no fabrication)
- Format (word count, structure, whether it carries irrelevant content)
- Language (fluency, no redundancy)
- Persuasion (whether it forms a compelling narrative)
- Material selection (whether it picks only main-thread material and omits noise like the failed attempt)
Each dimension is not just a single summary comment; it comes with explicit deduction anchors. The direct benefit is this: if a model shows both “compliance 0.45” and “format 0.05,” these two problems can be attributed separately — the former is a judgment error about “whether it should be written,” while the latter is an output-conformance problem about “how it is presented,” rather than being lumped into a vague “it’s just badly written.” Dimensioning plus anchoring is the precondition for the whole evaluation to locate problems.
A schematic structured judge output can be constrained like this:
{"audience_fit": {"score": 0.8, "reason": "tone is a bit technical, lacks warmth aimed at buyers"},"compliance": {"score": 0.45, "reason": "hits a marketing-restricted origin claim"},"factuality": {"score": 0.42, "reason": "generalizes the sample-sock/old-machine wording into mass production"},"format": {"score": 0.05, "reason": "the deliverable carries reasoning traces and a bulleted draft"},"language": {"score": 0.9, "reason": "sentences are fluent"},"persuasion": {"score": 0.6, "reason": "the main thread holds but the close is weak"},"material_sift": {"score": 0.3, "reason": "includes the failed attempt that was asked to be omitted"}}
The Score Gap: What Happens Between 0.92 and 0.35
With the same materials and the same prompt, average scores across models fall from 0.92 all the way to 0.35. This gap is not a continuous “good vs. better” difference but a set of cliff-edge jumps on a few dimensions. The two most representative:
1. Factuality: From 0.98 to 0.42
The factuality dimension almost determines the “foundation” of the brand story. High-scoring models accurately distinguish “sample socks use the old machine, mass production uses the new machine” and hold the qualified phrasing; low-scoring models take the vague colloquialisms in the interview directly as factual statements and generalize conclusions the material never supported. The 0.98-to-0.42 gap is essentially a divide between two generation tendencies: “faithful to the material” vs. “free improvisation.” For copy that must be delivered externally and involves brand promises, this fabrication is fatal — it quietly turns a copywriting task into a source of compliance risk.
2. Format: From 0.99 to 0.05
The 0.05 on the format dimension points to a very specific failure mode: the model dumps its internal chain of thought straight into the deliverable. It shows up as reasoning drafts, bulleted analyses, and even process text like “first, I should consider…” mixed into the body. The user asked for a finished story within 400 words and instead received a half-finished product with the thinking process attached. This kind of leakage is often not because the model “can’t write,” but because it never separated the “reasoning” and “delivery” channels — reasoning happens in a scratch area, and only the finished product is emitted in the final output area. When scoring, any body text that carries non-deliverable content zeros out the format score directly.
Technical Attribution of the Dimension Gaps
Reading the dimension scores together, we can draw a few engineering-relevant attributions:
Capability stratification happens in “constraint adherence,” not “language generation.” Most models sit high on the language dimension (0.8 and above), showing that generating fluent Chinese copy is no longer a barrier; what truly separates them is the “follow-the-rules” dimensions — compliance, factuality, and material selection. The evaluation focus should therefore shift from “prose quality” to “constraint satisfaction rate.”
Failures are classifiable, not random. The gap between 0.92 and 0.35 can almost always be mapped to a specific anchor: fabrication (factuality), smuggling in the chain of thought (format), hitting banned words (compliance), writing in noise material (material selection). Classifiable means fixable — by strengthening channel separation in the prompt, giving positive and negative examples, and making the banned list explicit, you can raise the corresponding dimension in a targeted way.
A single dimension zeroed out drags down the average. When a model’s format score goes to zero, even if its other dimensions are decent, the average drops significantly. This suggests evaluation shouldn’t look only at the mean but also at the distribution and the lowest dimension — in delivery scenarios, any “fatal” hit on any single dimension is enough to make the whole output unusable.
Bias in the Judge Model Itself
Using a model to score models is the current low-cost, scalable approach, but it introduces judge bias that must be stated explicitly as a boundary in the conclusions:
- Preference bias: The judge model may systematically favor a certain style (such as longer sentences or more “AI-flavored” expression), giving an advantage to models whose style is close to its own.
- Dimension coupling: If the prompt is poorly designed, the judge may score supposedly orthogonal dimensions together (for example, deducting a “badly written” output on both language and format at once), losing attribution ability. Emphasizing anchors, as noted earlier, is precisely to suppress this coupling.
- Stability: Scoring the same output multiple times can produce fluctuation; it is advisable to run several independent scores on the same deliverable and take the mean/variance, quantifying the noise.
Mitigations include: fixing and disclosing the judge prompt, giving judgeable anchors for each dimension, introducing human spot-check calibration when necessary, and running consistency checks on boundary samples.
Reproducibility and Online Version Drift
The most easily overlooked point in an evaluation report is reproducibility. Online models get silently updated, and the 0.92 captured on collection day may become 0.85 a few weeks later. So a rigorous report must:
- Record the collection time and call parameters (such as sampling config like temperature), along with concurrency and retry strategy;
- Freeze the prompt and material versions (ideally brought under version control);
- Keep the scoring logic and anchor definitions as code assets rather than scattered across documents;
- Explicitly declare “the numbers correspond to the version on collection day; to reproduce, run your own pipeline.”
A minimal reproducible evaluation loop looks roughly like this:
def run_eval(task, models, judge, rubric):results = {}for name, client in models.items():outputs = [client.generate(task.prompt, **task.params) for _ in range(task.n)]scores = [judge.score(o, rubric) for o in outputs]results[name] = aggregate(scores) # per-dimension mean + distribution + lowest dimensionreturn results
Only when the materials, prompt, sampling parameters, and scoring rubric are all frozen into executable assets does evaluation change from a “one-off experience” into a sustainable regression capability.
Recommendations for Engineering Practice
Based on the review above, if you want to evaluate the content-generation ability of large models in your own business, proceed in this order:
- Define the task first, then the dimensions: Write “what to write, what the constraints are, what counts as a violation” into judgeable rules, then design the evaluation dimensions accordingly.
- Orthogonal dimensions plus anchors: Score each dimension independently and provide clear add/subtract anchors so problems can be attributed.
- Focus on “constraint satisfaction” rather than “prose quality”: For copy delivered externally, factuality and compliance should carry more weight than language fluency.
- Write channel separation into the prompt: Explicitly require the model to output only in the finished-product area and not smuggle in reasoning, reducing format leakage at the source.
- Manage judge bias: Fix the judge prompt, score multiple times, calibrate with human spot-checks, and write the bias into the report as a known boundary.
- Ensure reproducibility: Freeze versions, parameters, and scoring logic, accept that “scores will drift with the online version,” and rely on your own pipeline.
Conclusion
The same material, the same standard, and a score gap from 0.92 to 0.35 reveals not “whose prose is better” but “who follows the rules better”: whether it stays faithful to the material, whether it holds compliance, and whether it separates the thinking process from the finished deliverable. For developers, the real value of this kind of evaluation lies not in ranking the models, but in using dimensional, anchored, reproducible methods to break the fuzzy notion of “generation quality” into a set of engineering metrics that can be located, fixed, and regressed.
