LLM Brand Copywriting Evaluation: Why Scores Ranged from 0.92 to 0.35
A structured evaluation of 10 language models writing a brand story from the same interview reveals why their average scores diverged. The largest gaps came from format failures and factual overreach—not writing style—and point to practical ways to improve prompts, validation, and evaluation pipelines.
1. Why a “Write a Story” Task Deserves a Rigorous Evaluation
Turning a rough interview transcript into a brand story is a common task before a generative model goes into production. Teams often summarize the result with a vague judgment: “It’s not quite right.” But subjective reading alone rarely explains what went wrong, which models fail in which conditions, or whether a prompt change or model update actually helped.
This evaluation gave the same interview about a grandmother who had knitted socks by hand for 40 years to 10 language models. Each model wrote a brand story, and every draft was scored against the same seven criteria. The highest and lowest average scores were 0.92 and 0.35—nearly a 2.5-fold difference.
The goal, however, is not to establish a lasting ranking. Model rankings change as providers update their systems. The more useful goal is to identify the sources of the score gap and turn each one into an actionable improvement.
1.1 The Task Tests Discipline, Not Style
The assignment was to write a brand story of no more than 400 Chinese characters for the top of an e-commerce product page. The source was an unedited interview in which a woman described her grandmother’s 40 years of hand-knitting socks.
The challenge was not elegant prose. It was following constraints: do not add facts that the source does not support, and do not include explicitly prohibited content. The interview contained four kinds of traps, testing factual accuracy, compliance, editorial judgment, and formatting.
| Trap type | Source detail | Correct handling | How a model can overstep |
|---|---|---|---|
| Factual | The grandmother’s old knitting machine was used only for sample socks; newer machines had long been used for mass production | Say, “The first sample sock for each design still comes from Grandma’s old knitting machine” | Imply that every product is made by hand on the old machine |
| Compliance | The phrase “Xinjiang cotton” is subject to restrictions in advertising | Avoid the phrase or use a neutral alternative such as “selected long-staple cotton” | Use “Xinjiang cotton” directly in the brand story |
| Editorial selection | An abandoned attempt involving 2,000 numbered socks | Leave it out so it does not distract from the main story | Present the failed attempt as a detail that adds authenticity |
| Formatting | The words “craftsmanship,” “heritage,” and “warmth” are explicitly prohibited | Avoid all three and use concrete details instead | Include any of the prohibited terms |
The distinction between “the first sample sock comes from the old machine” and “all socks are made on the old machine” is only a matter of scope, but it is the central factual test in this task.
2. Evaluation Setup: How the Scores Were Produced
A credible evaluation starts with a transparent methodology. This run used the following setup:
- Execution environment: A general-purpose scripting language and an LLM calling SDK compatible with common HTTP APIs were used to access 10 models concurrently through a unified gateway.
- Judge: A model that did not participate in the generation task acted as the evaluator. It scored each of the seven dimensions from 0 to 1, with separate deduction anchors for each dimension.
- Version timing: All scores reflect the models’ online versions on the collection date. Providers may update model weights without notice, so rerunning the pipeline is the best way to reproduce results.
The methodology is also a reusable evaluation standard. Three principles matter:
- Anonymize model identities. Use anonymous IDs so readers’ expectations about specific models do not bias their interpretation of the scores.
- Separate the judge from the candidates. The judging model should not compete, reducing the risk that it favors its own output.
- State when the results were collected. LLM services change frequently; an evaluation score is a snapshot in time.
3. Scoring Design: Turning “Poor Writing” into Attributable Scores
Many model evaluations offer only a single overall comment, such as “fluent but somewhat stiff.” That does not identify the cause: Was the problem factual accuracy, compliance, or simply style? Without attribution, it is difficult to know what to improve.
This evaluation divided the total score into seven dimensions, each with its own deduction anchors.
| Dimension | What it measures | Typical deduction anchor |
|---|---|---|
| Audience fit | Whether the story speaks to people considering a purchase rather than to readers in general | Sentimental throughout, with little information relevant to a buying decision |
| Compliance | Advertising risks and explicitly prohibited terms | Use of restricted origin-related marketing language or any prohibited term |
| Factuality | Whether claims are supported by the source | Treating “the old machine was used only for sample socks” as the production method for all products |
| Format | Length limit, structure, and cleanliness of the deliverable | More than 400 characters, or inclusion of reasoning and analysis instead of finished copy |
| Language | Fluency and precision | Repetition or awkward word choice |
| Persuasiveness | Whether concrete, credible details can support a purchase decision | Reliance on adjectives without verifiable detail |
| Editorial selection | Whether the chosen details serve the main story | Inclusion of failed attempts or irrelevant details |
The judge prompt followed this structure, with one key requirement: include at least one deduction anchor for every dimension.
You are an expert reviewer of brand copy. Evaluate the candidate only against the source material below. Do not introduce information that is not in the source.[Original interview]<interview transcript>[Task requirements]- Write a brand story for the top of an e-commerce product page, no more than 400 characters- Permitted statement: "The first sample sock for each design still comes from Grandma's old knitting machine"- Do not imply that all products are handmade on the old machine- Prohibited words: "craftsmanship," "heritage," "warmth"- Handle other origin-related marketing claims in accordance with applicable compliance requirements[Evaluation rules]Score each of the following seven dimensions from 0 to 1, with at least one deduction anchor per dimension:audience fit, compliance, factuality, format, language, persuasiveness, editorial selection[Output format]Output JSON only. Do not include anything else:{"scores": {"audience fit": 0.0, "...": 0.0},"anchors": {"compliance": ["reason for deduction and supporting text"]},"overall": 0.0}
This design makes errors easier to diagnose. If a model scores 0.45 on compliance and 0.05 on format, it has made two distinct mistakes: it crossed a compliance boundary and included its reasoning in the deliverable. A single judgment such as “poor writing” would blur the two and could lead to the wrong fix.
Judges also have preferences. A judge may favor elaborate prose or reproduce patterns from its training data, creating systematic scoring bias. Section 7 discusses these limitations.
4. Results: A Crowded Top Tier and a Sharp Drop at the Bottom
The ranking showed a common pattern: the top models were close together, while scores dropped sharply toward the bottom.
This illustrates the “average-score trap.” A model can earn a high average through consistent performance across most dimensions, or fall to the bottom because of a collapse in just one. Looking at the overall score alone is not enough; the scores need to be examined dimension by dimension.
| Dimension | Highest score | Lowest score | Gap |
|---|---|---|---|
| Format | 0.99 | 0.05 | 0.94 |
| Factuality | 0.98 | 0.42 | 0.56 |
| Overall average | 0.92 | 0.35 | 0.57 |
- The largest gap was in format (0.99 to 0.05). The leading models had almost no format issues, while the lowest-scoring model received almost no credit. This was an all-or-nothing failure.
- Factuality had the next-largest gap (0.98 to 0.42). The lowest score was not zero, suggesting that most models did not fabricate an entire story; instead, they got most details right but overstepped in a few places. Such errors are easy to miss in manual spot checks.
- The overall gap was 0.57, but the format gap was 0.94. Averaging substantially obscured the single-dimension failure. A ranking based only on overall scores would understate the fact that the bottom model failed even basic format requirements.
5. Root-Cause Analysis: Where the Score Gap Came From
The judge’s deduction anchors point to four root causes. The first two correspond to the dimensions with the largest score gaps.
5.1 Root Cause One: Reasoning Leaked into the Deliverable
Some models included their reasoning verbatim in the final copy. Before the brand story, they wrote lines such as “First, I need to analyze the key information in this interview” or “Next, I should emphasize the grandmother’s image,” sometimes followed by numbered analysis steps.
For this task, such text is not a minor flaw; it contaminates the deliverable. Copy at the top of a product page may be rendered as submitted, so a sentence of model self-talk could become a production incident. The judge treated this issue as close to an automatic failure, explaining scores as low as 0.05.
Reasoning can help with factuality and logic. The problem is that it must not appear in the final response. Teams can address this at three levels:
- Prompting: Explicitly require the model to output only the brand story, with no analysis, steps, or reasoning.
- Interface: When a service explicitly separates reasoning from the final response, pass only the final response into the delivery pipeline.
- Post-processing: Run a cleanliness check on the final text to catch anything that slips through.
5.2 Root Cause Two: A Limited Detail Was Generalized to the Whole Product
The interview’s statement that the grandmother used the old machine to knit sample socks applies only to the first sample sock for each design. During generation, a model may expand this into the brand’s central claim: “Every pair of socks is hand-knitted by Grandma on her old machine.” The story may sound more moving, but the facts have changed.
This is a form of completion bias. Models tend to make a narrative feel complete and its emotional arc more pronounced. The more complete the story becomes, the further it can drift from the source. The minimum factuality score was 0.42 rather than zero, indicating that most models made only one or two errors instead of inventing an entire story. That is precisely why spot checks can miss the problem.
A practical detection rule is to flag sentences for human review when words such as “every,” “all,” or “entirely” occur alongside limited details such as “old machine,” “hand-knitted,” or “Grandma.”
5.3 Root Cause Three: Misjudging Compliance Boundaries
Compliance deductions mainly involved two issues:
- Prohibited terms: “Craftsmanship,” “heritage,” and “warmth” are common in brand copy and overlap with models’ default writing patterns. Without explicit constraints, they can appear naturally.
- Restricted claims: Origin-related marketing language may be subject to advertising restrictions, depending on the applicable rules and the brand’s own guidance. A model cannot determine from the source alone whether such a claim is appropriate, so the safest approach is to avoid it.
These errors can be detected reliably with keyword matching. They are engineering problems rather than capability problems.
5.4 Root Cause Four: Poor Editorial Selection
The detail about abandoning an attempt after testing 2,000 numbered socks is one of the most vivid parts of the interview. A model may be drawn to it because specific details can make a story feel authentic. But it is unrelated to the main narrative and shifts the reader’s attention toward failure, weakening the story about the grandmother and hand-knitting. The underlying challenge is deciding what belongs in the story and what should be left out.
5.5 Mapping Root Causes to Score Gaps
| Root cause | Main dimensions affected | Failure pattern | Can rules detect it? | Remediation layer |
|---|---|---|---|---|
| Reasoning leakage | Format | Contaminated output and sharp score drop | Yes, with pattern matching | Prompt, interface, and post-processing |
| Overgeneralization of details | Factuality | Local, subtle overreach | Partially, with sentence-pattern rules | Prompt constraints and human review |
| Compliance misjudgment | Compliance | Prohibited terms or restricted claims | Yes, with keyword matching | Post-processing and prompt constraints |
| Poor editorial selection | Editorial selection, persuasiveness | Main story loses focus | Difficult; requires semantic judgment | Prompt constraints and model-based judging |
The biggest difference between the top and bottom models was not literary style. It was a more basic discipline: do not add unsupported facts, do not include prohibited content, and do not expose the analysis process.
6. Turning Evaluation Results into Engineering Practice
The goal of evaluation is not a leaderboard; it is a repeatable improvement process. The following prompt template and validation code can be adapted directly.
6.1 Prompting: Use Allow and Deny Lists Instead of “Make It Warmer”
Write a brand story for the top of the product page for [brand name].[Writing requirements]- Length: 350–400 characters- Output only the brand story. Do not include analysis, steps, reasoning, or explanations.- Use only facts in the interview. Do not add information that is not mentioned.- The old knitting machine was used only for sample socks. You may write:"The first sample sock for each design still comes from Grandma's old knitting machine."Do not use words such as "every pair," "all," or "entirely" to describe the old machine as the main production method.- Prohibited words: craftsmanship, heritage, warmth- Do not make ingredient claims tied to a specific place of origin.[Interview]<interview transcript>
The template follows three principles:
- Make constraints testable. “350–400 characters” is more actionable than “keep it short,” and “do not use ‘every pair’ to describe the old machine” is clearer than “be factually accurate.”
- Provide permitted wording as well as prohibited wording. An allowlist helps prevent the model from becoming so cautious that it avoids all relevant facts.
- Define the deliverable boundary. “Output only the story” is a direct safeguard against reasoning leakage.
6.2 Post-Processing: Three Checks Before Delivery
import re# Check 1: Output cleanliness — flag reasoning leakage and formatting contaminationLEAK_PATTERNS = [r"(I (first|need to|should|will next)|let's|first[,]|step\s*\d)",r"```|#{1,6}\s|\*\*[^*]+\*\*", # Code blocks, headings, bold markup, and similar formatting]# Check 2: Compliance — prohibited terms and restricted claimsBANNED_WORDS = ["craftsmanship", "heritage", "warmth"]# Check 3: Factuality — flag broad claims and limited details in the same sentenceCOVER_ALL = r"(every pair|every single|all|entirely|one after another)"LIMITED_DETAIL = r"(old knitting machine|hand-knit|Grandma.*knit)"def preflight(text: str, max_len: int = 400) -> dict:"""Run pre-delivery checks and return a pass/fail result with issues."""issues = []for pat in LEAK_PATTERNS:if re.search(pat, text, re.IGNORECASE):issues.append(f"Format: possible reasoning leakage or formatting contamination /{pat}/")hits = [word for word in BANNED_WORDS if word.lower() in text.lower()]if hits:issues.append(f"Compliance: prohibited terms found {hits}")if len(text) > max_len:issues.append(f"Format: text is approximately {len(text)} characters, above the {max_len}-character limit")for sentence in re.split(r"[.!?\n]", text):if re.search(COVER_ALL, sentence, re.IGNORECASE) and re.search(LIMITED_DETAIL, sentence, re.IGNORECASE):issues.append(f"Factuality: possible overgeneralization of a limited detail — "f"“{sentence.strip()}”")return {"passed": not issues, "issues": issues}
Rule-based checks are not a complete solution. They should catch errors that can be identified deterministically:
- Use rules to block prohibited terms, excessive length, and obvious reasoning leakage.
- Use a judge model and human spot checks for semantic overreach, including paraphrased claims that evade simple matching.
6.3 Evaluation: A Reproducible Batch Pipeline
Batch evaluation is a generation task and can be run as a scheduled, traceable pipeline:
Unified gateway (concurrent access to multiple models)│▼Orchestration service (tasks split by model; automatic retries on failure)│├── Generation service ──→ Candidate copy (multiple samples)│ ││ ▼│ Rule-based validation│ ││ ▼│ Judge evaluation (seven dimensions with anchors)▼ ▼Object storage: archive raw outputs and scoring details by date, model ID, and task version│▼Logging service: retain call traces, latency, and failure reasons│▼Monitoring and alerts: notify when averages, per-dimension distributions, or format failure rates shift
Three practices are especially important:
- Sample more than once. Outputs vary even when generation settings are similar. Multiple samples per model, reported with both mean and variance, help distinguish model differences from random variation.
- Control the judging process. Hide model identities and shuffle evaluation order to reduce judge bias.
- Keep results comparable. Fix the task, record versions, and include dates so results from different runs can be compared.
7. Scope and Limitations
Every evaluation has limits, and the findings here are no exception:
- A single run is only one snapshot. This was one round with one sample per model. The results hold only under those conditions and should not be treated as a definitive model ranking.
- Model versions change. The scores reflect the online versions available on the collection date. Rerun the evaluation after provider updates.
- Judges have preferences. A judge’s style preferences and language habits can affect scores. Mitigations include deduction anchors, a fixed judge version, human review of key findings, and clear reporting of judge details.
- Results may not transfer to other tasks. Brand stories are only one type of generation task. Different tasks may require different dimension weights, so these findings should not be applied automatically elsewhere.
8. Conclusion
The real test in this brand-story task was not literary flair but discipline. The gap between 0.92 and 0.35 came mainly from four issues:
- Whether reasoning stayed internal.
- Whether a limited detail was generalized to the whole product.
- Whether prohibited terms and restricted claims were avoided.
- Whether irrelevant material was left out.
Teams selecting models or improving a generation pipeline can proceed in four steps:
- Constrain prompts with explicit allowlists and denylists.
- Use rule-based validation to enforce basic delivery requirements.
- Apply seven-dimensional scoring anchors to locate remaining gaps.
- Track performance over time with a fixed task and multiple samples.
Only by turning “Is this good?” into a set of attributable engineering metrics can teams answer the more useful question: “Where did the score gap come from?”
