Multi-Model Brand Story Generation Evaluation: Where Score Gaps Come From
This article explains why scores for brand story generation can differ sharply across models, even when the writing looks similar. It covers four common sources of gaps: format leakage, factual overreach, compliance violations, and weak material selection. It also describes a seven-dimension scoring design, hard validation checks, and an engineering workflow for batch runs and reproducible results.
In brand story generation, a task driven by source material and dense constraints, scores across several large language models can range from about 0.92 down to about 0.35. Many teams first assume the gap reflects differences in writing skill. In practice, the separation usually comes from delivery discipline: do not write what the material does not contain, avoid banned words, keep the format clean, and never let internal reasoning leak into the final output. This article uses an anonymized single-turn evaluation as its background. It discusses where score gaps come from, how to design evaluation dimensions, and how to run batches reliably and reproduce results. The scores cited here illustrate the attribution method only. They do not represent the long-term capability of any specific product.
1. Why Tasks Like This Produce Good Writing but Failed Delivery
The evaluation task can be summarized as follows: write the brand story that appears at the top of a product detail page for a sock e-commerce brand. The text must not exceed 400 Chinese characters. The source material is an unedited interview transcript that contains four types of distractors:
- The old loom only produces sample socks. Mass production uses new machines.
- One raw material claim carries marketing restriction risk.
- A failed attempt, in which 2,000 pairs were numbered and then abandoned, is mentioned.
- Three words are explicitly banned: “匠心” (craftsmanship spirit), “传承” (heritage), and “温度” (warmth).
The client allows the sentence “The first pair of each sample sock is still made on grandmother’s old loom.” The client does not allow any implication that all products are made on the old loom. The failed attempt should be left out because it would dilute the main storyline. The restricted material claim must not appear, and the banned words must be avoided. In other words, this task tests discipline rather than vocabulary: write nothing the material does not support, avoid anything the material prohibits, and keep the delivery format clean.
Once a model includes internal reasoning, explanatory prefixes, disclaimers, or a comparison of candidate versions in its deliverable, the format and compliance scores drop sharply, even if the story itself is good. This produces a clear split. Top-scoring samples meet all requirements for language, facts, and format. Bottom-scoring samples may score around 0.05 on format, and their factual accuracy can fall from near full marks to about 0.4.
2. Seven Dimensions: Breaking “Poor Writing” Into Attributable Failures
The evaluation uses seven dimensions. Each dimension is scored from 0 to 1, and each has deduction anchors. A single overall comment is not enough.
- Audience fit: whether the text suits the reading context at the top of a product detail page.
- Compliance: whether it avoids banned words, restricted claims, and misleading implications.
- Factual accuracy: whether it stays faithful to the material, does not present sample socks as mass-produced, and does not invent details.
- Format: whether it outputs only the deliverable, meets the length limit, and has a clean structure.
- Language: whether it reads smoothly and naturally, with a sound narrative rhythm.
- Persuasiveness: whether it forms a credible brand narrative.
- Material selection: whether it captures the main storyline and drops distracting information.
Scoring each dimension separately makes attribution possible. For example, a sample with low compliance and very low format can be diagnosed precisely, rather than being lumped together as “bad writing.” Judge prompts should spell out the anchors for each dimension. For format, the prompt can specify that reasoning chains, explanatory prefixes, Markdown code fences, JSON fragments, excessive length, or unremoved banned words each lead to deductions. For factual accuracy, the prompt can state that the old loom may be described only for sample socks, and that any implication of mass production on the old loom leads to a heavy deduction.
Judges have their own biases. They may prefer longer texts, more storytelling, or more structured expression. Engineering practices that reduce these biases include: keeping the judge model separate from the models under test, scoring each dimension independently, anonymizing model IDs, randomizing sample order, using two judges cross-checked when needed, and running a calibration set to see whether the judge is overly sensitive to length or format.
3. Four Common Fault Lines in Score Gaps
1. Format gaps: leaked reasoning is the largest source of contamination
Format scores near 0.05 usually do not mean the model cannot write a story. They often mean the model printed its reasoning verbatim. For example, the output might begin with “Analyzing the material: the old loom only makes sample socks…” and then continue with “Final story: …”. A human reader might tolerate this. An automated delivery and scoring pipeline cannot, because the deliverable is contaminated.
A less obvious case occurs when a gateway forwards reasoning fields without stripping them, or when the prompt does not clearly say “output only the brand story text, with no explanations, no title, and no candidate versions.” Once explanatory prefixes enter the deliverable, length checks, banned-word scans, and factual claim extraction all become unreliable. Format and compliance scores fall together.
2. Factual gaps: turning “allowed to say” into “always true”
High factual scores require the model to separate statements that the material supports from conclusions that the material does not support. The material allows the sentence “The first pair of each sample sock is still made on grandmother’s old loom.” It does not allow the conclusion “All socks are made on the old loom.” Weaker samples often confuse sample socks with mass production. Others turn the failed attempt into an inspirational story about “insisting through 2,000 trial pairs.” That version is moving, but it violates the material selection constraints. It also shifts the story’s focus from “forty years of sample sock craftsmanship” to a narrative about a failed attempt.
3. Compliance gaps: banned words and restricted claims need hard checks
Compliance is a hard constraint, not a question of polish. If a banned word such as “匠心,” “传承,” or “温度” appears, the sample should lose points immediately. A restricted raw material claim cannot pass by swapping in a synonym. The engineering approach should combine two methods. Rule-based scanning provides recall. Judge review determines whether a statement creates a misleading implication.
4. Material selection gaps: good language does not mean the right focus
Low material selection scores often appear as text with many details but the wrong priorities. The failed attempt, the technical explanation of the loom, and the origin of the raw material can all take up space in a 400-character limit. The strongest samples keep one storyline: the grandmother spent forty years making only sample socks, mass production was handled by new machines, and the brand kept the ritual of the first sample pair. When language scores are high but material selection scores are low, the model can write well but does not know what to leave out.
4. Engineering the Evaluation Pipeline
The illustrative pseudocode below focuses on the order of operations: run hard checks first, then apply dimension-level judging, and finally aggregate results for attribution.
DIMS = ["受众", "合规", "事实", "格式", "语言", "说服", "选材"]def run_case(task, client):raw = client.generate(task.prompt, temperature=0)out = strip_reasoning(raw) # Illustrative: strip reasoning fields and explanatory prefixeshard = hard_checks(out, task)scores = {d: judge(task, out, d, task.anchor[d]) for d in DIMS}return {"hard_fail": hard, "scores": scores, "mean": mean(scores)}
Hard checks should cover at least the following: the text does not exceed 400 characters, it contains only the story text, it includes no banned words, it contains no restricted raw material claims, and it includes no misleading assertions such as “all products use the old loom.” On the judging side, each of the seven dimensions is scored independently against its anchors.
Records should keep the collection time, runtime environment, prompt version, material version, and judge version. Server-side versions can drift and change scores, so reproduction should rely on rerunning the fixed pipeline.
5. Recommendations
First, separate reasoning mode from delivery mode. The prompt should state explicitly that the model outputs only the final brand story, without explanations, outlines, or candidate versions.
Second, build a material fact sheet that lists separately what is allowed, what must not be inferred, the banned words, and the restricted claims.
Third, validate output against a schema and strip reasoning in post-processing, so that reasoning chains cannot reach the deliverable.
Fourth, write anchored judge prompts for each dimension to avoid bias from overall impressions.
Fifth, do not rely on average scores alone for top-scoring samples. Check for hidden risks, such as high language scores paired with low factual accuracy.
The core finding is that a gap between 0.92 and 0.35 does not mainly reflect whether a model can write a story. It reflects whether the model can deliver consistently under constraints. Make format, facts, and compliance hard checks, and assign language, persuasiveness, and material selection to dimension-level judging. This turns score gaps into a reproducible, attributable engineering problem that can be improved.
