Benchmarking LLMs on Brand Story Writing: Where the 0.92-to-0.35 Score Gap Comes From
We benchmarked ten large language models on a brand story writing task using seven scoring dimensions under identical prompts. The top model scored 0.92 while the lowest scored 0.35. This article breaks down the evaluation rubric, presents the full results, and analyzes the four key factors behind the score gap: format compliance, hallucination control, dimensional balance, and audience fit. It also offers practical engineering advice for rebuilding this evaluation pipeline in your own business.
Why Brand Story Writing Deserves Its Own Benchmark
Brand stories are among the most demanding types of marketing copy because they require a combination of skills that few other tasks test simultaneously. A good brand story must match the brand’s tone and target audience, avoid compliance risks such as exaggerated claims, stay grounded in the provided source material without inventing company history, and demonstrate strong structure, language, and persuasive technique. Compared with summarization or question answering, brand story writing exposes a model’s ability to balance instruction following, source constraints, and creative expression.
For these reasons, we chose brand story writing as our benchmark task, designed seven scoring dimensions, and ran a horizontal comparison of ten large language models under an identical prompt.
Evaluation Design: How the Seven Dimensions Are Scored
To make scoring reproducible, every dimension has explicit rubric criteria on a 0–1 scale:
- Audience fit: Whether the language style and narrative perspective match the target readership (e.g., young consumers or enterprise decision-makers);
- Compliance: Whether the output contains absolute claims (“the best,” “number one”), fabricated credentials, or exaggerated promises that violate advertising regulations;
- Factual accuracy: Whether the story is strictly grounded in the input material, with no hallucinated founding dates, award records, or performance metrics;
- Format compliance: Whether the output meets the required length, paragraph structure, and heading specifications;
- Language quality: Fluency, vocabulary richness, and sentence rhythm;
- Persuasiveness: Whether the story has emotional tension and naturally leads to the brand’s value proposition;
- Material selection: Whether the model makes sensible choices about which parts of the source material to use and how to organize them, with the key points highlighted.
Each model generated three samples. Reviewers scored them independently against the rubric, and the results were averaged, with final rankings based on the mean score across all seven dimensions.
Results Overview
| Rank | Model | Audience | Compliance | Factual | Format | Language | Persuasion | Selection | Avg |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Model A | 0.91 | 0.98 | 0.91 | 0.98 | 0.90 | 0.88 | 0.91 | 0.92 |
| 2 | Model B | 0.88 | 0.98 | 0.76 | 0.99 | 0.90 | 0.86 | 0.97 | 0.91 |
| 3 | Model C | 0.87 | 0.90 | 0.84 | 0.98 | 0.86 | 0.88 | 0.96 | 0.90 |
| 4 | Model D | 0.86 | 0.94 | 0.76 | 0.98 | 0.82 | 0.84 | 0.96 | 0.88 |
| 5 | Model E | 0.92 | 0.89 | 0.61 | 0.98 | 0.87 | 0.91 | 0.90 | 0.87 |
| 6 | Model F | 0.88 | 0.85 | 0.68 | 0.99 | 0.88 | 0.84 | 0.82 | 0.85 |
| 7 | Model G | 0.78 | 0.96 | 0.76 | 0.56 | 0.89 | 0.82 | 0.68 | 0.78 |
| 8 | Model H | 0.87 | 0.83 | 0.42 | 0.70 | 0.88 | 0.84 | 0.76 | 0.76 |
| 9 | Model I | 0.35 | 0.30 | 0.62 | 0.10 | 0.45 | 0.66 | 0.28 | 0.39 |
| 10 | Model J | 0.35 | 0.45 | 0.58 | 0.05 | 0.32 | 0.52 | 0.20 | 0.35 |
The gap between the top and bottom models is a massive 0.57 points. Below, we break down where that gap comes from.
Four Key Observations Behind the Score Gap
1. Format Compliance Is the First Divide
The top eight models generally scored above 0.70 on format, with most reaching 0.98. The two bottom models scored just 0.10 and 0.05. Their typical failures included completely ignoring the specified word limit, failing to follow the required paragraph structure, and even turning the “brand story” into a feature list. This shows that low-ranking models have a fundamental weakness in parsing long instructions and controlling output structure — this is not a writing-style problem but a basic instruction-following problem. For business scenarios that require producing structured copy at scale, failed format compliance makes the output practically unusable.
2. Hallucination Is the Universal Weakness of Mid-Tier Models
On factual accuracy, even the second-ranked model only scored 0.76, and the fifth-ranked model dropped to 0.61. Typical failure patterns included writing nonexistent founder biographies for the company, inventing specific sales figures, and adding claims like “internationally certified” out of thin air. This kind of hallucination carries extremely high risk in marketing copy — it triggers deductions on both the factual and compliance dimensions simultaneously. Notably, compliance scores and factual scores show a clear positive correlation, suggesting that fabrication and exaggerated claims are two manifestations of the same underlying problem.
3. What Top Models Have in Common: No Weak Dimensions
The top-ranked model scored no lower than 0.88 on any dimension, and achieved 0.98 on compliance — it used no absolute claims in any sample and proactively substituted risky phrasing with neutral expressions like “committed to” and “continuously exploring.” The second-ranked model scored 0.97 on material selection (the most precise use of source material), but its 0.76 on factual accuracy dragged it down. The lesson: the ceiling of overall writing capability is determined by the weakest dimension. Excellence in a single area cannot compensate for a weak spot.
4. Audience Fit: Hard to Differentiate, Easy to Collapse
The top eight models clustered between 0.78 and 0.92 on audience fit, showing that mainstream models have a basic sense of who they are writing for. But the two bottom models scored only 0.35 — their output was stiff in language, inconsistent in narrative perspective, and read like parameter listings rather than stories. Audience fit collapse almost always coincides with a simultaneous drop in language quality (0.45 and 0.32 here), as the two dimensions are tightly coupled.
Engineering Recommendations for Rebuilding This Evaluation
If you need to reproduce this evaluation in your own business, we recommend the following:
# Evaluation pipeline sketch (pseudocode)rubric = {"audience": "Does the perspective and tone match the target audience profile","compliance": "Scan for absolute claims, fake credentials, exaggerated promises","factuality": "Entity-level alignment with input material; flag newly invented entities","format": "Structured validation of word count / paragraphs / headings; can be checked programmatically",}# Score the format dimension with a rules engine; score the rest with reviewer rubrics plus multi-person cross-review
- Automate the format dimension as much as possible: Word count, paragraph count, and heading structure can all be checked programmatically, eliminating subjective bias;
- Perform entity alignment for factuality: Extract the entity set from the input material and from the output; the difference is your list of suspected hallucinations;
- Build a sensitive-word and pattern library for compliance: Absolute claims and promise-style statements can be pre-screened with regex and word lists, then manually reviewed;
- Weight dimensions according to your business: In marketing scenarios, compliance and factual accuracy should carry significantly higher weights than literary polish.
Conclusion
The core takeaway from this benchmark can be summed up in one sentence: the gap in brand story writing lies less in “how beautifully it’s written” and more in “whether boundaries are held.” Top models win through balanced performance across all dimensions, especially stability on the two “bottom-line dimensions” of compliance and factual accuracy. Bottom models lose on basic capabilities: loss of format control and audience misalignment. For enterprise users evaluating models, prioritize hallucination control under source-material constraints — it matters far more to a production content pipeline than subtle differences in writing style.
