Multi-Model Copywriting Test: Who Actually Sounds Human Under Tight Constraints
A 10-model test of short housewarming copy shows that size and brand do not guarantee usable output. The same brief—two messages with audience, length, tone, imagery, and hard bans—produced scores from 0.88 to 0.22. This article covers the scenario, scoring dimensions, failure modes, and how to evaluate models for production copy.
Why a “housewarming wish” is a serious test
When people judge LLM writing, they often start with long summaries, code, or Q&A. In real work, many jobs have little jargon but strict rules on tone, audience, setting, and what not to say.
Writing two housewarming notes looks trivial. It actually stacks constraints:
- Adapt wording to different recipients
- Hit a word limit
- Sound sincere without exaggeration
- Use the home’s real context (a modest renovated resale, not a luxury property)
- Use concrete images instead of abstract wishes
- Avoid clichés
- Deliver finished copy, not a reasoning dump
That mix tests execution: whether the model can turn several natural-language rules into stable, checkable output.
Ten models were tested, from flagship to cheaper general-purpose ones. Top composite score was 0.88, bottom 0.22—about a 4× gap. Scale and market positioning did not fully track quality.
Scenario: one theme, two uses
The brief: write housewarming copy for a client (“Xiaolin”) to send to a friend. The friend moved into a renovated second-hand home of about 60 square meters—not high-end.
Two pieces of copy were required.
1. WeChat / chat message
For a former colleague. Limits:
- Under 50 characters/words (short)
- Natural and warm
- Light tease about the old-home renovation is allowed
- Must stay warm
- Must not read as a formal greeting
The hard part is “familiar but not over the line.” Too formal feels mass-sent; too much teasing can feel like a put-down of the home.
2. Card text
For a close university roommate, on a gift card:
- 80–100 characters/words
- Must include one concrete image tied to a resale or renovation
- Sincere tone
- A wish for future life
- No stale idiom-style clichés
This tests organization: turn “renovated second-hand place” into a lived-in picture (fresh paint, a polished windowsill, traces in old flooring, a lamp that just went on).
Hard constraints that actually separated models
Banned or required:
- No stock congratulations
- No area, appreciation, or price
- No rhyme
- Do not dress a normal home as a mansion
- Do not output analysis
- Two finished drafts, one per task
Models often satisfied parts and failed the whole. Some avoided clichés then closed with “may the new home carry happiness and open a new chapter.” Some used a real image then rhymed like an ad. Worse: they explained strategy at length and never delivered two copy-paste messages. For production, analysis without delivery is a fail.
Scoring: from “sounds nice” to measurable dimensions
Seven dimensions, 0–1 each, then average.
1. Audience fit
Does the copy match relationship and channel? Chat should be light; the card can be fuller. Two near-identical rewrites mean the model did not distinguish contexts.
2. Constraint compliance
Length, bans, required imagery, tone bounds. One clear violation can make the copy unusable.
3. Factual and scene consistency
Not only errors—also whether the scene is reasonable. Calling a modest resale a “mansion,” “large flat,” or talking “upside” ignores the brief and can embarrass the sender.
4. Format execution
Only the required drafts, two clearly labeled pieces, length and structure. Format is easy to ignore; in automation it decides whether downstream systems can parse the output.
5. Natural language
Does it sound like a person? Template feel, mechanical repetition, or over-rhetoric fail even if grammar is fine.
6. Persuasion and emotional completeness
Good wishes feel written for this person. Shared history and modest life detail help; invented details become another kind of cliché.
7. Material quality
Everyday images (old wall, sill, light, storage) beat “new life, new start.” Details must serve tone, not pile up renovation specs.
Results: high scores are not just scale
Public ranking (excerpt):
| Rank | Model | Audience | Compliance | Factual | Format | Language | Persuasion | Material | Mean |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Flagship A | 0.90 | 0.98 | 0.58 | 0.99 | 0.94 | 0.85 | 0.89 | 0.88 |
| 2 | General B | 0.89 | 0.98 | 0.63 | 0.94 | 0.92 | 0.88 | 0.78 | 0.86 |
| 3 | Flagship C | 0.88 | 0.97 | 0.58 | 0.99 | 0.84 | 0.87 | 0.82 | 0.85 |
| 4 | Lightweight D | 0.82 | 0.98 | 0.57 | 0.99 | 0.79 | 0.81 | 0.73 | 0.81 |
| 5 | General E | 0.84 | 0.98 | 0.62 | 0.68 | 0.78 | 0.82 | 0.72 | 0.78 |
Patterns:
First, compliance scores were generally high: most models catch explicit bans and format. Following rules is not the same as writing well.
Second, factual scores lagged compliance. Models follow “don’t mention area” more reliably than they keep the home modest.
Third, small models are not automatically worse. Some were steadier on constraints than some flagships. Short copy leans on instruction following and style control more than long-context reasoning.
Fourth, flagships can be overrated. One dumped analysis in another language and never produced usable wishes. Knowledge and marketing do not equal delivery.
Typical failure modes
1. Long analysis, missing drafts
The model profiles the audience and never finishes two texts.
Better instruction: output only two final drafts, no process; label “chat version” and “card version.” In automation, lock a parseable structure.
2. Template blessings
“New chapter,” “happiness always,” “all goes well.” Not wrong—just generic. Anchor on one lived image (windowsill, new paint, a lamp at night) and write around it.
3. Humor that insults
Friends may joke about a small old place; the model should not hammer size, age, or shabbiness. Stress the change after renovation and the feeling of a home they arranged, not the original flaws.
4. Over-rhetoric and rhyme
Parallelism and rhyme read like ads, against “sincere, not oily.” Private wishes work better with short, slightly spoken lines. Score “literary flair” separately from fit.
5. Length counted wrong
Some count only Chinese characters and ignore punctuation, titles, and labels. Unify counting rules in the eval harness.
A more reliable eval process
For ongoing model comparison, do more than one-off taste tests.
1. Fixed test set. Real tasks: short messages, cards, support replies, product copy, social posts. Each with explicit constraints and risk points.
2. Fixed collection environment. Timestamp, model version, parameters, prompts. Hosted models can change silently.
3. Split auto-checks and human review. Length, banned terms, number of outputs: automatic. Tone, audience, emotion: human or a separate judge model.
4. Multiple samples. Record mean, max, and min to separate ceiling from stability.
5. Keep raw outputs. Scores plus original text, error types, and comments. Needed when you later change prompts, models, or rubrics.
Conclusion: don’t pick models by fame and parameter count
Short copy can be a strict composite test. Strong models must write fluently and respect relationships, scene facts, multiple limits, and deliver usable drafts.
Teams should not choose only by size, brand, or a single benchmark. Build a set close to your work and score compliance, stability, language, and delivery.
Cheap models can be enough for light copy; flagships can still fail to ship. Lock tasks, rubric, and reproduction, then decide what belongs in production.
