Choosing an LLM for Writing Tasks: From Aggregate Scores to Task-Level Evaluation

    Aggregate Elo-style leaderboards barely separate the models at the top, and writing is not a single capability. This article explains why headline scores mislead, breaks writing down into outlining, translation, long-form drafting and editing, describes the tradeoff between analytical and expressive models, covers quality decay in long outputs, and shows how tiered model routing, multi-model pipelines and task-level blind evaluation produce better results at lower cost.

    1. The statistical traps in aggregate scores

    When teams pick a model for writing work, the first move is often to open a public head-to-head leaderboard, sort by aggregate Elo, and take the top entry. The problem is that the gaps between the top few models are routinely over-interpreted.

    Take a typical head-to-head evaluation. When the aggregate Elo gap among the leading models is only about twenty points, that difference is statistically close to a coin flip. Three factors explain why:

    • Overlapping confidence intervals. Elo is a relative rating derived from win rates, so it carries sampling error. With limited sample sizes, a gap of roughly 20 points usually falls inside the confidence interval, and cannot support a claim that model A is definitively better than model B.
    • A deliberately balanced task mix. General-purpose leaderboards blend many heterogeneous task types — code, math, Q&A, writing, agent tasks — to stay broadly relevant. Writing is only a small slice of the weight, and once it is diluted this way, differences in that single capability rarely surface in the total score.
    • Judge preference noise. Human judgments of “style” are highly subjective, and different judges, prompts and rating scales all introduce variance.

    So “similar aggregate score means similar capability” is a dangerous misreading. What actually determines production quality is task-level performance.

    2. Decompose writing: it is not one capability, but a set of capabilities

    Only when writing is broken down into concrete actions do the real differences hidden by aggregate scores become visible. A typical content production chain has at least four stages, and each one tests a different dimension:

    1. Outlining and logical decomposition — tests structured reasoning, causal mapping, premise identification and anticipating counterarguments. Fundamentally an analytical capability.
    2. Translation and terminology consistency — tests semantic fidelity, contextual alignment, and stable mapping of domain terminology.
    3. Long-form drafting — tests narrative pacing, emotional description, lexical variety, and believable dialogue. Fundamentally an expressive capability.
    4. Revision and polishing — tests sensitivity to tone, structure and redundancy, plus the ability to rewrite while preserving the original intent.

    Evaluate at this level and the gaps between models widen sharply: two models that look evenly matched on an aggregate board can perform in completely opposite ways at the outlining stage versus the drafting stage.

    3. The capability spectrum: analysis versus expression

    A pattern that practice keeps confirming: models that are strong at logical reasoning tend to excel at structured decomposition but to be comparatively weak at creative expression, and vice versa. This is not coincidence but a tradeoff that follows from training objectives and capability structure.

    • Analysis-leaning models tend to enumerate causal relationships, premises and potential counterarguments item by item when breaking down a complex topic. Their output is an “argument skeleton” rather than a “narrative frame.” This correlates positively with their code and reasoning ability — a model that handles complex code logic decomposes arguments with natural systematicity and completeness. The cost is that its prose often reads overly formal and stiff, and official documentation commonly concedes that such models are “more limited in expression” on casual or highly creative tasks.
    • Expression-leaning models are better at delivering narrative that reads comfortably and can produce near-final copy straight out of the box. But when rigorous argument is required, they may sacrifice logical precision to keep the prose flowing.

    The key selection conclusion follows: no single model is optimal for every writing subtask. The sensible approach is to match models to stages along the capability spectrum.

    4. The quality decay curve in long-form generation

    The long-form drafting stage has another widely underestimated problem: quality degrades with length.

    Many models perform acceptably within a few thousand characters, but once output exceeds their effective generation range, the second half tends to degrade in three ways:

    • Drift: the argument or narrative wanders off the main thread, and goals set in the opening are gradually forgotten.
    • Repetition: sentence patterns, points and descriptions recur, with information gain approaching zero.
    • Structural collapse: transitions between paragraphs break down and logical coherence falls.

    In other words, “can it write long-form text” and “can it write several thousand characters well“ are two different questions. Evaluating long-form ability requires measuring quality stability across length, not just reading a few hundred characters of sample output.

    5. The cost–performance trade-off: tiered routing, not full-price routing

    Beyond quality, cost is the hard constraint that decides whether something can ship at scale. Pricing across writing-capable models varies enormously — per-million-token prices across tiers can differ by tens of times. Some models also offer accelerated modes at a higher unit price.

    Run every stage on the expensive model and costs spiral quickly. Run everything on the cheap model and draft quality falls short of the bar. The sensible strategy is tiered routing: spend money only on the stages that actually shape the final output.

    The table below illustrates one such cost–quality allocation:

    Stage Capability required Routing strategy Share of calls
    Outline / decomposition Structured reasoning Low-cost model, multiple rounds High
    Long-form drafting Creative expression High-cost model for quality, limited rounds Low
    Translation Semantic fidelity Mid-tier, batch concurrency Medium
    Revision / polishing Tone sensitivity High-cost, final draft only Low

    The core idea: the outlining stage is cheap per call, so you can afford to run several rounds and compare options; drafting and polishing are expensive, so they are reserved for the critical path.

    6. Building a multi-model writing pipeline

    Given the above, the recommended engineering approach is to break writing into an orchestratable pipeline where models with different capabilities each do what they do best. The pseudocode below shows a workable orchestration skeleton:

    1. def writing_pipeline(topic, style_guide, target_length):
    2. # Stage 1: use an analysis-leaning model to generate several outlines, then pick the best
    3. outlines = [
    4. analytical_model.generate_outline(
    5. topic=topic,
    6. requirements=["causal relations", "premises", "potential counterarguments"],
    7. temperature=0.7,
    8. )
    9. for _ in range(3) # Run several rounds in the low-cost stage
    10. ]
    11. best_outline = select_by_score(outlines, rubric=structure_rubric)
    12. # Stage 2: use an expression-leaning model to draft section by section, controlling decay
    13. sections = []
    14. for section in best_outline.sections:
    15. draft = expressive_model.write_section(
    16. outline=section,
    17. style_guide=style_guide,
    18. context=accumulated_context, # Carry prior text to suppress drift
    19. )
    20. sections.append(draft)
    21. # Stage 3: polish the final draft once to keep the tone consistent
    22. final = expressive_model.polish(
    23. text=merge(sections),
    24. target_length=target_length,
    25. requirements=["remove repetition", "unify tone", "fix transitions"],
    26. )
    27. return final

    This skeleton reflects three points:

    • Separation of responsibilities: the analysis-leaning model owns the skeleton, the expression-leaning model owns the flesh, and polishing unifies the style.
    • Section-wise length control: long text is generated section by section from the outline with prior context attached, which mitigates quality decay.
    • Cost tiering: cheap stages allow multiple trial-and-error rounds, while expensive stages converge on the critical path.

    For enterprise teams, this kind of multi-model orchestration is typically handled through a unified LLM service platform. Managed LLM services on cloud platforms such as Baidu AI Cloud illustrate the pattern: a single interface layer lets a team switch flexibly between models in different capability tiers and pay by usage, without maintaining separate integration code for each model — which is especially convenient for building the pipeline described above.

    7. A practical task-level evaluation method

    To keep selection from being guesswork, you need a reusable evaluation process:

    1. Define a task list: enumerate real writing scenarios (outlining, translation, long-form drafting, revision) instead of an undifferentiated “writing.”
    2. Build a per-dimension scoring rubric: judge outlines on logical completeness; judge drafts on style, pacing, emotional authenticity and stability over length; judge revisions on intent preservation and improvement in feel.
    3. Run blind, multi-judge evaluation: hide the model’s identity and have several judges score against the rubric, which reduces brand bias and single-preference noise.
    4. Add supporting automatic metrics: quantifiable measures such as repetition rate, sentence-length variance and logical-connective density help flag problems like repetition and monotonous pacing.
    5. Evaluate by length slices: assess quality separately at 500-, 2000- and 5000-character tiers to observe the decay curve.
    6. Include cost: among options that clear the quality bar, prefer the combination with the better price-to-performance ratio.

    A simplified weighted aggregation can be written as:

    1. def score_model(task_scores, quality_weight=0.7, cost_weight=0.3):
    2. quality = weighted_avg(task_scores) # Per-dimension quality score
    3. cost = normalize(1 / avg_price) # Lower price yields a higher score
    4. return quality_weight * quality + cost_weight * cost

    Conclusion

    When choosing a writing model, the thing to avoid most is picking the highest aggregate score on a leaderboard. Aggregate ratings can barely distinguish between models in the top band, while what actually determines output quality is task-level performance. Break writing into four actions — outline, draft, translate, revise; understand the tradeoff between analysis-leaning and expression-leaning models; watch for quality decay in long output; then use tiered routing and a multi-model collaboration pipeline to balance cost and quality. This methodology is worth far more than any rank on a leaderboard. The goal of selection is not “which model is strongest,” but “which model fits which stage.”