Evaluating General-Purpose LLMs for Constraint-Driven Creative Writing

    High scores on general benchmarks do not necessarily translate into usable writing assistance. This article presents a practical framework for evaluating general-purpose large language models in constraint-driven creative writing, covering source processing, structural compliance, style fidelity, minimal-edit behavior, defensive writing, long-range consistency, hard constraints, soft scoring, regression testing, and workflow-oriented model selection.

    1. Background: General Performance Does Not Equal Writing Usability

    When selecting a general-purpose large language model, teams often rely on overall benchmark rankings, coding ability, mathematical reasoning, or a few open-ended writing samples. For professional writers, however, these metrics do not answer the central question: can the model process source material, organize structure, and generate language without changing the author’s intent?

    In other words, the key question is not whether the model writes beautifully, but whether it follows constraints closely enough to preserve what the author is trying to say.

    Two types of goals are often in conflict. The first involves objective-oriented tasks such as question answering, summarization, and code generation. These tasks usually have relatively clear optimal answers. The better a model is at automatic completion, correction, and planning, the higher it tends to score.

    The second involves constraint-driven creative writing. These tasks have no single optimal answer, and the process itself is part of the result. The author’s style, reasoning path, argument structure, tone boundaries, and narrative perspective are all constraints that should not be “optimized away.” If a model automatically rewrites the text according to general aesthetic preferences, the result may be smoother and more complete while still diluting the author’s original expression and losing practical value.

    As a result, the writing experience of a general-purpose LLM should not be evaluated through isolated single-turn examples. It should be tested within a realistic workflow:

    • Is extracted information from source documents traceable?
    • Is a literature review faithful to the original material?
    • Does the outline follow the specified argument tree?
    • Does the style remain stable across a long document?
    • Do local edits stay within the requested scope?
    • Does the model introduce defensive boilerplate?

    This article does not evaluate any specific product. Instead, it abstracts these experiences into a reproducible evaluation and engineering framework.

    2. Definition: What Is Constraint-Driven Creative Writing?

    A simplified formulation is:

    • (S): a collection of source materials
    • (C): author-defined constraints
    • (A): implicit author intent
    • (D): the text generated by the model

    The evaluation goal is not simply to maximize the general quality of the output (Q(D)). It is to maximize constraint compliance and fidelity to the author’s intent while preserving factual accuracy and format requirements.

    Formally, hard constraints should be satisfied first, followed by optimization of softer quality attributes:

    1. Goal: Generate D = Model(S, C)
    2. Hard constraints:
    3. - Factual consistency
    4. - Traceable citations
    5. - No forbidden expressions
    6. - Correct structural order
    7. Soft objectives:
    8. - Style similarity to the author’s samples
    9. - Stable narrative perspective
    10. - Minimal changes outside the requested scope
    11. Failure conditions:
    12. - Changing the author’s position without permission
    13. - Over-balancing viewpoints
    14. - Rewriting creative expression as explanatory prose

    Wish-Based Automation

    “Wish-based automation” is an important concept in this context. It describes a workflow in which the user provides a high-level objective and the model independently plans, decomposes the task, selects an expression strategy, and continuously self-corrects.

    This capability is valuable for long-running tasks, tool use, and complex workflows. However, if post-training is overly optimized for this behavior, the model may treat “helping the user find the best solution” as its default strategy.

    In creative writing, this can appear as:

    1. Automatically adding background information and explanations that the author did not request.
    2. Turning strong opinions into neutral “on the one hand, on the other hand” discussions.
    3. Adding causal links and concluding statements to create a more complete logic.
    4. Inserting disclaimers, cautious language, or meta-commentary.
    5. Rewriting implicit, open-ended, or ironic expressions into direct explanations.

    These behaviors are not always negative. They may improve robustness in business summaries, risk analysis, and compliance review. In constraint-driven creative writing, however, they can damage the author’s voice. Together, they can be described as a combination of defensive writing and over-logicalized writing.

    3. Evaluation Framework: Three Task Layers and Six Metric Categories

    3.1 Task Layers

    A real writing workflow should be divided into three layers rather than evaluated only through the final draft.

    Layer 1: Source and Literature Processing

    The task is not merely to summarize. The model must extract claims, evidence, methods, and limitations while preserving source traceability.

    The main evaluation areas are:

    • Factual consistency rate
    • Citation coverage
    • Cross-document conflict detection

    A critical check is whether the model “completes” conclusions that do not actually exist in the source material simply to make the prose flow more smoothly.

    Layer 2: Structural Organization

    Given an argument tree, paragraph order, length proportions, narrative perspective, and prohibited items, the model generates an outline or section draft.

    The main evaluation areas are:

    • Structural compliance
    • Coverage of key nodes
    • Handling of conflicting constraints

    This layer should allow the author to preserve incomplete logic or open questions. The model should not automatically convert them into textbook-style structures.

    Layer 3: Language Generation

    The model receives style samples, counterexamples, sentences that must be preserved, and a defined scope for local edits.

    The main evaluation areas are:

    • Style fidelity
    • Minimal editing
    • Long-range consistency
    • Rate of defensive expressions

    3.2 Metric Design

    A “hard gates plus soft scores” approach is recommended. Hard constraints should be checked first. If they fail, the quality score is no longer comparable with outputs that satisfy the basic requirements.

    Soft scores can then be assessed through expert blind review, rule-based checks, and pairwise comparisons.

    Dimension Core Question Test Method Typical Metrics
    Style compliance Does the output preserve sentence patterns, tone, and narrative perspective? Provide a reference sample and a counterexample, then conduct pairwise rewriting evaluation Blind preference rate, style drift
    Structural compliance Does the output follow the specified paragraphs and argument order? Check outlines and drafts against structural labels Structural compliance rate, node coverage
    Content fidelity Were conclusions, numbers, and quotations from the source changed? Use citation backtracking and fact verification Factual consistency rate, unsupported claims
    Long-range consistency Are terminology, characters, and positions stable throughout the text? Compare globally after segmented long-form generation Conflicts per 1,000 words, terminology stability
    Minimal editing Were non-target areas changed without permission? Compare the original and revised versions Protected-text retention, edit distance
    Defensive expression rate Did the model add boilerplate, disclaimers, or excessive balancing? Combine rule-based counting with human review Occurrences per 1,000 words, first-pass acceptance rate

    Creative writing cannot be evaluated through automated metrics alone. Whether a text sounds like a particular author often requires blind review by the author or an editor familiar with the relevant corpus. Automated metrics are useful for regression testing and alerts, but they should not replace final judgment.

    3.3 Test Set Construction

    Test sets should be based as much as possible on real tasks, with identifying information removed. They should include at least the following five categories:

    1. Multi-document literature review: Extract disagreements, limitations, and unresolved questions from several sources without introducing conclusions that are not supported by the materials.
    2. Structurally constrained writing: Specify paragraph order, the function of each paragraph, the target audience, and prohibited terms.
    3. Style continuation: Provide an author sample and require the model to preserve sentence-length distribution, rhetorical density, and narrative perspective without “elevating” the theme.
    4. Local rewriting: Allow changes only to specified sentences and require all other paragraphs to remain verbatim.
    5. Long-document consistency: Generate a document of several thousand words or more and check terminology, character relationships, and positions for drift.

    Each test case should record three types of constraints:

    • What must be satisfied
    • What must not appear
    • What may be changed

    A baseline prompt should also be provided to determine whether the model expands the task boundary on its own under the same source conditions.

    For important test cases, adversarial conditions can be added, such as conflicting length requirements, ambiguous factual gaps, or deliberately preserved ambiguity. These conditions help test whether the model correctly prioritizes constraints.

    4. An Engineering Workflow for Evaluation

    The evaluation process should be versioned, reusable, and as close as possible to the production workflow. A simplified evaluation function is shown below:

    1. def evaluate_case(case, generate):
    2. output = generate(
    3. source=case.source,
    4. prompt=case.prompt,
    5. constraints=case.constraints
    6. )
    7. hard_report = {
    8. "facts": check_fact_consistency(output, case.source),
    9. "citations": check_citation_trace(output, case.source),
    10. "structure": check_outline(output, case.outline),
    11. "forbidden": check_forbidden_terms(output, case.forbidden),
    12. "protected": check_protected_spans(output, case.protected_spans)
    13. }
    14. if not all(hard_report.values()):
    15. return {
    16. "pass": False,
    17. "hard_report": hard_report,
    18. "output": output
    19. }
    20. soft_report = {
    21. "style_blind_preference": human_pairwise(output, case.reference),
    22. "defensive_rate": count_defensive_patterns(output),
    23. "edit_distance": diff_ratio(output, case.expected_edit_scope),
    24. "long_range_conflicts": check_global_consistency(output)
    25. }
    26. return {
    27. "pass": True,
    28. "hard_report": hard_report,
    29. "soft_report": soft_report,
    30. "output": output
    31. }

    The key principle is to check hard constraints before applying soft quality scores. Otherwise, a draft with excellent style but altered facts may receive an unjustifiably high score.

    For local rewriting tasks, protected-text retention should be a hard metric. For literature reviews, citation traceability should take priority over language fluency.

    5. Diagnosing Common Failure Modes

    5.1 Automatic Balancing and “Contrarian” Rewriting

    Models often add opposing viewpoints, extra qualifications, and conditions, turning a clearly stated position into a neutral review. In commentary, essays, and other writing with a distinct stance, this directly weakens the text.

    A diagnostic method is to provide a strongly opinionated sample and explicitly prohibit the introduction of counterarguments. The evaluator can then check whether the model still adds them automatically.

    5.2 Over-Logicalization

    Models may use explicit transitions, rigid topic structures, and summary paragraphs to wrap around writing that was originally open-ended, fragmented, or implicit.

    This may benefit technical documentation but disrupt literary or essayistic prose. Useful diagnostic signals include:

    • Transition-word density
    • The proportion of the text occupied by summary paragraphs
    • The number of inserted explanatory sentences

    5.3 Defensive Writing

    Typical examples include repeated uncertainty statements, frequent reminders that “the actual situation must be considered,” and assistant-like or compliance-oriented language inserted into creative passages.

    These expressions can damage narrative immersion. A rule library can be used to count common phrases, followed by editorial review to determine whether they are contextually necessary.

    5.4 Breaks Between Workflow Stages

    The model may produce review-style language during source processing and then fail to switch tone during creative generation. Alternatively, it may suddenly change the grammatical person, sentence length, or rhetorical density in the middle of a long document.

    A diagnostic approach is to generate the same task in segments and compare stylistic distance and terminology consistency between sections.

    5.5 Hallucinations Under Style Pressure

    When trying to imitate a particular style, a model may invent quotations, statistics, or personal experiences. These problems must be intercepted through source backtracking and fact verification. Style scores alone cannot detect them reliably.

    6. Practical Recommendations

    6.1 Use a Prompt Contract

    Replace loose instructions with a structured “prompt contract.” Clearly define the goal, required content, prohibited items, permitted editing scope, uncertainty-handling rules, and output format.

    For example:

    1. Goal: Generate a first draft of Section 3 from the provided materials.
    2. Must preserve:
    3. - The author’s original position
    4. - The terminology list
    5. - First-person narrative perspective
    6. Prohibited:
    7. - New facts not present in the source materials
    8. - An added conclusion section
    9. - Rewriting the position as a neutral review
    10. Allowed changes:
    11. - Expand only paragraphs marked [EXPAND]
    12. Handling uncertainty:
    13. - Mark uncertain claims with [VERIFY] at the end of the sentence.
    14. - Do not complete missing information independently.
    15. Output format:
    16. - Markdown
    17. - H2 headings
    18. - No paragraph longer than 300 words

    6.2 Separate Fact Processing from Style Generation

    First use structured extraction to identify claims, evidence, and citation locations. Then perform style-oriented generation.

    During the style stage, the model should rewrite language only within a controlled scope. After generation, run a diff comparison to confirm that factual fields were not changed.

    6.3 Build a Regression Set

    Save failed cases from real projects as fixed test cases and rerun them automatically whenever the model, prompt, or parameters change.

    For creative writing, the regression set does not need to be very large. It does, however, need to cover critical styles, structural requirements, and known failure modes.

    Before each release, conduct blind review and record human preferences and failure causes rather than relying only on average scores.

    6.4 Use Scenario Weights for Model Selection

    Model selection should be based on scenario-specific weights instead of general benchmark rankings.

    For constraint-driven writing:

    • Style drift should be treated as a critical failure.
    • Unauthorized rewriting should be treated as a critical failure.
    • Defensive expressions should be treated as a critical failure.

    For source and literature processing, factual consistency and citation traceability should take priority. For long-form generation, long-range consistency should receive the highest weight.

    Only by embedding evaluation into the actual workflow can subjective writing experience be converted into reproducible technical conclusions.

    7. Conclusion

    A general-purpose LLM’s writing ability is not the same as controllable writing ability.

    Strong performance on general tasks does not prove that a model can respect an author’s style, structure, and cognitive boundaries. Likewise, occasional impressive outputs do not demonstrate that the model is stable and useful across a long workflow.

    For professional writing, the more important questions are:

    • Can the model complete an unconventional task under explicit constraints?
    • Can it maintain a boundary between source processing and creative expression?
    • Can it perform local edits without exceeding its authority?
    • Can it limit its “help” to what the author actually needs?

    Evaluating a production-grade general-purpose LLM should therefore begin with the real writing workflow. The evaluation framework should combine hard constraints, soft quality, failure-mode analysis, and engineering regression tests.

    Only when style compliance, structural constraints, defensive-writing detection, and long-range stability are evaluated together can teams determine whether a model is genuinely suitable for constraint-driven creative writing rather than drawing conclusions from a handful of showcase examples.