Evaluating General-Purpose LLMs for Constraint-Driven Creative Writing
High scores on general benchmarks do not necessarily translate into usable writing assistance. This article presents a practical framework for evaluating general-purpose large language models in constraint-driven creative writing, covering source processing, structural compliance, style fidelity, minimal-edit behavior, defensive writing, long-range consistency, hard constraints, soft scoring, regression testing, and workflow-oriented model selection.
1. Background: General Performance Does Not Equal Writing Usability
When selecting a general-purpose large language model, teams often rely on overall benchmark rankings, coding ability, mathematical reasoning, or a few open-ended writing samples. For professional writers, however, these metrics do not answer the central question: can the model process source material, organize structure, and generate language without changing the author’s intent?
In other words, the key question is not whether the model writes beautifully, but whether it follows constraints closely enough to preserve what the author is trying to say.
Two types of goals are often in conflict. The first involves objective-oriented tasks such as question answering, summarization, and code generation. These tasks usually have relatively clear optimal answers. The better a model is at automatic completion, correction, and planning, the higher it tends to score.
The second involves constraint-driven creative writing. These tasks have no single optimal answer, and the process itself is part of the result. The author’s style, reasoning path, argument structure, tone boundaries, and narrative perspective are all constraints that should not be “optimized away.” If a model automatically rewrites the text according to general aesthetic preferences, the result may be smoother and more complete while still diluting the author’s original expression and losing practical value.
As a result, the writing experience of a general-purpose LLM should not be evaluated through isolated single-turn examples. It should be tested within a realistic workflow:
- Is extracted information from source documents traceable?
- Is a literature review faithful to the original material?
- Does the outline follow the specified argument tree?
- Does the style remain stable across a long document?
- Do local edits stay within the requested scope?
- Does the model introduce defensive boilerplate?
This article does not evaluate any specific product. Instead, it abstracts these experiences into a reproducible evaluation and engineering framework.
2. Definition: What Is Constraint-Driven Creative Writing?
A simplified formulation is:
- (S): a collection of source materials
- (C): author-defined constraints
- (A): implicit author intent
- (D): the text generated by the model
The evaluation goal is not simply to maximize the general quality of the output (Q(D)). It is to maximize constraint compliance and fidelity to the author’s intent while preserving factual accuracy and format requirements.
Formally, hard constraints should be satisfied first, followed by optimization of softer quality attributes:
Goal: Generate D = Model(S, C)Hard constraints:- Factual consistency- Traceable citations- No forbidden expressions- Correct structural orderSoft objectives:- Style similarity to the author’s samples- Stable narrative perspective- Minimal changes outside the requested scopeFailure conditions:- Changing the author’s position without permission- Over-balancing viewpoints- Rewriting creative expression as explanatory prose
Wish-Based Automation
“Wish-based automation” is an important concept in this context. It describes a workflow in which the user provides a high-level objective and the model independently plans, decomposes the task, selects an expression strategy, and continuously self-corrects.
This capability is valuable for long-running tasks, tool use, and complex workflows. However, if post-training is overly optimized for this behavior, the model may treat “helping the user find the best solution” as its default strategy.
In creative writing, this can appear as:
- Automatically adding background information and explanations that the author did not request.
- Turning strong opinions into neutral “on the one hand, on the other hand” discussions.
- Adding causal links and concluding statements to create a more complete logic.
- Inserting disclaimers, cautious language, or meta-commentary.
- Rewriting implicit, open-ended, or ironic expressions into direct explanations.
These behaviors are not always negative. They may improve robustness in business summaries, risk analysis, and compliance review. In constraint-driven creative writing, however, they can damage the author’s voice. Together, they can be described as a combination of defensive writing and over-logicalized writing.
3. Evaluation Framework: Three Task Layers and Six Metric Categories
3.1 Task Layers
A real writing workflow should be divided into three layers rather than evaluated only through the final draft.
Layer 1: Source and Literature Processing
The task is not merely to summarize. The model must extract claims, evidence, methods, and limitations while preserving source traceability.
The main evaluation areas are:
- Factual consistency rate
- Citation coverage
- Cross-document conflict detection
A critical check is whether the model “completes” conclusions that do not actually exist in the source material simply to make the prose flow more smoothly.
Layer 2: Structural Organization
Given an argument tree, paragraph order, length proportions, narrative perspective, and prohibited items, the model generates an outline or section draft.
The main evaluation areas are:
- Structural compliance
- Coverage of key nodes
- Handling of conflicting constraints
This layer should allow the author to preserve incomplete logic or open questions. The model should not automatically convert them into textbook-style structures.
Layer 3: Language Generation
The model receives style samples, counterexamples, sentences that must be preserved, and a defined scope for local edits.
The main evaluation areas are:
- Style fidelity
- Minimal editing
- Long-range consistency
- Rate of defensive expressions
3.2 Metric Design
A “hard gates plus soft scores” approach is recommended. Hard constraints should be checked first. If they fail, the quality score is no longer comparable with outputs that satisfy the basic requirements.
Soft scores can then be assessed through expert blind review, rule-based checks, and pairwise comparisons.
| Dimension | Core Question | Test Method | Typical Metrics |
|---|---|---|---|
| Style compliance | Does the output preserve sentence patterns, tone, and narrative perspective? | Provide a reference sample and a counterexample, then conduct pairwise rewriting evaluation | Blind preference rate, style drift |
| Structural compliance | Does the output follow the specified paragraphs and argument order? | Check outlines and drafts against structural labels | Structural compliance rate, node coverage |
| Content fidelity | Were conclusions, numbers, and quotations from the source changed? | Use citation backtracking and fact verification | Factual consistency rate, unsupported claims |
| Long-range consistency | Are terminology, characters, and positions stable throughout the text? | Compare globally after segmented long-form generation | Conflicts per 1,000 words, terminology stability |
| Minimal editing | Were non-target areas changed without permission? | Compare the original and revised versions | Protected-text retention, edit distance |
| Defensive expression rate | Did the model add boilerplate, disclaimers, or excessive balancing? | Combine rule-based counting with human review | Occurrences per 1,000 words, first-pass acceptance rate |
Creative writing cannot be evaluated through automated metrics alone. Whether a text sounds like a particular author often requires blind review by the author or an editor familiar with the relevant corpus. Automated metrics are useful for regression testing and alerts, but they should not replace final judgment.
3.3 Test Set Construction
Test sets should be based as much as possible on real tasks, with identifying information removed. They should include at least the following five categories:
- Multi-document literature review: Extract disagreements, limitations, and unresolved questions from several sources without introducing conclusions that are not supported by the materials.
- Structurally constrained writing: Specify paragraph order, the function of each paragraph, the target audience, and prohibited terms.
- Style continuation: Provide an author sample and require the model to preserve sentence-length distribution, rhetorical density, and narrative perspective without “elevating” the theme.
- Local rewriting: Allow changes only to specified sentences and require all other paragraphs to remain verbatim.
- Long-document consistency: Generate a document of several thousand words or more and check terminology, character relationships, and positions for drift.
Each test case should record three types of constraints:
- What must be satisfied
- What must not appear
- What may be changed
A baseline prompt should also be provided to determine whether the model expands the task boundary on its own under the same source conditions.
For important test cases, adversarial conditions can be added, such as conflicting length requirements, ambiguous factual gaps, or deliberately preserved ambiguity. These conditions help test whether the model correctly prioritizes constraints.
4. An Engineering Workflow for Evaluation
The evaluation process should be versioned, reusable, and as close as possible to the production workflow. A simplified evaluation function is shown below:
def evaluate_case(case, generate):output = generate(source=case.source,prompt=case.prompt,constraints=case.constraints)hard_report = {"facts": check_fact_consistency(output, case.source),"citations": check_citation_trace(output, case.source),"structure": check_outline(output, case.outline),"forbidden": check_forbidden_terms(output, case.forbidden),"protected": check_protected_spans(output, case.protected_spans)}if not all(hard_report.values()):return {"pass": False,"hard_report": hard_report,"output": output}soft_report = {"style_blind_preference": human_pairwise(output, case.reference),"defensive_rate": count_defensive_patterns(output),"edit_distance": diff_ratio(output, case.expected_edit_scope),"long_range_conflicts": check_global_consistency(output)}return {"pass": True,"hard_report": hard_report,"soft_report": soft_report,"output": output}
The key principle is to check hard constraints before applying soft quality scores. Otherwise, a draft with excellent style but altered facts may receive an unjustifiably high score.
For local rewriting tasks, protected-text retention should be a hard metric. For literature reviews, citation traceability should take priority over language fluency.
5. Diagnosing Common Failure Modes
5.1 Automatic Balancing and “Contrarian” Rewriting
Models often add opposing viewpoints, extra qualifications, and conditions, turning a clearly stated position into a neutral review. In commentary, essays, and other writing with a distinct stance, this directly weakens the text.
A diagnostic method is to provide a strongly opinionated sample and explicitly prohibit the introduction of counterarguments. The evaluator can then check whether the model still adds them automatically.
5.2 Over-Logicalization
Models may use explicit transitions, rigid topic structures, and summary paragraphs to wrap around writing that was originally open-ended, fragmented, or implicit.
This may benefit technical documentation but disrupt literary or essayistic prose. Useful diagnostic signals include:
- Transition-word density
- The proportion of the text occupied by summary paragraphs
- The number of inserted explanatory sentences
5.3 Defensive Writing
Typical examples include repeated uncertainty statements, frequent reminders that “the actual situation must be considered,” and assistant-like or compliance-oriented language inserted into creative passages.
These expressions can damage narrative immersion. A rule library can be used to count common phrases, followed by editorial review to determine whether they are contextually necessary.
5.4 Breaks Between Workflow Stages
The model may produce review-style language during source processing and then fail to switch tone during creative generation. Alternatively, it may suddenly change the grammatical person, sentence length, or rhetorical density in the middle of a long document.
A diagnostic approach is to generate the same task in segments and compare stylistic distance and terminology consistency between sections.
5.5 Hallucinations Under Style Pressure
When trying to imitate a particular style, a model may invent quotations, statistics, or personal experiences. These problems must be intercepted through source backtracking and fact verification. Style scores alone cannot detect them reliably.
6. Practical Recommendations
6.1 Use a Prompt Contract
Replace loose instructions with a structured “prompt contract.” Clearly define the goal, required content, prohibited items, permitted editing scope, uncertainty-handling rules, and output format.
For example:
Goal: Generate a first draft of Section 3 from the provided materials.Must preserve:- The author’s original position- The terminology list- First-person narrative perspectiveProhibited:- New facts not present in the source materials- An added conclusion section- Rewriting the position as a neutral reviewAllowed changes:- Expand only paragraphs marked [EXPAND]Handling uncertainty:- Mark uncertain claims with [VERIFY] at the end of the sentence.- Do not complete missing information independently.Output format:- Markdown- H2 headings- No paragraph longer than 300 words
6.2 Separate Fact Processing from Style Generation
First use structured extraction to identify claims, evidence, and citation locations. Then perform style-oriented generation.
During the style stage, the model should rewrite language only within a controlled scope. After generation, run a diff comparison to confirm that factual fields were not changed.
6.3 Build a Regression Set
Save failed cases from real projects as fixed test cases and rerun them automatically whenever the model, prompt, or parameters change.
For creative writing, the regression set does not need to be very large. It does, however, need to cover critical styles, structural requirements, and known failure modes.
Before each release, conduct blind review and record human preferences and failure causes rather than relying only on average scores.
6.4 Use Scenario Weights for Model Selection
Model selection should be based on scenario-specific weights instead of general benchmark rankings.
For constraint-driven writing:
- Style drift should be treated as a critical failure.
- Unauthorized rewriting should be treated as a critical failure.
- Defensive expressions should be treated as a critical failure.
For source and literature processing, factual consistency and citation traceability should take priority. For long-form generation, long-range consistency should receive the highest weight.
Only by embedding evaluation into the actual workflow can subjective writing experience be converted into reproducible technical conclusions.
7. Conclusion
A general-purpose LLM’s writing ability is not the same as controllable writing ability.
Strong performance on general tasks does not prove that a model can respect an author’s style, structure, and cognitive boundaries. Likewise, occasional impressive outputs do not demonstrate that the model is stable and useful across a long workflow.
For professional writing, the more important questions are:
- Can the model complete an unconventional task under explicit constraints?
- Can it maintain a boundary between source processing and creative expression?
- Can it perform local edits without exceeding its authority?
- Can it limit its “help” to what the author actually needs?
Evaluating a production-grade general-purpose LLM should therefore begin with the real writing workflow. The evaluation framework should combine hard constraints, soft quality, failure-mode analysis, and engineering regression tests.
Only when style compliance, structural constraints, defensive-writing detection, and long-range stability are evaluated together can teams determine whether a model is genuinely suitable for constraint-driven creative writing rather than drawing conclusions from a handful of showcase examples.
