Evaluating LLMs for Creative Writing and Strong Instruction-Following: A Practical Guide

    This article provides a practical guide to evaluating large language models in creative writing and strong instruction-following scenarios. It explains why evaluation is shifting from general intelligence to controllability, identifies three common failure modes, defines core concepts, and proposes a multidimensional evaluation framework. It also covers test-set design, prompt design, automated scoring, human review, defensive-writing mitigation, overfitting detection, long-context consistency, engineering evaluation pipelines, and common pitfalls. The goal is to help readers build reproducible, regression-friendly evaluation systems for real-world generative applications.

    From “Usable” to “Controllable”: The Focus of LLM Evaluation Is Changing

    In early large language model applications, evaluation often focused on general question answering, knowledge coverage, commonsense reasoning, and basic code generation. If a model could produce answers that were structurally complete, semantically fluent, and factually correct, it was often considered sufficiently usable. However, as generative applications moved into real business scenarios, evaluation standards began to change.

    For developers, a model must not only “answer questions” but also remain stable in complex tasks. For product teams, a model must not only “produce output” but also generate acceptable content under multiple constraints, such as format, style, tone, length, role setting, narrative perspective, and business rules. In scenarios such as creative writing, long-document continuation, intelligent programming, automated report generation, character dialogue, and marketing copy production, a single high-quality response is not enough. What matters more is whether the model has stable instruction-following ability, long-range consistency, and reusable output quality.

    As a result, evaluation is shifting from asking “Is the model smart?” to asking “Is the model controllable?” This is why strong instruction-following tasks, creative writing tasks, and long-range cognitive tasks need to be evaluated separately.

    Three Common Problems: Defensive Writing, Task Overfitting, and Broken Experience

    In creative writing and strong instruction-following scenarios, the main problem is usually not that the model completely fails to complete the task. Rather, the output may feel unnatural, unstable, or misaligned with the task goal. Based on common practice, these issues can be summarized as three typical problem types.

    The first is defensive writing. Defensive writing refers to a tendency in which the model favors conservative expression, adds explanations, repeats premises, emphasizes safety boundaries, inserts summary statements, or weakens expressive intensity. This behavior may be valuable in tasks such as information organization, risk warnings, and process descriptions. However, in creative writing, storytelling, character monologue, and advertising copy, it can significantly weaken the expressiveness of the text. The model may write something that is “correct” but not vivid, logically clear but lacking tension, or structurally complete but rigid.

    The second is excessive overfitting to common task formats. Many models perform well on tasks such as general summarization, classification, extraction, simple question answering, structured output, and routine code snippet generation because these tasks have relatively clear input-output patterns. However, when the task becomes a non-standard cognitive task—such as multi-thread narrative, metaphorical expression, counter-intuitive settings, complex character motivation, open-ended creative continuation, or strong style imitation—the model may easily fall back on existing patterns and produce templated expressions. On the surface, it may still follow the task requirements, but in practice it may have already deviated from the essence of the task.

    The third is experience disruption caused by excessive logical stacking. Some models, in order to demonstrate “logical rigor,” frequently use bullet points, progressive structures, summaries, and causal connectives, even in creative text. They may also insert too much explanatory content. As a result, the information density of the text increases, but aesthetic continuity and emotional flow are damaged. What users see may not be a naturally generated piece of writing, but a set of explanatory paragraphs forced into a structured form.

    Defining the Core Concepts: Strong Instruction-Following, Creative Writing, and Long-Range Tasks

    To scientifically evaluate model performance in these scenarios, it is first necessary to clarify several core concepts.

    Strong instruction-following is not simply about being “obedient.” It includes at least four layers:

    1. Accurately understanding the task goal.
    2. Identifying and executing explicit constraints, such as format, length, banned words, role setting, and output structure.
    3. Making reasonable tradeoffs when multiple constraints conflict.
    4. Maintaining stable output without sacrificing content quality.

    Creative writing places greater emphasis on expression quality. It requires the model not only to complete the task requirements but also to maintain naturalness and coherence in language rhythm, emotional progression, scene details, character tone, and narrative perspective. Creative writing is not simply about piling up ornate words. Instead, it is about generating readable and extensible content under constraints.

    Long-range tasks focus on the model’s ability to maintain consistency across multi-turn, multi-segment, and multi-state inputs. For example, long-form writing requires consistent character settings, complex task execution requires consistency of intermediate states, and multi-turn dialogue requires consistent role memory. If a model performs well only on short texts but begins to drift in settings once the context becomes longer, it still cannot support high-quality applications.

    Evaluation Framework: From a Single Score to Multidimensional Assessment

    For these scenarios, it is recommended to build a multidimensional evaluation framework rather than relying on a single aggregate score. A relatively complete evaluation system can include the following dimensions.

    1. Instruction-Following

    This dimension focuses on whether the model completes all hard constraints, including format requirements, length ranges, role settings, banned words, output structure, and tone requirements. For complex prompts, evaluators should check whether the model omits key constraints or fails to produce a reasonable result when multiple constraints conflict.

    2. Style Consistency

    This dimension focuses on whether the model maintains a unified tone and narrative perspective. For example, if the model is required to use restrained first-person narration, evaluators should check whether it consistently maintains the “I” perspective. If the model is required to write in a cold or austere style, evaluators should check whether it suddenly shifts into an explanatory tone.

    3. Creative Quality

    This dimension focuses on whether the content has novelty, visual imagery, emotional tension, and extensibility. This should not rely only on subjective human impressions. Methods such as multi-person scoring, blind comparison, and multi-sample stability evaluation can also be used.

    4. Long-Range Consistency

    This dimension focuses on whether the model maintains consistency of characters, locations, timelines, task states, and rule settings over longer contexts. For story generation, evaluators should check whether character relationships remain stable. For complex task execution, evaluators should check whether intermediate results remain traceable.

    5. Anti-Overfitting Ability

    This dimension focuses on the model’s performance in unconventional tasks, compound constraints, and cross-domain transfer. Evaluators can add constraints, change task contexts, or introduce uncommon settings to observe whether the model degrades into templated output.

    6. Balance Between Safety Compliance and Usability

    This dimension focuses on whether the model can avoid generating violating content while also avoiding excessive refusal. Under compliance requirements, the model should try to complete the task as much as possible rather than reducing user experience through excessive conservatism.

    Test Set Design: From “Asking Anything” to Structured Samples

    High-quality evaluation depends on high-quality test sets. For creative writing and strong instruction-following tasks, it is recommended to divide the test set into multiple layers.

    A baseline set can cover common tasks, such as short story openings, character dialogue, product copy, scene descriptions, and dialogue continuation. These tasks can reflect the model’s basic capabilities.

    A perturbation set can add constraints to baseline tasks, such as limiting length, limiting rhetoric, specifying narrative perspective, banning certain words, or requiring an open ending. By observing model performance after constraints are added, evaluators can assess instruction-following stability.

    An adversarial set can be used to design conflicting constraints. For example, it may require “do not summarize” while the task itself has an analytical tendency, require “keep the tone light and humorous” while the subject is serious, or require “use first person” while the preceding text provides a third-person setting. The adversarial set can reveal whether the model has the ability to judge constraint priority.

    A long-range set can be used for multi-turn or multi-segment tasks, such as continuously generating chapters, maintaining character states, preserving foreshadowing relationships, and keeping terminology consistent across a technical plan. The long-range set is especially suitable for evaluating context retention.

    A business set can be designed to be closer to real applications, such as marketing copy, training materials, product descriptions, script fragments, and user message rewriting. The value of the business set is that it makes evaluation results closer to real usage experience.

    A structured test sample can include the following fields:

    1. {
    2. "case_id": "creative_001",
    3. "task_type": "creative_writing",
    4. "prompt": "Write a first-person monologue set on a rainy night street, expressing restrained and slowly changing emotions.",
    5. "constraints": {
    6. "max_length": 300,
    7. "min_length": 160,
    8. "perspective": "first_person",
    9. "tone": "restrained",
    10. "must_include": ["umbrella", "neon"],
    11. "ban_words": ["in summary", "in conclusion", "life"]
    12. },
    13. "evaluation_points": [
    14. "Maintains first-person perspective",
    15. "Avoids summary-like expressions",
    16. "Includes scene details",
    17. "Shows emotional progression"
    18. ]
    19. }

    The benefit of this structured design is that automated checks, human reviews, and regression tests can all be carried out around a unified standard.

    Prompt Design: Making the Model Understand What It Must Do and What It Must Not Do

    In strong instruction-following tasks, prompt design needs to clarify the task goal, constraint boundaries, priorities, and quality standards. An effective prompt template usually includes the following parts.

    First, role and goal. The model’s role in the current task and the core goal it needs to achieve should be clearly stated. The role description should not be too long, otherwise the model may fall into explanatory expression.

    Second, hard constraints. These include output format, length, perspective, tone, required elements, and forbidden elements. Constraints should be specific and checkable, avoiding vague expressions.

    Third, priority rules. When multiple requirements conflict, the model needs to know which one takes precedence. For example, one can specify: safety compliance takes priority over format requirements; format requirements take priority over creative expansion; core task goals take priority over secondary stylistic decoration.

    Fourth, output restrictions. The model should be explicitly required not to explain the task, not to repeat the prompt, not to output its reasoning process, and not to add extra explanations. This helps reduce defensive writing.

    An example prompt is as follows:

    1. System prompt:
    2. You are a writing assistant that strictly follows task constraints. Output only the final content. Do not restate the requirements, do not explain the reasons, and do not add a summary. If constraints conflict, handle them in the following order: safety and compliance > hard formatting > content quality > stylistic expansion.
    3. User prompt:
    4. Write a first-person monologue on the theme “returning home on a rainy night.” Requirements: length between 180 and 260 characters; include “umbrella” and “neon”; use a restrained tone; do not use words such as “loneliness,” “life,” or “summary”; do not output analysis or explanation; maintain first-person perspective; output plain text.

    The advantage of this type of prompt is that the constraints are clear and highly verifiable, making it suitable for strong instruction-following evaluation.

    Automated Scoring: Using Computable Metrics to Reduce Subjective Bias

    For quantifiable aspects, automated scoring should be prioritized. Automated scoring does not aim to completely replace human review, but to quickly identify obvious problems.

    Common automated metrics include format pass rate, length compliance rate, banned-word hit rate, required-word coverage rate, repetition rate, sentence-length distribution, connective density, role-pronoun consistency, and paragraph structure compliance rate.

    For example, the following pseudocode can be used to check basic constraints:

    1. FUNCTION check_text(text, case)
    2. errors = []
    3. IF length(text) > case.max_length THEN
    4. errors.add("length_exceeded")
    5. END IF
    6. IF length(text) < case.min_length THEN
    7. errors.add("length_too_short")
    8. END IF
    9. FOR EACH word IN case.ban_words
    10. IF contains(text, word) THEN
    11. errors.add("banned_word:" + word)
    12. END IF
    13. END FOR
    14. FOR EACH word IN case.must_include
    15. IF NOT contains(text, word) THEN
    16. errors.add("missing_word:" + word)
    17. END IF
    18. END FOR
    19. IF case.perspective == "first_person" AND NOT starts_with(text, "I") THEN
    20. errors.add("first_person_missing")
    21. END IF
    22. IF matches_any(text, ["in summary", "in conclusion", "therefore", "we can see"]) THEN
    23. errors.add("summary_like_phrase")
    24. END IF
    25. RETURN {
    26. "passed": errors is empty,
    27. "errors": errors
    28. }
    29. END FUNCTION

    Although these checks are simple, they can effectively identify common problems such as uncontrolled length, banned words, incorrect perspective, and overly strong summary-like language.

    Human Review: The Final Gate for Creative Quality

    Automated metrics can address whether formatting requirements are met, but they can hardly fully determine whether the writing is good. Therefore, human review remains irreplaceable. It is recommended to combine multi-person independent scoring with blind comparison.

    The scoring sheet can be designed around the following dimensions:

    • Instruction-following: whether all user requirements are fully met and whether key constraints are omitted.
    • Naturalness of expression: whether the text reads like natural writing and whether it contains mechanical stacking.
    • Creative quality: whether there are novel expressions, details, and emotional tension.
    • Style stability: whether the specified tone and perspective are consistently maintained.
    • Long-range consistency: whether characters, scenes, and timelines remain consistent.
    • Editability: whether the content is convenient for later revision, expansion, or structuring.

    Each dimension can use a 1-to-5 scoring scale with clear anchors. For example:

    • 5: Fully satisfies the requirements and is excellent in expression.
    • 4: Basically satisfies the requirements but has minor issues.
    • 3: Partially satisfies the requirements but is noticeably unnatural.
    • 2: Violates key constraints.
    • 1: Task failure or content is unusable.

    During human review, reviewers should not only look at the final text. They should also record whether the model over-explains, whether it repeats the prompt, whether it adds a safety statement at the beginning, and whether it forces a summary at the end. These details often reflect more accurately than a single total score whether a model is suitable for strong instruction-following creative tasks.

    Governing Defensive Writing: Reducing “Correct but Unengaging”

    Defensive writing can be addressed at three levels: prompt design, evaluation metrics, and training preferences.

    At the prompt level, unnecessary explanations, repetitions, and disclaimers should be explicitly prohibited. Requirements such as “output the main text directly,” “do not analyze the task,” “do not add a summary,” and “do not explain the creative approach” can be added.

    At the evaluation level, an “explanatory sentence ratio” metric can be introduced. For example, one can measure the frequency of expressions such as “first,” “second,” “in summary,” “therefore,” “we can see,” and “it should be noted.” If such expressions appear too frequently in creative writing, it usually indicates that the model is shifting toward defensive expression.

    At the training preference level, model expression can be optimized through preference data. Preference data should include high-quality creative texts and should clearly label negative examples such as over-explanation, repeated requirements, and templated summaries. This makes it easier for the model to learn the behavior pattern of “maintaining natural expression while satisfying constraints.”

    It is important to emphasize that defensive writing is not always negative. In scenarios such as medical advice, legal notices, financial disclaimers, and safety boundaries, conservative expression may be necessary. The issue is that in creative writing tasks, the model needs the ability to switch expression modes rather than using the same conservative style for all tasks.

    Identifying and Improving Overfitting in Creative Writing

    Overfitting in creative writing usually manifests as follows: when facing a new task, the model still uses highly similar sentence structures; when facing an open-ended setting, it tends to provide common tropes; when facing strong style requirements, it can only make surface-level imitation and cannot go deeper into character logic.

    This problem can be improved from three aspects.

    First, expand task diversity. The test set should not contain only common topics. It should also include non-standard cognitive tasks, such as absurd settings, nonlinear narratives, uncommon perspectives, low-resource styles, and complex emotional transitions.

    Second, add contrast samples. The same task can be designed in multiple versions: a weak-constraint version, a strong-constraint version, a style-enhanced version, and a logic-enhanced version. By comparing differences in model outputs, evaluators can determine whether the model truly understands the task or is merely applying a template.

    Third, introduce multi-turn perturbations. After generation, the model can be asked to revise the content, for example: “shorten it by half,” “change the perspective,” “add conflict,” or “delete explanatory sentences.” If the model can only perform simple edits and cannot maintain the core settings, this indicates insufficient long-range control ability.

    Evaluating Long-Range Tasks: State Preservation and Structural Consistency

    The key to long-range tasks is not the quality of a single paragraph, but consistency across paragraphs. For story generation, character states should be maintained. For technical document generation, terminology and chapter structure should be maintained. For complex task planning, goals, constraints, and intermediate results should be maintained.

    A state object can be introduced to break down a long-range task into checkable fields. For example:

    1. {
    2. "scene": "a rainy night street",
    3. "characters": {
    4. "protagonist": "a tired traveler returning home",
    5. "supporting_character": "a stranger holding an umbrella"
    6. },
    7. "objects": {
    8. "umbrella": "an old black umbrella",
    9. "neon": "lights from a distant shop"
    10. },
    11. "timeline": [
    12. "the protagonist exits the station",
    13. "sees the neon lights",
    14. "encounters the stranger",
    15. "decides to keep going"
    16. ],
    17. "constraints": {
    18. "tone": "restrained",
    19. "perspective": "first-person",
    20. "ban_words": ["life", "loneliness", "summary"]
    21. }
    22. }

    After each generation round, evaluators can check whether the state object has been violated. For example, whether a character suddenly changes how they are addressed, whether the scene is lost, whether banned words appear, whether the timeline jumps, or whether the perspective switches. This state-based evaluation makes it more accurate to determine whether a model has long-range task capabilities.

    Engineering Evaluation Pipelines: From One-Time Tests to Continuous Regression

    For developer teams, evaluation should not be just a one-time script or document. It should be integrated into a continuous iteration process. It is recommended to build a standardized evaluation pipeline.

    First, sample management. Test samples should be stored centrally, with versions, task types, constraints, expected outputs, and scoring criteria recorded. This avoids scattered samples and makes later regression easier.

    Second, batch execution. Model outputs should be generated in batches, with comparisons across different sampling strategies and prompt versions. Batch execution can expose occasional failures and stability differences.

    Third, automated scoring. Hard metrics such as format, length, banned words, required words, and perspective should be checked first to quickly filter out unqualified samples.

    Fourth, human spot checks. Samples that pass automated checks should be reviewed by humans, with a focus on creative quality, naturalness, style consistency, and long-range stability.

    Fifth, metric monitoring. Metrics such as pass rate, average score, failure type distribution, repetition rate, and explanatory sentence ratio should be recorded. Alerts should be triggered when metrics drop abnormally.

    Sixth, version regression. Whenever prompts, model strategies, or post-training data change, the evaluation set should be rerun to confirm that no regression has been introduced.

    At the platform implementation level, teams can use object storage to save evaluation samples, use message queues to decouple task submission from scoring execution, use logging services to record call chains and failure reasons, and use monitoring and alerting to track fluctuations in evaluation metrics. These general cloud capabilities can help teams transform model evaluation from a manual operation into a sustainable engineering process.

    Common Pitfalls: Do Not Judge Model Quality by a Single Aggregate Score

    In practical evaluation, several pitfalls are common.

    The first is looking only at the average score. The average score may hide critical problems. For example, a model may perform very well on simple tasks but have a high failure rate under complex constraints. If only the average score is considered, the risk will be underestimated.

    The second is looking only at single-round results. LLM outputs are stochastic, and one high-quality result does not indicate stability. Consistency should be evaluated through multiple samples.

    The third is focusing only on safety while ignoring experience. An overly conservative model may not make obvious mistakes, but it can greatly reduce task value. Evaluation needs to observe both compliance and usability.

    The fourth is testing only common tasks. The value of creative writing and strong instruction-following scenarios is often reflected in non-standard tasks. If only common problems are tested, it is easy to misjudge model capabilities.

    The fifth is mistaking fluency for high quality. Fluent language does not necessarily mean excellent content. A model may produce grammatically smooth sentences while having hollow characters, broken emotion, and rigid logic. Evaluation should focus on the match between expression and task goals.

    Conclusion: Build an Evaluation System Oriented to Real Tasks

    The performance of LLMs in creative writing, long-range tasks, and strong instruction-following scenarios cannot be simply summarized as “smart” or “dumb.” What really matters is whether the model can produce stable output under complex constraints, maintain natural expression in creative tasks, preserve consistency across long contexts, and avoid defensive writing and task overfitting.

    For developers and enterprise users, building an evaluation system that is quantifiable, reproducible, and regression-friendly is more valuable than pursuing a single impressive generation result. The evaluation system should cover multiple dimensions, including instruction-following, style consistency, creative quality, long-range state preservation, anti-overfitting ability, safety compliance, and editability. At the same time, automated checks should be combined with human review, and prompt design, test set construction, engineering pipelines, and version regression should be integrated into a complete process.

    Ultimately, the goal of model capability evaluation is not to find a “universal model,” but to clarify the boundaries and strengths of different models across different task types. Only when evaluation is made sufficiently detailed can teams select appropriate capability combinations for real business needs and continuously improve the stability and quality of generated content.