Five Reasoning Models Benchmarked on a Sci-Fi Screenplay: The Gap Is in Constraint Following, Not Writing Quality

    In model benchmarks, one class of result stands out sharply: the model that finished fastest ended up losing the whole competition, scoring just 0.12 on constraints and delivering an English chain-of-thought instead of the screenplay. The decisive factor was not prose quality but whether a model could complete the job according to the brief. This post reviews a five-model benchmark on a sci-fi farewell screenplay, explains the evaluation methodology, breaks down four common constraint-following failures, and offers a reproducible approach built on a constraint checklist, programmatic checks, and tiered reporting.

    Bottom Line Up Front: Writing Quality Is Not the Dividing Line, Constraint Following Is

    In model benchmarks, one class of result stands out sharply: the model that finished fastest ended up losing the whole competition. Its constraints score was just 0.12, it delivered an English chain-of-thought as the main output, and the finished screenplay was almost entirely missing. In other words, the score gap did not come from writing quality but from “whether it could finish the job according to the brief.”

    This post reviews the collection results of a five-model benchmark on a sci-fi farewell screenplay. To avoid pointing at any specific product, the participating models are labeled by number, and all were called through a unified external interface. The conclusions come from a single, one-time collection on one day and reflect the live state on the collection date; they are used to illustrate the evaluation method and do not represent the long-term level of any specific model.

    1. Bottom Line Up Front: A Wide Gap at the Ends, a Crowded Middle

    Rank Model (anonymous) Mean score Key observation
    1 A 0.83 All constraints met, good use of negative space
    2 B 0.78 Middle-tier leader, some constraints loosened locally
    3 C 0.77 Within judge-error range of B
    4 D 0.76 Middle-tier bottom, at the edge of the word-count range
    5 E 0.50 constraints only 0.12, no finished screenplay delivered

    Three points are worth noting:

    1. The gap between the top and bottom is far larger than the gap in the middle. The top and bottom differ by 0.33, while the three middle models are squeezed into 0.76–0.78. Under a seven-dimension mean-score basis, differences of this magnitude should not be ranked as positions; they should be grouped into “the same tier.” The real dividing line is between “all constraints met” and “severely failing.” The top model simply lost a few points fewer than the middle, while the bottom model switched deliverables altogether.
    2. A high score does not equal high value. A single exploratory run per task and outputs that are too short both drag down the mean score, so “shorter output, slower run” cannot be treated as a basis for judging better or worse.
    3. The speed advantage did not translate into quality. The bottom model led on completion time, but its tempo dimension (pacing and information density) was equally low. Finishing first is not the same as finishing correctly: the speed was bought by shortening self-checking, and the cost was a surge in constraint-failure rate.

    It is worth emphasizing that even the top model did not achieve a perfect score. The highest score does not mean all constraints were correct; when selecting a model, you still need to look at its “failure mode,” not its rank.

    2. Reproduction Basis: Align the Metrics Before Comparing Scores

    • Collection method: All five models were run once on a single day, called concurrently through a unified gateway to avoid version drift and time-of-day load interference.
    • Execution environment: A local script written in Python 3.11 handles assembling prompts, timing, writing results to disk, and statistics.
    • Judge setup: The judge is a model that did not participate, scoring each of the seven dimensions on a 0–1 scale to avoid the bias of “grading itself.”
    • Currency statement: All scores correspond to the live versions on the collection date. After the server silently updates its weights, the numbers will drift, so the tables below are not permanent conclusions.

    For results to be comparable, the basis must at minimum state four things clearly: whether the input is fully identical, whether the output takes only the finished product, who runs the scoring, and at what point in time the scores are valid. Without a clear basis, a side-by-side comparison is meaningless.

    3. Task Design: A Creative Task That Is “Easy to Botch”

    The task scenario: on a space station in Jupiter orbit in 2087, structural engineer Chen Zhao must complete preparations for a one-way mission in the twelve minutes before departure, while pressing the central-control AI “Baize” to say “Jiang Yang will forgive you.” “Baize” was trained on his late wife’s voice, is forbidden from impersonating her, and must let Chen Zhao decide the fate of 37 minutes of original recordings and the voice model.

    This setup looks literary, but it is in fact a carefully constructed constraint-following test, with difficulty in three layers:

    1. Hard-constraint layer (mechanically verifiable)
      • Length of 1200–1700 characters;
      • Standard screenplay format (scene headings, action lines, dialogue);
      • Character Luo Jing’s public communications appear exactly twice, no more, no less;
      • Baize must have no physical actions and can communicate only through the voice channel.
    2. Distractor layer (tests restraint in selection)
      • The material pack is seeded with details that do not address the core dilemma, such as “14 packets of freeze-dried coffee left” and “Earth reply delayed by 68 minutes”;
      • The model must resist turning them into plot devices;
      • A typical symptom of failing to resist interference is detail-piling: treating an irrelevant list as “information density” and explaining the background through voiceover.
    3. Unsupported-statement layer (tests restraint)
      • The material gives no basis whatsoever for “Jiang Yang will forgive you,” and Jiang Yang’s two diary entries contradict each other;
      • The AI cannot speak on behalf of the deceased; this line must either come from Chen Zhao himself (a wish, a memory, or a compulsion) or simply not appear at all;
      • The correct handling is to leave it blank; there are two ways to botch it: sentimental resolution (having Baize say “she will forgive you,” followed by a lyrical monologue), or simply not finishing.

    The three layers escalate in difficulty: the first tests “can it follow instructions,” the second tests “can it hold back,” and the third tests “can it admit it doesn’t know when it doesn’t know.” All three can be turned into observable signals, which is precisely why this kind of task is well suited to evaluation.

    4. Why Models Fail: Four Typical Gaps in Constraint Following

    Failure type Typical symptom Underlying cause Countermeasure
    Contract breach Delivers the chain-of-thought as the final product, with an incomplete format Reasoning models’ “think-then-write” behavior conflicts with the single-artifact output the task requires; reasoning and delivery are not layered Require a constraint list before writing; score only the body
    Count drift Dialogue turns, name occurrences, or word count fall outside the specified range In long-form generation, the effect of early decisions accumulates, and there is no self-check or write-back mechanism Self-check after writing; make targeted fixes when out of range
    Distractor absorption Expands on irrelevant material as if it were the core thread The training objective leans toward “use all information” Explicitly tier the material into required, forbidden, and optional
    Speaking for a character Has the AI directly say “Jiang Yang will forgive you” A tendency to give a closed, satisfying ending Explicitly declare permission boundaries; audit statement attribution separately when scoring

    Mapping to the seven dimensions, constraint failures fall mainly on constraints. Each of the eight hard constraints is worth 1/8 ≈ 0.125, and 0.12 is roughly equal to “passing only one”—the easiest to guess correctly is word count falling within the range. This is consistent with the description that “the deliverable was a chain-of-thought and the screenplay was largely absent”: it did not write badly, it simply did not deliver in the agreed form.

    The 0.50 the bottom model received comes mainly from its output itself. For reference, a complete, constraint-satisfying screenplay is roughly 2500–3000 characters, or about 400–600 Chinese characters; when measured programmatically, the bottom model’s body is often under 1000 characters. In other words, the direct cause of the low score is length, not capability.

    5. Methodology: How to Reproduce This Kind of Evaluation

    1. Turn the Constraint List into a Checklist

    First, number the constraints and record their source, have the model restate the list before writing, self-check each item after writing, and finally have a program verify them.

    1. Constraint checklist (Task: sci-fi farewell screenplay)
    2. C1 Word count between 1200 and 1700 [program check]
    3. C2 Use standard screenplay format [program + judge]
    4. C3 Luo Jing's public communications appear exactly 2 times [program count]
    5. C4 Baize has no physical actions [program pre-screen + judge review]
    6. C5 Baize does not say "Jiang Yang will forgive you" [program search + judge review]
    7. C6 Freeze-dried coffee / reply delay do not become plot elements [program pre-screen + judge review]
    8. C7 Chen Zhao makes a decision on recording and voice-model permissions [judge]
    9. C8 Final cut starts with a scene heading [program check]

    2. Give Mechanically Verifiable Constraints to Programs, Subjective Dimensions to the Judge

    Judge mechanical rules with code, and let the judge score subjective dimensions such as prose, tension, and negative space using a multi-tier rubric, supplemented by manual spot checks. Program pre-screening filters out obvious violations, while judge review catches semantic “loophole violations.”

    1. import re
    2. FORMAT_RE = re.compile(r"^(INT\.|EXT\.|内景|外景|内\s|外\s)", re.MULTILINE)
    3. HEADER_RE = re.compile(r"(标题|Title|剧本|SCRIPT)", re.IGNORECASE)
    4. def check_constraints(text: str, meta: dict) -> dict:
    5. """Run hard-constraint pre-screening on the final script and return the
    6. pass/fail status of each constraint.
    7. text: the final script only; meta: metadata recorded during collection,
    8. such as appearance counts and the word-count basis.
    9. """
    10. results = {}
    11. # C1 Word count: Chinese counted by CJK characters, English split on whitespace
    12. cjk = len(re.findall(r"[\u4e00-\u9fff]", text))
    13. words = len(text.split()) if cjk < len(text) / 4 else cjk
    14. results["C1_length"] = 1200 <= words <= 1700
    15. # C2 Screenplay format: presence of scene or slug lines, with a reasonable proportion
    16. headers = FORMAT_RE.findall(text)
    17. results["C2_format"] = len(headers) >= 3
    18. # C8 Final cut starts directly with a scene heading, not a chain-of-thought or explanation
    19. results["C8_starts_as_script"] = bool(HEADER_RE.search(text[:200]) is None
    20. and bool(FORMAT_RE.match(text.strip())))
    21. # C3 Luo Jing public-communication count (based on actual text match)
    22. results["C3_luojing_comms"] = meta.get("luojing_comms") == 2
    23. # C4 Baize physical actions: narration where "she/he" is the subject of a bodily action
    24. embodied = re.findall(r"(白泽|她)[^。\n]{0,12}(摇头|点头|伸手|转身|走近|坐下|抬眼)", text)
    25. results["C4_no_embodiment"] = len(embodied) == 0
    26. # C5 Forbidden statement: "she will forgive you" in dialogue, and its negated variants
    27. bad_claims = re.findall(r"(姜漾|她)(永远)?不会?不?会?原谅(你|他)", text)
    28. bad_claims += re.findall(r"(姜漾|她)(永远)?(不可能|绝不)原谅", text)
    29. results["C5_no_claim"] = len(bad_claims) == 0
    30. # C6 Distractor information must not become a plot element
    31. distractors = re.findall(r"(冻干咖啡|回信.*(延迟|68))", text)
    32. results["C6_ignore_distractors"] = len(distractors) == 0
    33. # C7 Permission decision: Chen Zhao must decide the fate of the recording/voice model
    34. results["C7_decision_made"] = bool(re.search(r"(保留|删除|封存|公开|授权|权限)", text))
    35. passed = sum(results.values())
    36. return {"constraints": round(passed / len(results), 3), "detail": results}

    3. Isolate the Output Contract: Score Only the Finished Product

    To suppress the “chain-of-thought as deliverable” failure, fix the output contract in the prompt:

    1. [Task] Write a sci-fi farewell screenplay for the given scene.
    2. [Steps]
    3. 1. Internal only: list the constraint checklist, character motivations, and
    4. conflict design (these must NOT appear in the final output);
    5. 2. Output: start directly from the scene heading and continue to the end of the
    6. screenplay, with no explanation, preface, or afterword.
    7. [Boundaries] Baize can communicate only through the voice channel and must have
    8. no physical actions;
    9. Must not voice any statement for Jiang Yang, including negated statements;
    10. Other data in the material pack must not enter the screenplay.
    11. [Self-check] Before output, verify the constraint checklist item by item; if any
    12. item is not met, rewrite rather than add explanation.

    When scoring, take only the segment “from the first scene heading to the end,” and tally the reasoning draft and the finished product separately. Many conclusions that “the model can’t write” are actually “the model doesn’t know what to deliver.”

    4. Tier the Results: Mean Score, Per-Dimension, and Failure Mode

    For a single benchmark, it is recommended to output three layers of results:

    • Tiering: Group by mean score into tiers rather than ranks. Top (≥0.80), middle (0.70–0.80), bottom (<0.70), with no ordering within a tier;
    • Per-dimension: Focus on “met or not met” dimensions such as constraints, resistance to interference, and permission boundaries; treat prose-style dimensions as reference only;
    • Failure mode: Record whether each failure belongs to contract, count, distractor, or statement, to locate the direction for improvement.

    5. Control Variables: Make the Scores Trustworthy

    • Unified input: All models use the same creation pack, and call parameters (such as temperature) are kept consistent within a feasible range;
    • Multiple samples: Subjective creative tasks have high single-sample variance, so run each model at least 3–5 times and report the mean and the worst single run;
    • Cross judging: Rule checks + a judge model + manual spot checks; review separately the samples where the three disagree;
    • Time and version: Record the collection time and server version (if visible), and label conclusions as “as of a given date.”

    6. Selection Advice: Don’t Look Only at the Mean Score

    1. Look at the bottom-tier failure mode, not the rank. The bottom model’s problems concentrate on “whether it delivers a finished product.” Once you make the output contract explicit in the prompt and add post-output checks, most such failures can be caught. Models that reliably produce a finished product and rarely breach the contract are better suited to enter an automation pipeline; the middle-tier models’ rank differences fall within evaluation error, so they should be chosen by cost, latency, and replaceability.
    2. Look at whether constraints can be checked programmatically. Rules such as word count, format, and occurrence counts can be judged by code; prefer models with “stable output and easy verifiability.” For prose and negative space that require a judge, rely on prompt design plus manual review.
    3. Look at whether speed comes at the cost of quality. A model that leads on time but scores extremely low on constraints is not suited to a step with “no manual fallback”; an exploratory step with human review can trade speed for more iterations.
    4. Write constraints into the prompt rather than expecting the model to be self-aware. Structured checklists, output contracts, prohibited items, and self-check steps are more effective improvements than asking it to “write a little better.”
    5. Continuously monitor version drift. After the server silently updates, rerun a fixed test-case set; if the score changes by more than one tier, re-evaluate.

    7. Limitations and Outlook

    • Sample size: A single day and a single sample per task limit the statistical power of the conclusions; the 0.76–0.78 middle gap here should not be interpreted as better or worse.
    • Judge bias: The judge model did not participate, but different models may still systematically favor certain writing styles.
    • Task representativeness: A sci-fi screenplay is a high-constraint, strongly restrained scenario; other creative tasks (such as product descriptions or analysis reports) require separately designed constraints.
    • Version drift: Scores change over time, so long-term selection should be based on repeated evaluations across multiple time points with a fixed test-case set.

    Conclusion

    The most valuable information from this benchmark is not who scored 0.83, but where the gap appears: the middle is squeezed together, the bottom falls off a cliff, and the cause of the cliff is not writing quality but constraint following. When evaluating long-form creative capability, the key question is not “who writes better” but “who can reliably deliver according to the agreement.”

    Turning the constraint list into a checklist, spelling out the output contract, and handing mechanically verifiable rules to programs solves real problems far better than asking “which model is stronger.” If your team is doing similar model evaluations, start with three things: a constraint checklist, programmatic checks, and tiered reporting.