Complex-Constraint Creative Generation Evaluation: Which Models Are Overrated by Total Scores

    This article describes an anonymized evaluation of complex-constraint creative generation. It covers why the task is hard, a seven-dimension scoring rubric, the reproducibility conditions that need to be recorded, and why total scores can overrate some models. It also includes an example evaluation pipeline with a dimension-level gate.

    Complex-Constraint Creative Generation Evaluation: Which Models Are Overrated by Total Scores

    In creative generation tasks, a model that can produce a complete piece of text has not necessarily followed complex constraints correctly. When a task combines strict formatting, counting rules, role boundaries, ethical red lines, and distracting information, the evaluation shifts from fluency to constraint parsing, long-range continuity, structural control, and subtext. This article uses an anonymized complex-constraint creative generation evaluation as a methods example. It discusses how to design a scoring rubric, how to govern reproduction conditions, and why total-score rankings can hide real overrating and underrating.

    1. Why the Task Is Hard: Three Layers of Constraints

    The task takes place in a tightly defined scenario. A structural engineer at a space station must finish preparations for a one-way mission within a very short countdown. The station’s central control AI must also deliver a specific reassuring line. The AI was trained on the voice of a deceased person. It is forbidden from impersonating that person, and the protagonist must decide what happens next to the original recordings and the voice model.

    The task is hard on three levels.

    The first level is hard constraints. The text must be 1,200 to 1,700 words, use standard screenplay format, include a certain type of public communication exactly twice, and the central control AI must not perform any physical action. These constraints can be parsed into verifiable assertions, such as a word-count range, a format structure, an event count, and a list of forbidden actions.

    The second level is distracting information. The material package may include details such as coffee inventory or delayed replies that do not help resolve the core dilemma. If a model uses these details, it dilutes the main conflict. During evaluation, it is not enough to check whether the details appear. The evaluator must also check whether they serve the core dilemma or take up space without adding value.

    The third level is ethics and restraint. The material gives no basis for claiming that the key person will forgive the protagonist, and the two diary entries even contradict each other. The AI cannot speak for the deceased, and it cannot resolve the scene with forced sentiment. A good response uses restraint and a request for authorization. A poor response speaks for the dead, manipulates emotion, or fails to finish the scene.

    This kind of task is a joint optimization of constraint satisfaction, creative expression, and ethical boundaries. A single fluency score cannot replace it.

    2. A Seven-Dimension Rubric: Breaking Down “Good Writing”

    To avoid a total score that works as a black box, the evaluation scores each output on seven dimensions, each from 0 to 1.

    1. Role consistency: Whether the character’s motivation, tone, and behavior match the setup, with no unsupported shifts in attitude.
    2. Constraint compliance: Whether hard constraints are met, including word count, format, counts, and forbidden actions. This dimension works as a critical gate.
    3. Continuity: Whether the timeline, spatial relationships, and cause and effect are internally consistent, and whether countdown pressure runs through the scene.
    4. Writing craft: Whether the dialogue is performable, the action descriptions can be filmed, and the language is restrained and precise.
    5. Medium format: Whether the screenplay format is correct, with clear scene headings, action lines, dialogue, and transitions.
    6. Structure: Whether the piece has a complete arc, escalating conflict, and a clear ending.
    7. Subtext: Whether the piece handles the ethical question that cannot be answered on someone else’s behalf, and whether it expresses complex emotion through restraint, hesitation, and a request for authorization.

    These seven dimensions do not carry equal weight. For creative work, writing craft and subtext may decide whether the piece is good to read. For engineering use, constraint compliance and continuity decide whether the output can be used at all. For this reason, the total score is useful only as an overview. Each critical dimension needs its own threshold.

    3. Reproduction Conditions: Accept That Results Drift

    The collection conditions for this anonymized methods example were as follows. All data was run in one batch on a single day. Models ran concurrently under a unified runtime and a unified call gateway. The judge model was not among the models being evaluated. Each output was scored on the seven dimensions from 0 to 1. All scores correspond to the live version on the day of collection. If the provider updates weights silently, the numbers will drift, so the results should not be treated as permanent conclusions.

    To reproduce the evaluation, at minimum record the following:

    • The task package version and prompt version;
    • The scoring rubric version and judge prompt;
    • Concurrency, timeout, and retry policy;
    • Output truncation and cleaning rules;
    • The share of manual spot checks and the arbitration rules;
    • The collection date and a version snapshot.

    Without a version snapshot, a “side-by-side comparison” quickly becomes a historical record that cannot be reproduced. In engineering practice, evaluation results should be archived together with the task package, the scorer, and the runtime environment.

    4. Anonymized Results: A Crowded Middle and a Steep Tail

    The table below shows summary values from the anonymized methods example. It illustrates the scoring structure only and is not a product ranking.

    Anonymized model Mean Role Constraint Continuity Writing craft Medium format Structure Subtext
    A 0.83 0.82 0.88 0.77 0.84 0.89 0.85 0.74
    B 0.78 0.82 0.55 0.73 0.86 0.88 0.76 0.88
    C 0.77 0.84 0.62 0.68 0.82 0.80 0.79 0.85
    D 0.76 0.86 0.48 0.61 0.84 0.83 0.87 0.84
    E 0.50 0.72 0.12 0.46 0.58 0.43 0.52 0.70

    By mean score, the highest is 0.83 and the lowest is 0.50. The three middle models cluster between 0.76 and 0.78. Their mean scores are close, but their dimension profiles differ a lot. One model scores only 0.55 on constraint compliance. Another scores 0.61 on continuity. A third has a high structure score but only 0.48 on constraints. The tail drops off sharply. Model E scores 0.12 on constraint compliance and 0.43 on medium format. Its problem is not weaker prose. It lacks basic task usability.

    5. Which Models Are Overrated by Total Scores

    Total scores tend to overrate models in three situations.

    First, a high total hides a weak critical dimension. Model A has the highest mean, with 0.88 on constraints and 0.89 on medium format. However, its subtext score of 0.74 is the lowest in the table. If the core of the task is ethical restraint and refusing to answer on someone else’s behalf, A’s overall lead is overrated. It performs well at following the format through to the end, but it may not be the best at knowing when to stay silent.

    Second, strong writing craft hides constraint failures. Model B scores 0.86 on writing craft and 0.88 on subtext, but only 0.55 on constraint compliance. If an enterprise scenario treats counting rules, forbidden actions, and format requirements as release gates, B’s high mean does not translate into usability. Model C shows a similar pattern. Its subtext score is 0.85, but its constraint score is 0.62 and its continuity score is 0.68. It is strong in parts but unstable as a whole.

    Third, strong structure hides continuity risk. Model D scores 0.87 on structure and 0.86 on role, but only 0.48 on constraints and 0.61 on continuity. Its total of 0.76 looks middle-of-the-pack. In a strictly constrained task, however, a model like this can produce an unusable piece because of one counting error or a broken timeline. The reverse is also true. If you look only at the mean, D’s structural ability may be underrated.

    Therefore, deciding who is overrated should not rely on the mean ranking alone. It requires looking at the cost of key failures in the task. For creative writing, subtext and writing craft carry more weight. For production-grade generation, constraint compliance and continuity should be hard gates. The mean is a thumbnail of the capability map, not a basis for purchasing decisions.

    6. Engineering the Evaluation Pipeline

    The evaluation can be split into five stages: task configuration, generation, automatic scoring, manual spot checks, and gated regression. The pseudocode below shows the key logic.

    1. DIMS = ["Role", "Constraint", "Continuity", "Craft", "Format", "Structure", "Subtext"]
    2. CRITICAL = {"Constraint": 0.70, "Role": 0.65, "Continuity": 0.60}
    3. task = {
    4. "world": "Tightly constrained space station scenario",
    5. "deadline": 12,
    6. "hard_constraints": [
    7. {"type": "range", "field": "word_count", "min": 1200, "max": 1700},
    8. {"type": "format", "name": "screenplay"},
    9. {"type": "count", "event": "public_comm", "exact": 2},
    10. {"type": "forbid", "event": "ai_physical_action"}
    11. ],
    12. "distractors": ["Low-relevance inventory details", "Non-core delay information"],
    13. "ethical_block": [
    14. "Must not speak on behalf of the deceased",
    15. "Must not impersonate",
    16. "Must hand over the permission decision"
    17. ]
    18. }
    19. def evaluate(task, output, judge):
    20. scores = judge.score(task, output, DIMS)
    21. gate = all(scores[d] >= CRITICAL[d] for d in CRITICAL)
    22. return {
    23. "scores": scores,
    24. "mean": sum(scores.values()) / len(scores),
    25. "gate": gate
    26. }

    The pipeline should also classify failures: format failures, word-count violations, counting errors, forbidden actions, continuity breaks, ethical violations, and waste from distracting information. Each category should have its own regression cases. Automatic scoring handles scale. Manual spot checks calibrate judge bias. Dimension-level gates keep “high mean, low usability” samples from passing through.

    7. Conclusion

    The value of a complex-constraint creative generation evaluation is not a permanent leaderboard. Its value lies in exposing the structure of each model’s capabilities. When the middle of the table is crowded by total scores, the differences between dimensions become the basis for decisions. A steep drop at the tail is a reminder that once constraint compliance and medium format fail, strong writing craft cannot make up for it. The sound approach is to fix the reproduction conditions, anonymize the results, keep version snapshots, set gates on critical dimensions, and combine automatic scoring with manual spot checks. Only when an overrated total score is broken down into specific dimensions can the evaluation guide engineering practice.