Evaluating Large Language Models for Long-Form Chinese Content Creation: Methods and Practices

1. Reframing the Problem: From “Which Model Is Best?” to “How Should We Evaluate?”

The original question—which model writes Chinese best—often leads to the wrong conclusion in production. There are three main reasons:

1. Model capability is multidimensional, so no single ranking exists.
   A model may lead in structural planning but lag in colloquial dialogue; it may be stable on short texts but collapse on long-range consistency across episodes.

2. Creative tasks have high variance.
   A single generation can be affected strongly by prompt wording, sampling temperature, and random seed. Running the same model with the same prompt three times can reduce usability from 80% to 20%. Judging from one experience is essentially measuring luck.

3. “Best” depends on rework cost, not first-draft impressiveness.
   A draft that reads well but needs 60% rewriting can be more expensive than a plain draft that needs only 15% fine-tuning.

The professional approach is to turn this vague question into three engineering problems:

• Measurable: break “writes well” into indicators that can be scored and regression-tested.
• Reproducible: fix the evaluation set, scoring rules, and number of samples so results are comparable.
• Optimizable: use evaluation results to improve prompts, workflows, and model selection, not just to produce a ranking.

2. Six Observable Dimensions of Chinese Creative Writing

Using Chinese short-drama scriptwriting as an example—a pattern that also applies to web novels, voiceover scripts, and marketing copy—the following dimensions can be observed:

Dimension	What to observe	Failure signs
Outline planning	Conflict density, hook placement, setup-turn-resolution within each episode, arrangement of satisfaction beats and emotional curves	Flat outline, no episode-ending suspense, evenly distributed pacing
Instruction following	Word-count range, character appearance limits, banned expressions, scene count, tense and narrative person	An 800-character request becomes 1,500 characters; designated characters do not appear
Long-range consistency	Character names and forms of address, character relationships, timeline, props and foreshadowing payoff	Episode 3 conflicts with Episode 1; a dead character reappears
Language texture	Degree of colloquialism, ratio of dialogue to narration, sentence-pattern diversity	Excessive written-language style, stacked four-character phrases, repetitive episode structure
Editability	Share of drafts usable after minor edits, i.e. rework rate	Requires whole-section rewriting rather than polishing
Stability	Variance across repeated samples with the same prompt	Large quality swings across three generations

It is important to emphasize: “formulaic writing” is not eliminated by switching models. It is a structural tendency of probabilistic generation models on long texts—high-frequency, safe, and generic expressions always have the highest probability. A practical strategy is to suppress its occurrence through negative constraints, a style exemplar library, and post-editing review, rather than expecting a model to be naturally immune.

3. Turning Subjective Experience into Metrics

At least the following six quantitative metrics should be implemented. Among them, rework ratio and first-pass usability rate are the closest to cost:

• Instruction pass rate
  instruction_pass = satisfied hard constraints / total hard constraints
  Hard constraints include word count, character appearance, scene count, banned expressions, and other script-checkable items.

• First-pass usability rate
  first_pass_rate = episodes usable after minor edits / total episodes

• Rework ratio
  rework_ratio = rewritten characters / total generated characters
  Calculated using line-level diffs.

• Consistency conflict density
  conflict_density = number of conflicts / 1,000 characters

• Style deviation
  Use a style exemplar set to calculate the weighted distance of sentence-length distribution, punctuation density, and written-word ratio.

• Unit cost
  Tokens and calls consumed per usable episode.

Hard constraints are suitable for fully automated checks. Subjective items, such as tension, character voice, and gratification, still require human blind evaluation.

4. Evaluation Sets and Scoring Methods

Evaluation-set construction should follow stratified sampling. Cross-stratify by:

• Genre: urban, costume, suspense;
• Single-episode length: 600, 800, or 1,200 characters;
• Task type: outline, episode plan, body text, or revision.

Each cell should contain 10–20 items, and the total should be controlled between 200 and 400 items, so that a full run can complete within 30 minutes for daily regression.

Unified context: the input given to every model must be completely identical: setting package + previous episode summary + current episode requirements. If the entire previous manuscript is directly inserted into the context, long-context ability becomes a confounding variable and may hide differences in creative writing ability. A rolling summary is recommended as the memory carrier across episodes.

Scoring rubric entries should be written as decidable conditions. For example:

{
  "task": "episode_script",
  "hard_constraints": [
    {"name": "word_count", "op": "between", "args": [750, 850]},
    {"name": "scene_count", "op": "<=", "args": [4]},
    {"name": "banned_words", "op": "not_contains", "args": ["involuntarily", "the air seemed to freeze"]}
  ],
  "soft_scores": [
    {"dim": "outline_tension", "weight": 0.25, "scale": 1},
    {"dim": "dialogue_naturalness", "weight": 0.25, "scale": 1},
    {"dim": "consistency", "weight": 0.2, "scale": 1},
    {"dim": "editability", "weight": 0.2, "scale": 1},
    {"dim": "format_compliance", "weight": 0.1, "scale": 1}
  ],
  "pass_line": 3.2
}

Bias control: in human blind evaluation, hide the model source and randomly swap the presentation order of two drafts to offset position bias. Each sample should be independently sampled three times, and the report should include both mean and variance. Samples with excessive variance should be separately flagged.

5. Production Implementation: Layered Generation Pipeline

Model-selection conclusions only produce benefits when embedded into the workflow. A proven structure is “layered decoupling + structured intermediate artifacts,” where each layer can be evaluated and replaced independently:

Setting library (characters, world rules, forbidden elements)
      ↓
Overall outline: storyline + emotional curve          → story_bible.json
      ↓
Episode plan: 200–300 character detailed requirements per episode → episode_plan.json
      ↓
Episode draft: generate about 800 characters per episode → episode_draft
      ↓
Review: consistency checks + style rewriting          → final

The key point is that intermediate artifacts must be structured. In the earlier practice, detailed episode requirements of 200–300 characters were able to reduce rework to 10%–20% because they moved creative decisions forward to a low-cost text layer: editing one line of episode requirements is far cheaper than rewriting an entire episode. After being formalized into a schema, they can be automatically checked:

{
  "episode_no": 12,
  "goal": "The female lead discovers a suspicious point in the will and forms a temporary alliance with the male lead.",
  "characters": ["Lin Wan", "Lu Zheng"],
  "scenes": [{"id": "S1", "place": "Study in the old mansion", "time": "Night"}],
  "beats": [
    {"type": "setup", "desc": "Lin Wan verifies the signature date"},
    {"type": "turn", "desc": "Lu Zheng produces a second document"},
    {"type": "hook", "desc": "The seal on the document belongs to a third party who died three years ago"}
  ],
  "word_count_target": 800,
  "continuity_refs": ["prop:will", "fact:old scar on Lu Zheng's left hand"],
  "banned": ["involuntarily", "the air seemed to freeze"]
}

The corresponding generation call should be wrapped in a thin layer that decouples the prompt template from the model:

EPISODE_PROMPT = """You are a Chinese short-drama screenwriter. Write the full script of the episode based on the following episode requirements.
[Hard constraints]
- Word count: {wc_min}-{wc_max} characters
- No more than {max_scenes} scenes
- Only the following characters may appear: {characters}
- Banned expressions: {banned}
[Episode requirements]
{plan}
[Continuity information]
- Ending of previous episode: {last_hook}
- Settings that must be carried forward: {continuity}
[Style requirements]
- Dialogue should be colloquial and use short sentences; dialogue and narration should be roughly 3:7
- Avoid written-style summary sentences and stacked four-character phrases
- The end of the episode must leave a clear hook
"""
def generate_episode(client, plan, ctx, model="primary"):
    prompt = EPISODE_PROMPT.format(
        wc_min=plan["word_count_target"] - 50,
        wc_max=plan["word_count_target"] + 50,
        max_scenes=len(plan["scenes"]),
        characters=", ".join(plan["characters"]),
        banned=", ".join(plan["banned"]),
        plan=json.dumps(plan["beats"], ensure_ascii=False),
        last_hook=ctx.last_hook,
        continuity="; ".join(plan["continuity_refs"]),
    )
    draft = client.chat(model=model, messages=[
        {"role": "system", "content": "You are a rigorous Chinese scriptwriter who strictly follows hard constraints."},
        {"role": "user", "content": prompt},
    ], temperature=0.7)
    return postcheck(draft, plan)

6. Consistency Checks: Rule Gates + Model Gates

Prompts alone are not enough to guarantee consistency. Dual gates are required.

Rule layer (deterministic, zero-cost, run first): regex-based checks for character-name whitelist/blacklist, forms of address, banned expressions, word count, scene count, and punctuation density. Any hit should be marked as a failure and trigger a targeted rewrite.

Model layer (handles semantic conflicts): explicitly provide the setting list to the review role, require item-by-item verification, and request structured conclusions, rather than asking vaguely whether there are any problems.

AUDIT_PROMPT = """Audit the script against the setting list. Output only JSON.
Setting list:
{canon}
Script:
{script}
Output format: {"violations":[{"type":"name|timeline|prop|logic","quote":"original excerpt","reason":"conflict explanation"}],"score":1-5}
"""
def audit(canon, script, client):
    result = json.loads(client.chat(model="auditor",
                 messages=[{"role": "user",
                            "content": AUDIT_PROMPT.format(canon=canon, script=script)}],
                 temperature=0.0))
    if result["violations"] or result["score"] < 4:
        return retry_with_feedback(script, result)   # targeted rewrite rather than full episode rerun
    return script

The value of targeted rewriting lies in cost control: only the conflicting excerpt and audit feedback are fed back to the model. Token consumption is about 20%–30% of full-episode regeneration, and it does not introduce new random deviations.

7. Style Control and Suppressing Formulaic Writing

• Style exemplar library: extract 20–50 segments from historically approved drafts as few-shot examples, covering dialogue, conflict, and reversal scenarios. Examples are far more effective than adjectives—showing three sample paragraphs is better than merely saying “make it colloquial.”

• Externalize the negative list: banned expressions and clichéd sentence patterns should be placed in a configuration file, so they can be iterated with business feedback without modifying code.

• Sampling strategy: use medium temperature for body-text generation to preserve diversity; use near-zero temperature for review and extraction tasks to ensure determinism; apply a repetition penalty to high-frequency repeated tokens.

• Front-load format constraints: specify scene count and beat count at the episode-planning layer. The probability of structural violations at the body-text layer will decrease significantly.

8. Online Operations: Regression, Monitoring, and Cost Control

The evaluation system only has long-term value when integrated into daily R&D workflows:

• Regression set intake: fix failed cases judged by humans as “requiring major rewriting” into regression samples. Any prompt or model change must first pass the regression set.

• Caching and incremental generation: cache generation results using (setting library version, episode requirement hash, model version) as the key. If only one episode requirement is modified, recompute only that episode to avoid rerunning the entire book.

• Observability: report the call chain, latency, token usage, failure rate, and retry count to a unified logging service. Configure alerts for sudden spikes in rework ratio and for failure rates exceeding thresholds.

• Cost model: calculate cost per usable episode = generation cost × average retry count / first-pass usability rate. A model with a low first-pass usability rate may still have a higher total cost even if its unit price is cheaper.

• Platform-based model selection: model capabilities iterate quickly across versions. It is recommended to integrate multiple model services through a unified gateway and use configuration switching rather than code changes for A/B testing and canary releases. In cloud environments, you can combine general capabilities such as a model service platform, object storage for evaluation sets and generated artifacts, and logging, monitoring, and alerting. Platforms such as Baidu AI Cloud provide corresponding services; specific integration methods should follow each provider’s official documentation.

9. Conclusion

“Which model writes Chinese best?” is not a question with a single fixed answer, but an engineering problem that must be defined by the task. The practical recommendation is to proceed in this order:

1. First, build the evaluation set and scoring rubric (1–2 days) to turn subjective experience into repeatable experiments.
2. Then, build the layered pipeline and use structured intermediate artifacts to move creative decisions forward to a low-cost layer. This step usually improves rework more than switching models.
3. Next, add consistency and style gates, treating formulaic writing and setting conflicts as detectable defects rather than model defects.
4. Finally, integrate regression sets and monitoring so that model-selection conclusions can be continuously updated as models iterate.

Under this framework, “which model is better?” naturally converges into a continuously updated leaderboard, and every change can be explained and traced.