How to Scientifically Evaluate an LLM’s Long-Form Chinese Writing Ability

1. Reframing the Problem: Why “Which Model Writes Chinese Best?” Has No Answer

Claims such as “Model X writes Chinese best” have almost no practical engineering value. There are three reasons:

First, subjective rankings are not reproducible. The same model can produce very different style quality across prompts, sampling parameters and context lengths. The spread between two configurations of one model can be larger than the gap between two models. A one-off impression that “A reads more smoothly” cannot be converted into reliable output for a team.

Second, ability is multidimensional, and the weighting is set by the use case. Fiction needs a feel for rhythm and emotional pacing. Official documents need formatting and terminology to be exact. Customer service scripts need compliance and tight short-sentence control. Screenplays need character consistency and preserved structural hierarchy. Leading on one dimension does not mean winning on total cost in the target scenario.

Third, “is the output good?” is not the same as “is it expensive to fix?” In production, the real cost is human review and revision time. If only 10–20% of an 800-character draft needs changes, it beats output that “reads better but needs half a rewrite.” The second one is a net loss.

So the argument of this article is: stop asking leaderboard-style questions and build an evaluation system that is self-proving and regression-ready. The rest breaks this down into five parts: evaluation dimensions, evaluation sets, automated regression, the generation pipeline, and selection decisions.

2. Breaking “Good Chinese Writing” Into Six Measurable Dimensions

Any capability you want to evaluate must first be decomposed into quantities that can be scored or counted. For long-form Chinese generation, use these six dimensions:

1. Sense of language and style consistency: rhythm of sentence length, natural use of idioms and colloquialisms, whether written and spoken registers are mixed inappropriately, and paragraph “breathing room.” Score it with a human rubric (1–5) and track the density of “pathological expressions” such as translationese, repeated sentence patterns, and stacked empty adjectives.
2. Structured long-form generation: given an outline, does the hierarchy hold? Concretely: no adding or removing plot nodes on its own, no reordering sections, no padding to hit a word count, and sub-nodes matching the granularity of their parent.
3. Instruction following and constraint satisfaction: character count ranges, person/POV, tense, banned words, format (storyboard, scene heading, dialogue form), and required elements. This should be measured automatically. It is the easiest dimension to quantify and the most often skipped.
4. Character and setting consistency: does the same character’s personality, tone of voice, forms of address and capability limits drift across rounds and chapters? Do setting conflicts occur?
5. Factual and timeline consistency: character ages, geographic relationships, timelines, and whether foreshadowing is paid off or contradicted. This is where long-form writing breaks most often.
6. Editability (revision cost): given a minimum-revision goal, what share of characters and what share of lines must change? This is the economic metric closest to production.

The first five measure whether the model writes well. The sixth measures whether it is affordable to use. Selection has to look at both.

3. Building the Evaluation Set: Turning Subjective Experience Into Data

3.1 Stratified Sampling Instead of Hand-Picked Prompts

The evaluation set should cover combinations across three axes: genre × difficulty × constraint density.

• Genre axis: dialogue-heavy scripts, explanatory documents, narrative fiction, marketing copy, official documents and reports.
• Difficulty axis: single passage (under ~300 characters) → single section draft (around 800 characters) → multi-node continuous writing (over 3,000 characters).
• Constraint density axis: no constraints → a few hard constraints (character count + POV) → high density (character cards, foreshadowing tables, banned words and format templates applied simultaneously).

In practice, high constraint density combined with long output separates models best. On low-constraint, short-output tasks most models pass, and real differences get drowned out by noise.

3.2 Rubric and Double-Blind Protocol

Each sample produces a scorecard with fixed fields. Keep the fields as orthogonal and as objectively decidable as possible:

rubric = {
  "structure_adherence": 1..5,   # Whether the output strictly follows the outline nodes
  "constraint_pass":   true/false, # Whether all hard constraints are met (decided automatically)
  "style_consistency": 1..5,
  "fact_conflict_count": int,     # Countable: lower is better
  "revision_ratio": 0.0..1.0,     # Characters revised by human reviewers / total characters
  "preference": "A" | "B" | "tie" # Blind preference
}

Three disciplines matter in the process: anonymization (strip model identifiers), randomization (randomize prompt order and candidate order to remove position bias), and independent scoring by two reviewers (disagreements go to arbitration). Otherwise you are measuring the evaluator’s expectations, not the model’s capability.

3.3 The Core Metric: First-Pass Usability

Use First-Pass Usability as the primary metric: the share of samples that can be adopted after generation with polishing only, with no human rewrite. The two supporting metrics are average revision character ratio and constraint satisfaction rate.

Be careful with numeric baselines. Anecdotes circulating online, such as “this model only needs one or two tenths revised on a single episode script,” are individual observations. They depend heavily on prompt design, how complete the character cards are, and sampling parameters, so they should not be quoted as general conclusions. The correct approach is to treat them as hypotheses to verify on your own evaluation set, and to use a first-pass usability of around 80% as a reference engineering target rather than an industry fact.

4. Automated Regression: Make Evaluation Run Continuously, Not Once

The model side drifts: version updates, sampling parameter changes, context strategy changes. Manual evaluation happens at most once. Regression has to be automated.

# Pseudo-code: main regression loop (structure only, no third-party dependencies)
def run_regression(suite, model_client, judges):
    results = []
    for case in suite:
        prompt_pack = build_prompt_pack(case)      # Assemble context: settings + outline + constraints
        outputs = [model_client.generate(prompt_pack, cfg)
                   for _ in range(case.repeat)]
        for out in outputs:
            hard = check_hard_constraints(out, case)   # Length / banned words / format / required elements
            fact = check_fact_conflicts(out, case.state)  # Compare against character cards and timeline
            results.append({
                "case_id": case.id,
                "constraint_pass": hard.passed,
                "fact_conflicts": fact.count,
                "score": judges.score(out, case.rubric),
            })
    report = aggregate(results)                  # Bucket statistics by genre / difficulty
    return report, diff_against_baseline(report)  # Compare with baseline; alert when threshold is exceeded

Three engineering points:

• Decide hard constraints with programs and soft quality with models or humans. Using another model as a “judge” scales evaluation up, but you must calibrate it against human scores and keep a percentage of real human spot checks.
• Read results bucketed by genre and difficulty. Do not let an overall average hide weaknesses. A model being strong on scripts and weak on official documents is completely normal.
• Set regression thresholds. A constraint satisfaction drop beyond a defined percentage should block release, instead of “looks fine, ship it.”

Version the evaluation set itself, archive it with the code in object storage, and send evaluation reports to logs and metrics systems so monitoring and alerting can track trends.

5. Layered Generation Pipeline: Why “Outline → Draft” Beats “Generate the Whole Long Text at Once”

The genuinely valuable lesson in the original experience was not a specific model but the generation method: produce a detailed outline from a small set of requirements, then draft section by section against that outline, which sharply lowers the revision ratio. This is a reusable engineering pattern that has nothing to do with model brand.

5.1 Externalizing Context: Turn “Memory” Into “Data”

The biggest enemy of long-form generation is context drift. The fix is to move state out of the conversation and into structured files:

{
  "characters": {
    "Character A": {"age": 27, "tone": "restrained, few exclamation marks", "speech_taboo": ["internet slang"]}
  },
  "timeline": [{"chapter": 1, "event": "key event", "unresolved": ["foreshadowing X"]}],
  "style_guide": {"sentence_avg_len": 18, "dialogue_ratio": 0.35, "banned_words": ["absolute", "epic"]}
}

Each generation step injects only the character cards, recap of prior events, and style guide relevant to the current node, instead of stuffing in the full text. This keeps the context length under control and reduces cross-contamination between sections.

5.2 Three-Tier Pipeline and Gates

Requirements → (1) Outline level (chapter / episode nodes) → (2) Detailed outline level (per node: goal, conflict, turn, characters on stage, character count)
     → (3) Draft level (node by node) → (4) Validator → (5) Local rewrite → (6) Final human review

The key is to put a validation gate between each pair of levels: the detailed outline must conform to the outline node set (no inventing or deleting nodes), and the draft must hit the key elements of the detailed outline. If a gate fails, roll back and rewrite rather than pushing forward. This saves several times the human effort compared with “generate everything, then do a big overall rewrite.”

5.3 Tune Sampling Parameters per Section Type

The common industry practice: raise sampling randomness for creative passages that need divergence; lower randomness and raise determinism for passages that must follow constraints strictly. Within a single long piece, use different configurations at different levels instead of one setting for the whole text.

5.4 Prompt Template Example (Abstracted, Portable to Any Model Service)

[Characters] Inject the character cards for characters on stage in this node only
[Prior context] Rolling summary of no more than N characters + list of unresolved foreshadowing
[This node's goal] Conflict / turn / information points that must be delivered
[Hard constraints] Character count range; POV; banned words; format; no plot additions not in the outline
[Output format] Scene heading + action description + dialogue; no explanatory text

Negative constraints such as “no plot additions not in the outline” usually buy more structural fidelity than adding a few more lines of “please write this brilliantly.”

6. Model Selection: Replace Gut Feel With a Weight Matrix

Plug evaluation results into business weights, and only then call it selection:

Scenario	Structural fidelity	Prose quality	Instruction following	Long context	Cost	Compliance / deployment
Series / screenplay	High	High	High	Medium	Medium	Medium
Official documents / reports	Medium	Low	Very high	High	Low	High
Marketing copy	Low	Very high	Medium	Low	High (high volume)	Medium
Technical documentation	High	Medium	High	High	Medium	Medium

The recommended rollout sequence: offline evaluation set → small-traffic A/B test → gradual rollout → continuous regression. Also decouple the model access layer from business code, and handle model routing, timeout retries, rate limiting and fallback in a unified inference gateway, so switching models or versions requires no business logic changes. The same decoupling applies if you use model services and engineering capabilities from a platform such as Baidu AI Cloud. Keep the evaluation scripts, evaluation sets, logs and metrics on your own engineering side, so you hold long-term authority over quality instead of being tied to a single model’s release cadence.

7. Common Pitfalls

• Evaluation set leakage and overfitting. Re-tuning prompts against the same prompts turns evaluation into memorization. Keep a held-out set that never sees daylight.
• Judging generation quality only, not revision cost. This is the most common misjudgment. Make revision_ratio a first-class metric.
• Generating the whole long text in one shot. The longer the text, the more likely the middle section drifts off the outline. Layered generation is close to mandatory.
• Ignoring content safety boundaries. You can over-block legitimate creative writing, or let violating content through. Give compliance checking its own path in the validator and track the false-block rate.
• Not tracking version drift. Regression must run after every model update, otherwise production quality silently degrades.
• Not testing mixed Chinese-English scenarios. Many business prompts, terms and code snippets are naturally mixed Chinese and English, so the evaluation set should include mixed-language samples on purpose.

8. Checklist

1. Do not ask “which model writes Chinese best.” Ask “which option has the lowest total cost under my scenario weights.”
2. Decompose Chinese writing into measurable dimensions: structural fidelity, constraint following, prose quality, factual consistency, editability and the rest.
3. Build a layered evaluation set and insist on anonymization, randomization, double-blind and two-reviewer scoring.
4. Use First-Pass Usability as the primary metric, with revision character ratio and constraint satisfaction rate as supporting metrics.
5. Replace one-shot long output with a pipeline of “outline → detailed outline → node-by-node draft → validation → local rewrite.”
6. Externalize settings and state as structured data, and inject only what the current context needs.
7. Automate evaluation and regression, decouple model access, and make decisions with the weight matrix.

Do these seven things and your team gets more than a verbal conclusion about “which model is better.” You get a quality baseline that can be re-run, compared and audited at any time.