LLM Chinese Writing Ability Evaluation: A Practical Framework and Model Selection Guide

1. Start with the problem: “best in Chinese” cannot be measured without a task

In model selection discussions, one mistake comes up again and again: treating “Chinese writing ability” as a single scalar and ranking models against it. In practice, the same group of models often reverses order across tasks. One model clearly wins on lyrical essays, another produces a more stable structure in technical whitepapers, and a third sounds more grounded in short social-media copy. The answer to “which one is best” therefore depends on three prerequisites:

1. Task type: Long-form writing, conversational companions, information extraction, official and report writing, fiction, and professional content generation place different demands on language ability.
2. Quality criteria: Readers may care about readability, emotional impact, information density, format stability and reusability, or all of these in different proportions.
3. Constraint strength: Does the model need to strictly follow word counts, heading hierarchy, tone, a terminology list, or output structures such as JSON or tables?

The right engineering approach is not to look for the “strongest model.” Instead, build a Chinese writing evaluation system tailored to your business, so that candidate models produce comparable scores on the same task set and under the same scoring criteria. The rest of this article lays out an approach you can implement directly.

2. A six-dimension model for Chinese writing ability

Breaking a vague notion like “good writing” into observable, scoreable dimensions is the key to a converging evaluation. Six dimensions are recommended:

1. Language naturalness
This covers whether the word choice sounds idiomatic and whether sentences carry the breathing rhythm of a native Chinese speaker. It also covers whether translationese creeps in (literal calques such as “conduct a … operation” or “from the perspective of …”). Another part of it is whether the model leans on piling up four-character idioms and vague emotional statements. Typical failure modes include over-nested long sentences and too many connectives.

2. Structure and coherence in long-form writing
This covers whether a text of more than 2,000 characters has a single clear focus, a clear hierarchy, and real logical progression between paragraphs rather than restated content. It also covers whether settings and character details stay consistent throughout. Long-form output is currently the most discriminative dimension. Many models perform similarly within roughly 500 characters, but past the context compression threshold they start to show problems. Paragraphs begin to act independently, conclusions leak out early, and summaries repeat.

3. Context and cultural fit
This covers how well a model handles Chinese idioms and set phrases, colloquial sayings, policy language, industry jargon, internet memes, and implicit or indirect meaning. It also covers whether the model understands the Chinese writing convention of “cohesion through meaning rather than explicit formal connectives.” Finally, it covers social context such as holidays, forms of address, and honorifics.

4. Style and role control
This covers whether a single model can switch between styles such as official documents, colloquial speech, news writing, academic prose, scripts, and children’s books. It also covers whether it keeps its persona, tone, and forms of address stable across multiple turns of dialogue without “personality drift.”

5. Faithfulness and factual consistency
The baseline of Chinese writing ability is not rhetorical flourish but not fabricating information. The focus is on whether the model stays careful when citing data, policy provisions, code APIs, or biographical facts. It should also explicitly flag uncertainty when it encounters uncertain information, instead of dressing up made-up content in fluent wording.

6. Instruction following and format stability
This dimension measures how often hard constraints are met. Examples include minimum and maximum word counts, banned words, Markdown heading levels, table column counts, JSON field names and types, and mixed CJK and Latin text with numbers and units. It can be judged automatically by a program, which makes it cheap to run and highly discriminative.

3. Chinese writing style archetypes (vendor-neutral)

Without naming specific vendors, the Chinese writing styles commonly observed across the industry can be grouped into four archetypes. For model selection, matching a task to an archetype is more reliable than following leaderboard rankings.

• Literary-narrative: Strong at storytelling, argumentation, and building up complex viewpoints. Paragraphs connect naturally, and the model can sustain a rhythmic long-form text. The weakness is that it can be overly restrained and over-explanatory, and its prose can become stilted and bookish. It also lacks colloquial and internet-savvy tone.
• Balanced generalist: Offers a stable logical structure and clear explanations of complex technical problems. It can adjust between formal, colloquial, and technical tones on demand. The weakness is that its default writing often looks templated. Examples include excessive bullet points, forced conclusions, and overuse of “first / second / in summary.”
• Context-friendly: More attuned to Chinese internet language, everyday conversation, and short text. It produces natural dialogue with a high degree of anthropomorphism. The weakness is that long-form depth and factual rigor are weaker. It also tends to tell users what they expect to hear.
• Vertical specialist: Accurate expression and standard terminology in vertical domains such as code, law, medicine, and finance. The weakness is that its literary expression is rigid and it transfers poorly across writing styles.

4. Evaluation design: task set, rubrics, and statistical testing

4.1 Build the task set

Select 60–150 real tasks weighted by business importance. The set should cover all of the following areas:

• Long-form: commentaries, proposals, whitepapers
• Creative: short fiction, advertising copy, script excerpts
• Dialogue: customer support, casual conversation, multi-turn follow-up questions
• Professional: technical documentation, reports, key points from legal documents
• Constraint-driven: word limits, specified JSON, a specified terminology list

Each task fixes the system prompt, the user input, the scoring rubric, and reference answer key points (rubric anchors).

4.2 Scoring rubric

Rubrics need clearly defined anchors. Otherwise, variance between human raters will drown out differences between models.

{
  "task_id": "longform_comment_017",
  "task_type": "long_form_writing",
  "dimensions": [
    {"name": "language_naturalness", "weight": 0.2,
     "anchor_1": "Obvious translationese, connectives piled up",
     "anchor_3": "Generally fluent, with a few awkward constructions",
     "anchor_5": "Native-level prose with no redundant embellishment"},
    {"name": "structure_coherence", "weight": 0.25,
     "anchor_1": "Paragraphs show no logical progression", "anchor_3": "Clear hierarchy but with repetition",
     "anchor_5": "Progressive argument with internally consistent details"},
    {"name": "context_culture", "weight": 0.15},
    {"name": "style_control", "weight": 0.15},
    {"name": "faithfulness", "weight": 0.15,
     "anchor_1": "Contains verifiable fabricated facts",
     "anchor_3": "Uncertainties left unstated", "anchor_5": "Uncertain information explicitly flagged"},
    {"name": "instruction_following", "weight": 0.1}
  ],
  "hard_constraints": ["1200-1500 characters", "no headings below H2", "plain Markdown output only"]
}

4.2.1 Automatically checkable hard metrics

Run programmatic checks first, and reserve human review budget for the subjective dimensions. Checks include:

• word count
• heading levels
• banned words
• JSON parse rate
• field completeness
• rules for inappropriate content
• repeated n-gram ratio, used to detect verbatim repetition
• sentence-length variance, used to detect monotonous sentence patterns

4.3 Blind human review and statistical testing

Use double-blind pairwise comparison. Hide the models’ identities, shuffle the order at random, and have at least two reviewers score each sample independently. Track agreement between reviewers with Cohen’s Kappa for categorical scores or Krippendorff’s α for continuous scores. An α below 0.6 means the rubric anchors are unclear and the rubric needs recalibration, rather than the scores being taken at face value.

Each pair of models is compared on the same tasks, so the scores are paired samples. A non-parametric test is recommended instead of comparing means directly:

from scipy.stats import wilcoxon
# a_scores / b_scores: total scores of two models on the same tasks
# (averaged across reviewers)
stat, p_value = wilcoxon(a_scores, b_scores, alternative="two-sided")
n = len(a_scores)
diff = [x - y for x, y in zip(a_scores, b_scores)]
nz = [d for d in diff if d != 0]
pos = sum(1 for d in nz if d > 0)
print(f"Paired samples: {n}, non-zero diffs: {len(nz)}, A wins: {pos}")
print(f"p = {p_value:.4f} -> {'significant' if p_value < 0.05 else 'not distinguishable'}")

Practical recommendations:

• Sample each model three times per task and use the mean to cancel out randomness.
• Report 95% confidence intervals rather than point scores.
• When p ≥ 0.05, write the conclusion as “the two models are not distinguishable on this task.” Then make the decision using engineering metrics such as cost, latency, context length, and support for on-premises deployment.

4.4 Using LLM-as-judge correctly

Automated scoring can cut costs substantially, but three practices are essential:

1. Give the judge model the full rubric and anchors, rather than simply asking it to “score from 1 to 10.”
2. Require it to cite evidence from the output for each dimension before giving a score. This discourages scoring based on overall impression.
3. Calibrate on 50 human-labeled samples first. Only use the judge for large-scale screening once its correlation with human scores (Spearman’s ρ) reaches 0.7 or higher.

5. Engineering implementation: constrain output, route by task, and build a regression baseline

Replace adjectives with a style card and examples. Instructions like “write more vividly” are nearly impossible to execute. Constraints that can be checked are far more effective:

[STYLE CARD]
- Person and point of view: third-person limited, focused on the protagonist
- Sentence length: 25-40 characters on average, alternating long and short;
  never use three consecutive sentences of the same length
- Banned expressions: "first / second / in summary", "empower", "closed loop",
  "it is worth noting"
- Each paragraph carries a single information point, and consecutive paragraphs
  must advance by cause and effect or by time
- Mark uncertain factual content as "according to public information,
  pending verification"
[OUTPUT] Plain Markdown, one H1 heading, body of 1200-1500 characters

Generate long-form content in stages and check consistency. Produce long-form text in three stages: outline → sections → final pass. First generate an outline that includes argument points and the evidence needed for each. Then expand the outline section by section. Finally, run one global consistency review of characters, data sources, and terminology. This suppresses the late-stage quality collapse that often follows a single long generation.

Route by task instead of using one global model. Make model choice a policy layer. Route long-form and literary tasks to models with a more narrative style, and technical Q&A and reports to models with a stable structure. Route short text and conversational tasks to context-friendly models. Keep routing rules, weights, timeouts, and fallback paths in centralized configuration, rather than hard-coding them in business code.

Archive evaluation data and reports. Keep the sample set, scoring records, model versions, and prompt versions under version control together. Store raw text and evaluation reports in object storage. Connect metrics such as evaluation task and scoring latency and failure rate to logging and monitoring services. Pipelines like this can be built on Baidu AI Cloud by combining general-purpose capabilities such as model services, object storage, logging, and monitoring and alerting. The design is not tied to any specific vendor.

Turn evaluation into CI. Every time a prompt changes or a model version changes, run the fixed regression set. Output the per-dimension score differences and hard-constraint pass rates, and block the release if scores drop beyond a threshold. This is the only reliable way to prevent quality drift caused by tuning prompts by feel.

6. Common pitfalls

• Testing only short text: A 50-character answer does not reveal differences in long-form coherence, so long-form tasks must be part of every test.
• Using a single reviewer: One person’s preferences will be dominated by their taste in writing style. Always use multiple reviewers in a double-blind setup.
• Ignoring failure samples: Averages hide tail risks. Track factual error rates and format violation rates separately.
• Using “feels more human” as the only metric: Fluent but factually wrong output causes greater losses in production. The weight of faithfulness should not be below 15%.
• Selecting a model once and never revisiting it: Both models and prompts evolve, so a quarterly re-evaluation should be routine.

7. Conclusion

There is no global champion in Chinese writing ability across all tasks. The best choice only exists for a specific task, a specific set of quality criteria, and specific constraints. Combine the six-dimension capability model, the anchored rubric, double-blind pairwise comparison, and statistical testing. Add task routing and a regression baseline, and your team can reach trustworthy conclusions within hours instead of getting stuck in long-running debates based on reputation. The value of the evaluation framework is also independent of any specific model. It ensures that your basis for judgment still holds when models are updated.