Evaluating Chinese Writing Quality of LLMs: A Reproducible Benchmarking Guide

“Which model is best at Chinese, and why don’t the others come close?” is a common claim in model selection debates. As an actionable engineering conclusion, it is missing at least three pieces of information: what task was evaluated, how large the sample was, and whether the observed gap is statistically significant.

The deeper problem is that “Chinese writing ability” is not a scalar property. It is a bundle of sub-abilities that do not correlate perfectly with one another. The same model can clearly win on official document writing while losing on novel dialogue, colloquial rewriting, classical poetry imitation, or character consistency across very long documents. Collapsing a multidimensional vector into a single rank produces a weighted sum under the evaluator’s subjective weights — change the weights, and the ranking can flip.

One more factor is routinely ignored: models keep changing. Any ranking that does not pin down a version, the call parameters (temperature, system prompt, context length), and the evaluation date has a shelf life of a few weeks. Treating “which model is best” as a one-off external fact lookup is therefore a methodological error. The correct move is to convert it into a capability you run continuously: define “good” with your own business data, use statistics to decide whether a difference is real, and use regression evaluation to keep before-and-after comparable.

1. Decompose “writes good Chinese” into measurable dimensions

You cannot evaluate what you have not defined. Cover at least the following seven dimensions and weight them per business scenario.

Dimension	What to observe	Typical failure modes
Normativity	Characters and words, grammar, punctuation, traditional/simplified mixing, Chinese–English mixing	Misused idioms, wrong collocations, mixed half-width and full-width punctuation
Semantic accuracy and factual consistency	Whether it actually answers the question, whether details are fabricated	Hallucinated clause numbers, data, or quotations
Register and style control	Official documents, technical docs, spoken style, literary writing, marketing copy	Runaway formality, pervasive “AI tone”
Discourse and logic	Layering, transitions, overall-to-specific structure, argument chains	Piled-up paragraphs, repetition, conclusions detached from evidence
Cultural context	Idiom and allusion usage, classical poetry meter, homophones, implicit pragmatics	Misattributed allusions, inappropriate tone
Long-form consistency	Terminology, person and voice, timeline, world settings	Character names or positions drifting in long documents
Instruction constraint compliance	Word counts, format, banned words, tone	Over-length output, unclosed templates, ignored negative constraints

Register control and instruction constraint compliance are the two most discriminative dimensions in Chinese scenarios. Chinese has little morphological variation; tone is carried mainly by function words, sentence length, and word choice. Whether a model can reliably produce an “official document feel” or a “conversational feel” usually says more about the Chinese depth of its training data than whether its grammar is correct.

Turn these dimensions into a weight table where the weights come from the business, not from a generic leaderboard:

RUBRIC = {
    "normativity": 0.15,   # Normativity: characters, grammar, punctuation
    "faithfulness": 0.25,  # Semantic accuracy and factual consistency
    "register": 0.20,      # Register and style control
    "structure": 0.15,     # Discourse and logic
    "culture": 0.10,       # Cultural context
    "longform": 0.10,      # Long-form consistency
    "constraint": 0.05,    # Instruction constraint compliance
}

2. Three automated approaches and where they stop being useful

2.1 N-gram overlap metrics

Compute the n-gram overlap between generated text and a reference answer. It is extremely cheap and fully reproducible, which makes it a good regression check in CI. The limits are equally obvious: it is almost blind to paraphrase (“reduce latency” and “cut response delay” barely overlap in n-grams yet mean the same thing); Chinese results depend heavily on the tokenizer and segmentation granularity, so they are not comparable across experiments; and on open-ended generation it correlates only weakly with human preference.

Verdict: suitable for relative comparison of one model across versions under one fixed prompt, not for ranking different models.

2.2 Semantic similarity scoring

Use sentence embeddings or a scoring model to measure the semantic relatedness between the generated text and the reference answer, or between the generated text and the requirement itself. This mitigates the paraphrase problem and cheaply catches “answered a different question.” But it is largely insensitive to logic errors, factual fabrication, and tone mistakes — the “reads smoothly but is wrong” class of problems, which is exactly where Chinese writing quality actually diverges.

Verdict: suitable for large-scale coarse filtering and flagging anomalous samples, not as a final judgment.

2.3 Strong-model judging with pairwise comparison

The most practical engineering approach is to let a strong model act as the judge and emit discrete grades rather than continuous scores. Relative scoring (A better / tie / B better) is far more stable than absolute scoring, because absolute scoring suffers from severe score inflation and anchor drift.

Three biases must be handled:

• Position bias: the judge tends to favor answers that come first or last. Evaluate each pair twice with the order swapped, and record a tie if the two conclusions conflict.
• Length bias: the judge systematically prefers longer, more elaborately structured answers. State explicitly in the rubric that verbosity earns no credit and repetition is penalized, and monitor the correlation between win rate and length.
• Self-preference: models tend to endorse output that resembles their own style. Mixing judge sources helps.

2.4 Human evaluation

Still the gold standard, and the most expensive. Three points matter: have multiple annotators label independently and compute an agreement coefficient (low agreement means the rubric is ill-defined, not that the models are close); annotators must judge blind; and size the sample by your target margin of error. For proportion metrics, n ≈ z²·p(1−p)/e² — to keep the win-rate estimate within ±5% at 95% confidence, you need roughly 385 valid samples.

The more economical combination in practice: have humans label 200–300 items as a calibration set, use it to validate agreement with automated judging, and then let automated judging carry the scaled evaluation.

3. A runnable aggregation and significance check

import random
PAIR_PROMPT = """You are a strict evaluator of Chinese writing. Compare the two responses using these dimensions:
normativity, factual consistency, register fit, discourse logic, cultural context, instruction constraints.
Verbosity, padding, and empty clichés earn no credit. Output only one of: A / B / TIE.
【Writing requirement】{requirement}
【Response A】{a}
【Response B】{b}
"""
def judge_pair(client, requirement, a, b, model="judge"):
    """Judge each order once and record a tie on conflict, to remove position bias."""
    r1 = client.generate(PAIR_PROMPT.format(requirement=requirement, a=a, b=b), model=model)
    r2 = client.generate(PAIR_PROMPT.format(requirement=requirement, a=b, b=a), model=model)
    flip = {"A": "B", "B": "A", "TIE": "TIE"}
    if r1.strip() == flip.get(r2.strip()):
        return "TIE"          # verdict flips with position, so the difference is indistinguishable
    return r1.strip()
def win_rate(results):
    """results: list of 'A'/'B'/'TIE'; returns A's win rate and its 95% bootstrap CI."""
    n = len(results)
    wins = sum(1 for r in results if r == "A")
    losses = sum(1 for r in results if r == "B")
    point = wins / (wins + losses) if (wins + losses) else 0.0
    boots = []
    for _ in range(2000):
        sample = [results[i] for i in (random.randrange(n) for _ in range(n))]
        w = sum(1 for r in sample if r == "A")
        l = sum(1 for r in sample if r == "B")
        if w + l:
            boots.append(w / (w + l))
    boots.sort()
    lo, hi = boots[int(0.025 * len(boots))], boots[int(0.975 * len(boots))]
    return point, (lo, hi), n - wins - losses   # third value is the number of ties
# Example: 55/100 and 58/100 have heavily overlapping intervals, so no stronger claim is justified
print(win_rate(["A"] * 55 + ["B"] * 40 + ["TIE"] * 5))

There is only one rule for reading the output: if the confidence intervals overlap substantially, the correct conclusion is “indistinguishable at the current sample size,” not “they are close” or “A is slightly better.” Distinguishing a 2–3 percentage point gap requires a sharp increase in sample size; the realistic option at that point is to admit the difference is indistinguishable and decide on secondary metrics such as cost and latency.

If you need a single ordering across multiple candidates, fit a Bradley-Terry model on the pairwise results. It converts win/loss data into ability estimates with confidence intervals, which is more robust than ranking by win counts.

4. Common pitfalls

• Data contamination: once public benchmark items leak into training corpora, scores stop reflecting real capability. Mitigate with a private evaluation set, rotated regularly.
• Prompt overfitting: tuning prompts and system instructions for one candidate, then comparing the tuned version against the others’ default configurations. Give every candidate an equal tuning budget.
• Unaligned parameters: temperature, max output length, and context window directly affect both quality and cost. Fix them and record them in the experiment log.
• Extrapolating from a single sample: the variance of one generation can exceed the gap between models. Resample critical items 3–5 times and take the majority verdict.
• Version and configuration drift: silent server-side upgrades invalidate historical conclusions. Pin the model identifier and call parameters in the evaluation record, and run evaluation on a schedule.
• Chinese token efficiency: identical character counts are billed differently under different tokenizers. Include cost per thousand characters and latency per thousand characters in selection decisions instead of looking at unit price alone.
• The average-score trap: leading on the total score does not mean leading in your scenario. Always break results down by business segment.

5. Putting the benchmark into production

1. Build a private gold set. Sample from real business logs after de-identification, stratified by intent × domain × difficulty, starting at 200–500 items. Each item carries a human-written or human-revised reference answer plus acceptance criteria.
2. Weight by scenario. Reorder the dimension weights from the first section per business. Customer support increases weight on instruction compliance and factual consistency; marketing increases weight on register and cultural context.
3. Make decisions multi-objective. Alongside the quality score, record latency percentiles, unit cost, context capacity, and content compliance, and produce a decision table rather than a single ranking.
4. Decouple the inference layer. Put a unified inference gateway between business code and model services, and centralize prompt templates, routing strategy, and fallback switches at the gateway. Model switching then stops touching business code, and A/B tests and canary releases become feasible.
5. Close the loop online. Send requests and scores to your log service and object storage, configure alerts on quality-score drops, over-length rates, and constraint violation rates, and run weekly human spot checks that feed back into the next gold set version.

If you build on Baidu AI Cloud, this architecture can lean on its model services for inference and judging, object storage for benchmark sets and result snapshots, a log service for call traces, and monitoring and alerting to hold the quality floor — without building the entire stack yourself.

6. Summary

“Which model writes the best Chinese” is not a question answerable in one shot; it is an engineering measurement problem. The workable path is to decompose Chinese writing into the seven observable dimensions above and weight them per business, use order-swapped pairwise judging with bootstrap confidence intervals to decide whether a difference is real, calibrate automated judging with a small amount of human annotation, and finally turn the whole thing into a scheduled job plus online monitoring.

Once you can answer “within my corpus, my weights, and this sample size, what interval does the A–B difference fall into,” model selection stops being an argument and becomes a reproducible, regression-testable engineering decision you can defend to the business.