Evaluating and Optimizing Large Language Model Chinese Expression: A Practical Engineering Guide
Choosing a large language model for Chinese-language output is hard because "good Chinese" is a multi-dimensional construct and casual comparisons rarely hold up. This guide turns that subjective debate into reliable, versioned quality data. It breaks "good expression" into eight measurable dimensions, calls out six common evaluation pitfalls, and lays out a practical pipeline: a stratified test set with trap samples, controlled inference settings, constraint-based prompt templates, double-blind scoring with pairwise win rates, Elo ratings, and bootstrap confidence intervals, plus automated pre-filtering. It then covers engineering remedies—style libraries with retrieval augmentation, glossaries and banned-word lists, outline-first generation, deterministic post-processing, and a clear human-AI division of labor—to bring model quality up to production standards. Read this if you need to make defensible, reproducible model-selection and iteration decisions for Chinese text generation.
1. Why “who writes the best Chinese” is hard to answer directly
In engineering practice, debates about “which model is stronger in Chinese” often devolve into sharing subjective experiences. Hand the same English aphorism to different models to translate: some produce literal renderings, some produce neatly parallel short sentences, and others take a long, mathematical detour. Such comparisons feel intuitive, but their conclusions are neither reproducible nor regression-testable, so they cannot support selection decisions. There are three reasons:
“Good Chinese expression” is a multi-faceted construct. At minimum it mixes semantic fidelity, syntactic fluency, register fit, handling of culturally loaded content, information density, and discourse structure—six abilities that are not fully positively correlated. The relative ranking of the same batch of models can be completely reversed between official-document writing and literary creation.
Task heterogeneity makes conclusions non-transferable. Translating an aphorism, writing an 800-word episode outline, and writing a product description all test different combinations of abilities. The first leans on pragmatics and rhythm; the latter two on structural consistency and attention to detail. A ranking derived from a single task cannot be extrapolated.
Sampling randomness and prompt sensitivity. Generative model output is inherently stochastic, and the variance introduced by prompt ambiguity is often larger than the real gap between models. “The prompt was too vague” is one of the main reasons evaluation conclusions flip back and forth.
So the right question is not “which is best,” but “under controlled conditions, for my specific task, which model has higher expected quality and lower variance.”
2. Breaking “good expression” into measurable dimensions
Use the following eight-dimension rubric as the scoring backbone, giving each dimension a clear failure definition so raters don’t score on overall impression.
| Dimension | What to observe | Typical failure signals |
|---|---|---|
| Semantic fidelity | Omissions, mistranslations, unauthorized additions | Lost key qualifiers, reversed causality |
| Syntactic fluency | Natural collocations, absence of translationese | Redundancy like “perform an operation of…” |
| Register fit | Match to written / spoken / official / marketing registers | Internet slang in an official document |
| Cultural loading | Idioms, allusions, parallelism, rhythm | Literal imagery, misused idioms |
| Information density | Effective information / total words | Long synonymous repetition, no-information transitions |
| Discourse structure | Outline organization, paragraph cohesion, logical progression | Section content unrelated to its heading |
| Terminology consistency | Stability of the same term / name across a long text | Drifting translations of the same concept |
| Facts and safety | Introduction of non-existent facts, compliance | Fabricated citations, inappropriate statements |
Among these, discourse structure and terminology consistency are the watershed for long-form tasks, and also the parts most easily missed by short-sample evaluation: models that perform similarly at 300 characters can diverge sharply by 5,000 characters.
3. Six pitfalls of common evaluation methods
- Drawing conclusions from a single sample. Without quantifying variance, a 0.1-point difference in the mean means nothing.
- Letting models self-evaluate. There is systematic bias (preferring longer output, output more like their own), so it can only serve as an auxiliary signal.
- Substituting scale or popularity for actual measurement. Parameter size has no stable monotonic relationship with performance in a specific register.
- Length and format bias. Human raters naturally favor longer output with lists and headings; the rubric needs an explicit “redundancy penalty” item.
- Ignoring version drift. After a minor version update, a product’s writing style and structural tendencies can noticeably regress. Evaluation must record model identifier + evaluation date + prompt version hash, and treat the test set as a regression set reused over the long term.
- Test set contamination and overfitting. Repeatedly tuning on the same batch of prompts inflates metrics; keep a holdout set that never participates in tuning.
4. A practical evaluation pipeline
4.1 Test set: stratified design + trap samples
Grid-stratify by capability dimension × register × difficulty × domain, with several items per cell minimum, then add trap samples: mixed Chinese-English, technical terms, idioms prone to misuse, ultra-long quotations, strict word-count requirements, and requirements to keep names consistent. A total of a few dozen to one or two hundred items is enough for a valid comparison—coverage matters more than absolute count.
4.2 Controlled inference conditions
Same system prompt, same sampling parameters (illustrative config below), same decoding strategy, and record the model version:
# Evaluation run configuration (illustrative)task_id: zh_expression_v1model_id: <evaluated model identifier> # injected by the eval platform, not hard-coded in the promptsampling:temperature: 0.2top_p: 0.9max_tokens: 2048prompt_template: templates/zh_expression_v1.mdeval_date: 2026-08-27prompt_hash: <hash of template content>
Low temperature is used only to reduce variance and does not represent the production configuration; production parameters should be evaluated separately.
4.3 Prompt templates: constraining register beats piling on adjectives
“Write more elegantly” is nearly useless; specifying the reader, register, structure, word count, and forbidden items is what works. Style samples (3–5 passages of text in the same register) usually work significantly better than any adjective description:
You are a Chinese technical writing editor. Please complete the following task.[Reader] Backend developers with 3+ years of experience[Register] Written technical explanation; no exclamatory sentences, no internet slang[Structure] Conclusion first, then three reasons, finally one caveat[Length] 600-800 characters, no filler transition sentences[Forbidden] "In today's era of...", "empower", "close the loop", first-person marketing tone[Style samples](paste 3-5 passages of real text in the target register)[Content to process]{{content}}[Self-check before output] Is terminology consistent throughout? Any fabricated facts? Any empty filler paragraphs?
4.4 Scoring and statistics
- Double-blind scoring + a third-person arbiter: raters see only IDs, without knowing the model source or order.
- Absolute scoring and pairwise comparison in parallel: absolute scores reveal dimensional weaknesses; pairwise comparison (win/tie/loss) reveals relative superiority and is more robust to rater drift.
- Severe-error veto: when factual fabrication, serious register mismatch, or large-scale terminology drift occurs, the sample is judged a loss directly and does not count toward the mean improvement.
import mathfrom collections import defaultdict# outcomes[(model_a, model_b, task_id)] = 1.0 / 0.5 / 0.0# 1.0 means model_a wins; 0.5 is a tiedef win_rate(outcomes, model):win = tie = total = 0.0for (a, b, _), score in outcomes.items():if a == model:total += 1win += scoretie += 1 if score == 0.5 else 0elif b == model:total += 1win += 1 - scoretie += 1 if score == 0.5 else 0if total == 0:return 0.0, 0.0, 0.0return win / total, tie / total, totaldef elo(outcomes, k=24, base=1500.0):"""Online approximation of Bradley-Terry: yields comparable relative strength and a sense of confidence"""rating = defaultdict(lambda: base)for (a, b, _), score in sorted(outcomes.items()):exp_a = 1 / (1 + 10 ** ((rating[b] - rating[a]) / 400))rating[a] += k * (score - exp_a)rating[b] += k * ((1 - score) - (1 - exp_a))return dict(rating)def bootstrap_ci(scores, n=2000, alpha=0.05):"""Bootstrap resampling over per-sample scores to give a mean confidence interval, avoiding small-sample misjudgment"""if not scores:return (float("nan"), float("nan"))rng_means = []import randomm = len(scores)for _ in range(n):sample = [scores[random.randrange(m)] for _ in range(m)]rng_means.append(sum(sample) / m)rng_means.sort()lo = rng_means[int(alpha / 2 * n)]hi = rng_means[int((1 - alpha / 2) * n) - 1]return (lo, hi)
Interpretation principle: if the confidence intervals corresponding to two models’ win-rate differences overlap, they should be judged as “no significant difference,” rather than declaring one model better at Chinese.
4.5 Automated pre-filtering
Running a rule check before human scoring significantly reduces invalid samples: repeated n-gram ratio, full/half-width and digit/unit conventions, consecutive punctuation, glossary hit rate, and whether word count falls within the constraint range. These items are objective and automatable, suitable for continuous integration:
lint_report = {"char_count": 742, # within [600, 800]?"repeat_ngram_ratio": 0.031, # threshold: 0.08 and above counts as redundant"fullwidth_digit_hits": 0, # full-width digits"term_hit_rate": 0.94, # glossary coverage rate"banned_phrases": ["empower"], # banned-word hits"verdict": "pass | warn | reject"}
5. When a model falls short: engineering remediation
The real value of evaluation lies in locating weaknesses and fixing them in a targeted way, not in swapping models and hoping for luck. Common combined techniques:
- Style library + retrieval augmentation: accumulate high-quality “rewrite pairs” (source text / desired Chinese expression) into a style library, and at generation time retrieve the several most similar samples to inject into the context. This usually improves register fit and removing translationese most directly.
- Glossary and banned-word list: inject as hard constraints at generation time, and as validation rules at post-processing time, intercepting name and term drift on both ends.
- Outline-first, body-second two-stage generation: fix section topics and word budgets at the outline stage, generate section by section and backfill at the body stage, then run a global consistency check. This is the most effective structural control for long-form tasks.
- Deterministic post-processing: full/half-width, digits and units, punctuation, Chinese-English spacing, name unification—done with rule scripts, not left to the model.
- Human-AI division-of-labor boundaries: let the model handle structure building, first-draft expansion, and material organization; literary rhythm, puns, and imagery still get human polishing. Chinese’s compression and rhythm remain the parts where humans are currently more reliable.
- Bad-case reflow: for each failed sample, add the desired output into the regression set; run regression first in the next evaluation to prevent iterations from introducing regressions.
On Baidu AI Cloud, this workflow can be assembled into a closed loop: the model service handles controlled inference and version management, logs and monitoring carry invocation and quality metrics, object storage holds the test set and scoring results, and scheduled jobs run periodic regression. All these capabilities are used as general-purpose categories, making them easy to replace and migrate.
6. Conclusion
“Which model has the best Chinese” has no universal answer across scenarios—only expected quality and stability for a specific task under controlled conditions. The actionable approach: first break “good expression” into eight judgeable dimensions, use a stratified test set and blind scoring to obtain pairwise win rates and confidence intervals, and express conclusions as verifiable statements like “A is significantly better than B in the official-document register, while there is no significant difference in the literary register”; meanwhile maintain the evaluation set as a regression set over the long term, and use the style library, glossary, two-stage generation, and human-AI collaboration to bring model capability up to production standards. Only when evaluation turns from “personal experience” into “quality data with versions and confidence intervals” do model selection and iteration truly gain engineering reproducibility.
