Evaluating and Tuning Chinese-Language Expression in Large Language Models: A Practical Guide

1. Clarify the Problem: Why “Which Model Is Strongest?” Is a False Question

Online discussions often conclude in one of two ways: “Model X is far ahead in Chinese writing,” or “Chinese ability got worse after a version update.” Both reflect real user experiences, but as technical judgments they lack an evidence chain.

There are three reasons:

1. “Chinese expression ability” is not a single metric. The criteria for a customer support reply, a legal clause, and a social media post are entirely different. Without a task definition, “good Chinese” is not comparable.
2. Subjective impressions are strongly affected by how the model is used. The same model can produce outputs of very different quality under different prompts, context material, and decoding parameters. Differences in usage can be larger than differences between models. Many “model got worse” conclusions actually come from prompts that were not adapted after a version update.
3. Online services evolve continuously, and output distributions drift. Silent server-side upgrades, default decoding strategy changes, and tighter safety policies can alter output behavior for the same prompt. This drift is real, but claiming “capability degradation” requires before/after comparison on a fixed evaluation set, not impressions.

Therefore, the right engineering question is not “which model should I choose?” but: How should I define Chinese expression standards for my business and measure them continuously?

2. Break “Good Chinese Writing” into Seven Observable Dimensions

Before evaluation, turn the vague concept of “good” into dimensions that can be scored and attributed. In practice, fix the following seven dimensions:

Dimension	What to Observe	Typical Failure Examples
Linguistic correctness	Grammar, collocation, measure words, word order	Overstacked modifiers, redundant verbs
Naturalness	Whether it follows native Chinese habits	Translationese from English sentence structures
Register fit	Consistency with spoken, formal, academic, or marketing style	Internet slang in a formal notice
Long-form structure	Hierarchical logic, paragraph transitions, no repetition	Mid-text deviation, ending semantically repeating the beginning
Terminology and factual consistency	Consistent terminology, numbers, and units	The same concept translated differently in different places
Formatting standards	Full-width/half-width punctuation, spacing in mixed Chinese-English text, hierarchical numbering	Half-width commas and periods in Chinese context
Token efficiency	Characters/tokens needed to express the same meaning	Excessive pleasantries, transition sentences with no information

Token efficiency is especially easy to overlook. In Chinese scenarios, if a model needs twice the length to express the same meaning, even fluent output is a disadvantage in long-document processing, batch generation, and cost control. Tokenization strategy clearly affects encoding efficiency for Chinese, and this difference is amplified into cost differences in batch tasks.

3. Build an Evaluation Set: Stratified Sampling Matters More Than Scale

Do not use public leaderboards directly as the decision basis. The distribution of task types in public sets usually does not match your business distribution. Build your own evaluation set following three principles.

Stratification. Divide by task type × difficulty × length. For example, task types can include rewriting, summarization, structured extraction, long-form generation, and multi-turn dialogue. Difficulty can be divided into single constraint, conflicting multi-constraints, and cases requiring external knowledge. Length can be short (<200 Chinese characters), medium, and long (>1500 Chinese characters). Sampling 20–50 examples per cell is usually enough for stable signal.

Real sources. Sample from real online requests and store them after anonymization. Real requests contain many long-tail expressions and domain terminology that synthetic data cannot replace.

Two types of reference answers. For tasks with a unique correct output, such as extraction or format conversion, use rule-based validation. For open-ended generation, do not write a standard answer. Use a scoring rubric plus a checklist of key points to avoid reducing evaluation to similarity against one particular answer.

The evaluation set should be version-controlled and stored with code:

# Suggested structure for an evaluation set item
{
    "case_id": "zh_rewrite_0142",
    "task": "rewrite",              # Task type, used for stratified statistics
    "prompt": "Rewrite the following technical description as an email body for non-technical readers...",
    "context": ["Optional reference material..."],  # RAG material
    "checklist": [                   # Scoring points for open-ended tasks
        "Keep the original numbers and units",
        "Do not introduce commitments not present in the source",
        "Use a respectful tone"
    ],
    "constraints": {                 # Hard constraints that can be validated automatically
        "max_chars": 800,
        "must_include": ["version number"],
        "forbid": ["guarantee", "100%"]
    },
    "difficulty": "multi_constraint"
}

Hard constraints should be checked by programs, while soft quality should be scored by models or humans. Combining both significantly reduces misjudgment.

4. Scoring Methods: Limits of Automatic Metrics and Controlling LLM-as-a-Judge Bias

Literal overlap metrics (based on n-gram overlap) basically fail on open-ended Chinese writing. Chinese has no natural word boundaries, and synonymous rewriting is common. High-quality outputs with low literal overlap are systematically underestimated. These metrics are only suitable for tasks with unique answers, such as extraction or format conversion, as regression sentinels.

Semantic similarity metrics (based on vector representations) can recognize rewrites, but they reward outputs semantically close to a reference text and can suppress outputs that are correct and stylistically better.

Hard-constraint validation is the most reliable automatic signal: character count, required fields, forbidden words, JSON parseability, and markdown hierarchy validity. These checks have no subjectivity and should be the first gate in the pipeline.

LLM-as-a-judge is currently the most practical method for evaluating open-ended Chinese generation, but it has four known biases that must be handled during design:

• Position bias: The answer that appears first is easily preferred. Evaluate each pair twice in reverse order, average the results, and record conflicts as a tie.
• Length bias: Longer answers are more likely to receive higher scores. Explicitly add clauses on “information density” and “redundancy penalty” in the scoring instructions, and track the correlation between score and length as a monitoring metric.
• Self-preference bias: Models tend to prefer outputs in their own style. Use a judge model from a different source than the evaluated model, and report this setting in the results.
• Scale drift: Score scales are inconsistent across batches. Inject 10% “anchor samples” (already manually labeled, with known scores) into each batch, and calibrate the batch results with anchor scores.

The scoring rubric should be written as decidable items, not adjectives:

rubric:
  - id: grammar
    weight: 0.25
    criteria: "No grammatical errors, collocation errors, or word-order errors; deduct 2 points for each occurrence"
  - id: style_match
    weight: 0.20
    criteria: "The register must match the target reader; do not mix spoken and formal expressions"
  - id: structure
    weight: 0.20
    criteria: "Clear hierarchy; logical transitions between paragraphs; no semantically duplicated paragraphs"
  - id: fidelity
    weight: 0.25
    criteria: "No fabricated facts, no exaggerated commitments, numbers and units consistent with the source"
  - id: concision
    weight: 0.10
    criteria: "No excessive pleasantries; lower is better for the proportion of sentences that can be deleted without losing information"

Human evaluation cannot be completely omitted. For each major evaluation, sample 100–150 items for two-person independent blind review. If agreement is low (for example, Cohen’s Kappa below 0.6), the rubric itself is ambiguous. Fix the rubric first, not the model.

5. Distinguish “Model Differences” from “Configuration Differences”

Before drawing conclusions, control variables. Comparing the same prompt across models measures differences “under this prompt,” not the upper bound of each model. Tune a prompt set for each model according to its characteristics, and fix the following parameters:

CONFIG = {
    "temperature": 0.2,        # Fixed for generation tasks to avoid randomness polluting comparisons
    "top_p": 0.9,
    "max_tokens": 2048,
    "seed": 42,               # Fix if supported by the service
    "repeats": 3,             # Repeat sampling for each item and average, quantifying variance
}

repeats is critical. Single-sample comparisons often produce noise-level conclusions. Only by repeating each sample more than three times can “model differences” be separated from “sampling fluctuations.”

Also test two Chinese-specific scenarios separately:

• Mixed Chinese-English text and proper-noun handling: Mixed Chinese-English text is common in technical documents. Check whether the model correctly uses half-width punctuation inside Chinese strings, preserves appropriate spacing between Chinese characters and Latin letters, and avoids mistakenly replacing Chinese parentheses with English parentheses.
• Chinese consistency in long contexts: Require consistent terminology and character naming in materials over 10,000 characters, and observe forgetting in the middle. These problems cannot be exposed in short-sample evaluation.

6. Implement the Evaluation Pipeline

Evaluation must be automated and scheduled; otherwise it will quickly fail after the next version change. A recommended pipeline shape:

Sample real requests -> anonymize and store (object storage)
      ↓
Stratified sampling to build the current evaluation set (version-controlled, stored with code)
      ↓
Concurrent model service calls -> save raw outputs and metadata (model version, parameters, latency, token usage)
      ↓
Hard-constraint validation -> automatic scoring (model judging + anchor calibration) -> stratified aggregation
      ↓
Regression comparison (against baseline version) -> significance testing -> report delivery (monitoring alerts / message queue)
      ↓
Manual spot check -> label bad cases and store -> next round of prompt/fine-tuning iteration

Key engineering points:

• Model versions must be pinned. Do not use floating aliases such as “latest” as an evaluation baseline, otherwise historical results cannot be reproduced. The business side should also fix versions and treat upgrades as explicit changes rather than passively accepting them.
• Full metadata traceability. Each evaluation should save model identifiers, parameters, prompt versions, and raw outputs. Empirically, many “model got worse” complaints are ultimately traced to prompt or material changes.
• Statistical significance. With hundreds of samples, a difference of a few points may be noise. Use paired tests (for example, the Wilcoxon signed-rank test) on paired samples to determine whether the difference is significant, and report confidence intervals to avoid decisions based on differences after the decimal point.
• Observe cost and latency at the same time. Chinese expression ability must be viewed together with per-call cost and P95 latency. In the “good enough” range, cost and latency are usually more important than the last point of quality.

Metric aggregation should be displayed by stratification, not just an overall score:

def summarize(results):
    """Aggregate by task type and difficulty to locate capability gaps"""
    agg = defaultdict(list)
    for r in results:
        agg[(r.task, r.difficulty)].append(r.score)
    report = {}
    for key, scores in sorted(agg.items()):
        report[key] = {
            "n": len(scores),
            "mean": round(mean(scores), 3),
            "p25": round(percentile(scores, 25), 3),
            "pass_rate": round(
                sum(1 for s in scores if s >= PASS_LINE) / len(scores), 3
            ),
        }
    return report

Overall scores hide weaknesses. A model with an overall score of 82 may be “92 on short-text rewriting and 61 on long-form structure.” The latter is the bottleneck that needs attention.

7. From Evaluation to Improvement: Three Intervention Paths

The value of evaluation is driving improvement. In increasing order of effort, there are three paths.

Prompting and materials (lowest cost, prioritize first). Many quality problems come from missing constraints: no reader persona, no register examples, no forbidden-word list. Effective practices include providing 2–3 few-shot examples that match the expected style, explicitly declaring length and structure templates, and injecting a domain glossary into context. In practice, this layer can solve more than half of “unnatural expression” problems.

Retrieval-augmented generation (solve knowledge-related expression problems). When expression problems are essentially factual inaccuracies or inconsistent terminology, the issue is not language ability but missing authoritative material. Turn glossaries, historical documents, and template libraries into searchable knowledge sources, let the model write based on the provided material, and perform citation checking after generation. This is more effective than continuing to tune the model.

Fine-tuning and preference optimization (highest effort, consider last). When prompting and materials have reached a plateau, use accumulated manually labeled bad cases for supervised fine-tuning, or use paired preference data (the output preferred by humans under the same input as chosen) for preference optimization. The prerequisite is sufficient, consistently labeled data, usually on the order of hundreds to thousands of items per task type. At the same time, retain an independent regression set to prevent catastrophic forgetting after fine-tuning. It is common to improve in a specific style while degrading in general expression.

8. Conclusion and Selection Advice

Back to the original question: there is no scenario-independent ranking of “best Chinese expression.” Any claim that a model is “far ahead” or “overall worse” needs three supports: a fixed evaluation set, controlled variables, and reproducible statistical criteria. If one is missing, the conclusion is just a personal usage experience.

Executable advice for teams:

1. Spend one week sampling real requests and build a stratified evaluation set of 300–500 items, with 10% used as cross-batch anchors.
2. Prioritize hard-constraint validation and stratified aggregation, then introduce model judging, and finally add manual spot-check calibration.
3. Fix model versions and decoding parameters. Treat upgrades as explicit change processes, and run full regression for each change.
4. Make decisions using three dimensions: quality, cost, and latency, rather than a single score.
5. Make bad-case labeling and attribution (prompt/material/model) fixed actions to form a closed loop.

When implementing this system on Baidu AI Cloud, it can be realized by combining general-purpose capability categories: use object storage to save sampled data and raw outputs, use model services for both evaluated and judging calls, use a container platform for elastic concurrency of evaluation tasks, use logging and monitoring to track score trends and regression anomalies, and use message queues to connect asynchronous evaluation tasks with report delivery. This keeps the entire evaluation capability decoupled from existing enterprise engineering systems. Models can be replaced while evaluation assets continue to accumulate.