How to Evaluate Chinese Language Capability in Large Language Models: Methods, Rubrics, and Metrics

Why “Which Model Is Best in Chinese?” Is Hard to Answer

From an engineering perspective, the request to “compare Chinese expression ability” lacks three necessary conditions: a clearly defined task scope, quantifiable evaluation dimensions, and reproducible scoring rules. Without them, any conclusion is only a sampling result under a particular corpus and a particular set of judge preferences.

There are four structural reasons why such comparisons often diverge:

• Differences in corpus distribution: Different models have different proportions of Chinese training data and different genre coverage, such as news, literature, spoken language, official documents, and code comments. As a result, their capability curves naturally differ.
• Differences in alignment strategies: The output style reinforced during preference optimization directly affects whether text is concise and restrained or elaborate and expansive. These styles may receive opposite scores from different readers.
• High task coupling: The same model may perform very differently on long-form structure, short-text polishing, terminology translation, and spoken dialogue. A single aggregate score inevitably loses information.
• Subjectivity cannot be eliminated: Judgments such as “natural” and “compelling” operate at the pragmatic level and depend on reader age, industry background, dialect exposure, and aesthetic preference. They cannot be fully objectified by a single metric.

Therefore, a more engineering-relevant question is: For my specific business content and writing standards, how can I construct a reproducible Chinese expression evaluation scheme? The rest of this article addresses that question.

Breaking “Expression Ability” into Scoreable Dimensions

A vague judgment such as “the Chinese is good” must be decomposed into independent, observable dimensions. Otherwise, scoring noise will drown out real differences. A practical six-dimension framework is as follows:

1. Language fluency: Whether sentence-length distribution resembles human writing, whether translationese exists, such as overuse of redundant constructions, whether function words and conjunctions are used naturally, and whether punctuation and full-width/half-width conventions are correct.
2. Document structure: Whether paragraphs have clear primary and secondary levels, whether long-form text has a clear general-to-specific structure, whether transitions are natural, and whether there is repetition or topic drift.
3. Style adaptation: Whether the model can switch registers according to genre, such as official documents, technical documentation, marketing copy, conversational speech, or academic abstracts. This includes address forms, tone strength, and terminology density.
4. Pragmatic appropriateness: Whether forms of address, honorifics, and politeness markers are appropriate, whether indirect expression, refusal, correction, and other communicative behaviors follow Chinese-language conventions, and whether responses sound unnaturally translated.
5. Factual consistency: Whether information remains correct while the language stays fluent. This includes consistency of numbers, units, dates, person names, organization names, and whether long-form content avoids contradictions in earlier and later settings.
6. Instruction following: The satisfaction rate of hard constraints, such as word count, heading levels, list format, prohibited words, and output language, as well as whether format and content do not sacrifice each other.

Among these six dimensions, the first four are more subjective, while the last two are more automatable. The key principle in evaluation design is: do not leave deterministic judgments to humans, and focus human review on items that truly require human judgment.

Designing the Evaluation Set: Stratification, Anti-Memorization, and Difficulty Control

The credibility of an evaluation conclusion is bounded by the quality of the evaluation set. Several steps must be handled carefully.

Stratify sampling by genre. Sample allocation should reflect actual usage ratios rather than equal distribution. For example, if a content platform uses technical documentation, operational copy, and customer support responses in a ratio of 52, the evaluation set should follow that ratio. Otherwise, the evaluation may not match real-world experience.

Stratify by difficulty. Within the same genre, tasks can be divided into three levels:

• Regular tasks that can be generated directly;
• Constrained tasks with complex formatting or word-count requirements;
• Boundary tasks involving ambiguous instructions, cases requiring refusal, or prompts containing incorrect assumptions that must be corrected.

High discriminative power often appears in the boundary layer.

Reduce memorization effects. If evaluation prompts come from public datasets, models may have already seen the answers. Possible mitigation methods include rewriting prompt phrasing, replacing domain entities with synthetic entities, injecting a custom business glossary, and periodically rotating prompts.

# Anti-memorization: entity replacement and prompt rewriting
ENTITY_MAP = {"East China Region": "Linjiang District", "Product A": "Qiming Module"}
def sanitize(prompt: str) -> str:
    for src, dst in ENTITY_MAP.items():
        prompt = prompt.replace(src, dst)
    return prompt

Create context-length tiers. Long-form capability should be treated as a separate tier. Information recall, structural consistency, and coherence under very long inputs are independent capabilities. High performance on short text should not be extrapolated to long text.

Rubrics and Hybrid Scoring

Rubric Anchors Are Required

The reliability of a scoring rubric depends on whether each score level has a concrete description and example. Simply writing “1-5: language fluency” will inevitably lead to inconsistent severity among reviewers. Each dimension should have anchor descriptions for 3 to 5 levels. For example, a 5-point anchor for fluency could be: “No obvious translationese, function words and connectives are used in accordance with Chinese habits, and sentence-length distribution is close to comparable human-written text.” A 3-point anchor could be: “Some expressions are stiff or some sentence patterns are repeated, but readability is not affected.”

Structured Scoring Example

from dataclasses import dataclass
DIMENSIONS = ["fluency", "structure", "style_fit", "pragmatics",
              "consistency", "instruction_following"]
@dataclass
class ScoreItem:
    dimension: str
    score: int          # Score from 1 to 5
    reason: str         # Must provide locatable evidence
    evidence_span: str  # Quoted fragment from the generated text
def aggregate(items: list[ScoreItem], weights: dict[str, float]) -> dict:
    total, detail = 0.0, {}
    for item in items:
        v = item.score * weights[item.dimension]
        detail[item.dimension] = round(v, 3)
        total += v
    return {"total": round(total, 3), "detail": detail}

Weights should be set according to business objectives rather than evenly distributed by default. For example, in a customer support scenario, pragmatic appropriateness and instruction following should carry significantly more weight than literary refinement. For technical documentation, structural quality and terminology consistency are more important. The weights themselves should be calibrated through a small-scale manual annotation process so that the weighted total score maximizes correlation with human preference rankings.

Bias and Calibration in Model-Assisted Scoring

Using a model as a judge can reduce labor costs, but it introduces known biases:

• A preference for longer answers;
• A preference for outputs that resemble the judge model’s own style;
• Position bias when comparing two outputs side by side;
• Confusion between formatting elegance and content quality.

Necessary mitigation measures include:

• Anonymize candidate outputs, randomize order, and swap A/B positions to compare results;
• Require the model to output an evidence sentence; high scores without locatable evidence should be considered invalid;
• Introduce a human-annotated calibration subset and measure agreement between the model judge and humans; if agreement is insufficient, revise the prompt or change how the dimension is decomposed;
• Use rule-based checks instead of model judgment for tasks with hard constraints, such as word count and format.

What Automatic Metrics Can and Cannot Do

• N-gram overlap metrics, such as exact-match overlap or edit distance, are heavily affected by Chinese tokenization granularity and are insensitive to expression differences caused by synonym rewriting. They are suitable mainly for detecting whether a model copied a reference answer or for regression comparison, not for judging writing quality.
• Perplexity-based metrics depend on the reference corpus distribution, are not comparable across domains, and have limited correlation with human preference.
• Format and constraint checks can be fully automated: word-count ranges, heading levels, list markers, prohibited words, full-width/half-width usage, and glossary hits. These checks should be implemented as hard gates.
• Consistency checks can be implemented with rules or lightweight models: number and unit verification, entity consistency, and detection of contradictions between earlier and later parts of long-form text.

The resulting division of labor is: use rules for hard constraints, use calibrated scoring for language quality, and use small-scale human spot checks for subjective preferences.

Engineering Implementation: Pipeline, Consistency, and Regression

The real value of evaluation lies in the ability to run repeatedly, compare versions, and block regressions. A minimum viable pipeline contains five steps:

1. Manage the prompt pool and sample ratio, with versioned storage to ensure reproducibility.
2. Generate candidate outputs with fixed decoding parameters and record a complete parameter snapshot.
3. Perform rule checks and automatic scoring.
4. Conduct human spot checks and measure consistency.
5. Generate reports and compare against baselines.

Scoring consistency must be quantified; otherwise, it is impossible to determine whether differences are real. When multiple people score the same items, use an inter-rater consistency coefficient. For ordinal comparisons, use rank correlation:

def kendall_tau(x: list[float], y: list[float]) -> float:
    n = len(x)
    concordant = discordant = 0
    for i in range(n):
        for j in range(i + 1, n):
            sign = (x[i] - x[j]) * (y[i] - y[j])
            if sign > 0:
                concordant += 1
            elif sign < 0:
                discordant += 1
    denom = n * (n - 1) / 2
    return (concordant - discordant) / denom

A common engineering practice is: if the rank correlation between the model judge and human annotators falls below an acceptable threshold, the automatic score for that dimension should not be trusted. If human scoring consistency is low, the priority is to rewrite anchor descriptions rather than simply increasing the number of reviewers.

For regression, a baseline snapshot and alert threshold should be defined. If the mean score of any dimension drops beyond a preset margin relative to the baseline, release should be blocked. A regression sample set should also be maintained: low-quality outputs discovered in each round should be added to the prompt pool to prevent similar problems from recurring.

Recommendations for Model Selection and Continuous Optimization

• Build multiple baselines by scenario rather than maintaining a single leaderboard. Long-form writing, format-constrained structured output, and conversational dialogue should each have independent evaluation sets.
• Keep decoding parameters fixed. Temperature and sampling strategy can significantly affect Chinese writing style. Comparisons made without fixed parameters are not meaningful.
• Control cost. The evaluation pipeline’s request volume, log retention, and metric storage can rely on general-purpose logging services, monitoring and alerting systems, and object storage components. Connecting evaluation results with production metrics is necessary to validate offline conclusions.
• Distinguish capability problems from prompt problems. Many performance differences come from insufficient prompt engineering rather than the model itself. Before comparing models, standardized Chinese writing prompt templates should be unified. These templates should specify reader, genre, length, tone, and prohibited expressions.
• Accept that conclusions are time-sensitive. Models iterate quickly. Any ranking holds only for a specific model version, evaluation set, and point in time. Therefore, evaluation sets and version numbers must be recorded together.

In summary, there is no task-independent answer to the question of which model has the best Chinese expression ability. What can be built is a combination of a stratified evaluation set, anchored rubrics, hybrid scoring mechanisms, and a regression pipeline. Once subjective impressions are converted into a reproducible process, the question “Which one is more suitable?” shifts from an argument based on impression to a decision based on data.