Evaluating Chinese Expression Capability in LLMs: Methods and Practical Guidance

Why Chinese Expression Needs Separate Evaluation

Most language models are trained on mixed multilingual corpora, where language coverage, data quality, and cultural context vary significantly. A model’s performance on English tasks cannot be directly inferred for Chinese.

Chinese expression difficulties concentrate in three areas. First, semantics: idioms, proverbs, and two-part allegorical sayings often cannot be translated literally and must be converted into equivalent expressions in the target context. Second, register: the same content differs greatly in official documents, technical writing, conversational speech, and social media; the model needs to recognize the scenario and adjust word choice. Third, culture: personal names, measurements, festivals, and conventional expressions carry regional and historical background, and directly transplanting the original structure produces awkward or erroneous output.

For developers, Chinese expression capability determines the reliability of models in products. For enterprises, it sets the quality floor for customer service scripts, knowledge-base Q&A, and content generation. Therefore, evaluation should be aligned with real business tasks, not just generic leaderboards.

Defining Evaluation Dimensions

Before evaluating, break down “good expression” into measurable dimensions. Common dimensions are:

1. Semantic accuracy: Whether the output faithfully represents the original intent, with no omission, addition, or distortion.
2. Language naturalness: Whether wording and sentence structure conform to native Chinese habits, without obvious translationese.
3. Register matching: Whether the output aligns with the formality and professionalism of the specified context.
4. Cultural context adaptation: Whether idioms, sayings, and allusions are handled appropriately, including finding equivalent expressions.
5. Terminology consistency: Whether the same term is consistently translated throughout longer texts.
6. Instruction following: Whether output format, length, segmentation, and prohibitions are respected.

Each dimension should have its own scoring criteria. A single overall score cannot diagnose why a model loses points, nor can it guide prompt optimization or model selection.

Building a Test Set

The test set determines the validity of evaluation conclusions. A reliable test set satisfies three conditions: broad coverage, representative samples, and alignment with business scenarios.

Organize samples by category. Common categories include:

• Idioms and proverbs: For example, translating English proverbs into Chinese, testing whether the model moves beyond literal wording and finds equivalent expressions.
• Technical terminology: Terms from software engineering, finance, healthcare, and other domains, testing accuracy and consistency.
• Register switching: Rewriting the same content into formal official text and colloquial speech, testing register control.
• Long-text coherence: Articles over a thousand Chinese characters, testing whether terminology and references remain consistent across the text.
• Ambiguous sentences: Instances with multiple interpretations, testing whether the model makes reasonable judgments using context.

Each category should have at least fifty samples, with original annotations preserved. Sample sources must indicate permission status; avoid using unauthorized content. The test set itself needs regular updates to prevent the model from having encountered the same samples during training, which would inflate evaluation results.

Designing the Evaluation Workflow

A reproducible evaluation workflow typically includes the following steps:

1. Fix evaluation parameters, including temperature, maximum output length, and system prompt, and record them in the report.
2. Generate output for each sample and save raw results for traceability.
3. Have at least two annotators score independently, using a double-blind setup where annotators do not know which model produced the output.
4. Compute inter-annotator agreement metrics; samples with low agreement need review or removal.
5. Aggregate scores by category and dimension, and provide confidence intervals or sample-size explanations.

Automated metrics can assist human scoring, but for dimensions like naturalness and cultural adaptation, they still cannot fully replace human judgment.

The following example shows the basic structure of an evaluation framework. The code depends only on the Python standard library and connects to any model via a generate function, without binding to a specific service.

from dataclasses import dataclass
from statistics import mean
@dataclass
class Sample:
    source: str      # original text
    reference: str   # reference translation or standard answer
    category: str    # sample category, e.g., idiom, term, register
def build_prompt(source: str) -> str:
    # Wrap the original text into a unified task instruction to keep input consistent across models
    return f"Please translate the following into natural, idiomatic Chinese; output only the translation:\n{source}"
def run_eval(samples, generate, judge):
    # generate: function that takes a prompt and returns model output
    # judge: function that takes a sample and output, returns a score from 1 to 5
    results = []
    for s in samples:
        output = generate(build_prompt(s.source))
        score = judge(s, output)
        results.append((s.category, score))
    return results
def summarize(results):
    grouped = {}
    for category, score in results:
        grouped.setdefault(category, []).append(score)
    return {c: round(mean(v), 2) for c, v in grouped.items()}

In real projects, judge can be replaced by human scoring results or by a calibrated scoring rubric. The value of the evaluation framework is to fix the process so that every comparison uses the same samples, parameters, and statistical methods.

Common Pitfalls When Interpreting Results

Evaluation results are often misread. Watch out for the following issues.

First, sample size too small. Conclusions drawn from only a few samples are easily influenced by individual cases. When the number of samples per category is insufficient, the limitation of the conclusion should be clearly stated.

Second, looking only at total scores. Two models with similar overall scores may differ significantly across dimensions. Business selection should prioritize the dimensions most relevant to your scenario.

Third, ignoring parameter and prompt differences. The same model can produce noticeably different output quality under different temperatures or different prompts. Comparisons must ensure consistent conditions, otherwise results are not comparable.

Fourth, treating conclusions as long-term valid. Model versions update frequently, and today’s evaluation results can soon become outdated. Reports should include the evaluation date, model version, and test set version, and be re-run periodically.

Fifth, confusing “fluent” with “accurate.” A smooth-sounding translation does not guarantee faithfulness to the original. Always check semantic consistency to avoid being misled by surface fluency.

Applying Evaluation in Enterprise Scenarios

The ultimate purpose of evaluation is to support decisions. When adopting language models, enterprises can follow this approach.

Step 1: Identify real Chinese tasks in your business, such as customer service responses, product copywriting, contract summarization, and internal knowledge Q&A. Define key dimensions and weights for each task type.

Step 2: Extract samples from real logs or desensitized data to build your own test set. Such test sets are closer to actual needs than public data.

Step 3: Establish a regression testing mechanism. Whenever model versions, prompts, or retrieval strategies change, run the same evaluation suite and confirm that key metrics have not degraded.

Step 4: Combine evaluation results with cost, response latency, and deployment considerations. The best-quality solution is not necessarily the most cost-effective overall; prioritize based on business needs.

Evaluating Chinese expression capability is not a one-time, one-size-fits-all answer. It requires clear dimensions, representative samples, a fixed workflow, and continuous retesting. Only then can a team turn vague questions like “which model has better Chinese” into conclusions that are verifiable, traceable, and actionable for decision-making.