LLM Chinese-Language Quality Evaluation: A Practical Engineering Guide

1. Why Chinese-Language Expression Needs Dedicated Evaluation

When selecting a large language model, many teams first look at general-purpose benchmark scores. However, as soon as the model is applied to real business scenarios such as Chinese content creation, copywriting, dialogue rewriting, summarization, polishing, brand messaging, or script dialogue, teams often discover that a high general-purpose score does not necessarily mean strong Chinese writing ability.

Chinese has clear characteristics that affect expression quality, including word segmentation, idiomatic expressions, classical references, tone and rhythm, parallel sentence structures, discourse cohesion, cultural context, and reader expectations. A model with strong English capability may still produce unnatural Chinese. Similarly, a model that performs steadily on general question answering may not produce text that conforms to scriptwriting conventions or formal document styles.

Therefore, evaluating Chinese-language expression in a large model cannot rely on a single broad score. The evaluation needs to answer several practical questions:

• How should Chinese expression be evaluated?
• What common biases appear in human scoring?
• What common errors appear in evaluation datasets?
• Which tasks are suitable for automatic metrics?
• How can LLM-as-a-judge scoring be calibrated?

These questions determine whether the evaluation system can genuinely support model selection, version iteration, and business acceptance.

From a business perspective, content teams usually care about whether vocabulary is precise, whether dialogue feels natural, whether the plot is coherent, whether the register matches the requirement, whether long-form text remains logically consistent, and whether cultural expression feels idiomatic. Engineering teams need to turn these subjective impressions into measurable, reproducible, and iterable methods. This article provides a practical Chinese-language expression evaluation approach for that purpose.

2. Breaking Down “Chinese Expression” into Measurable Dimensions and Scoring Anchors

It is difficult to operate on a vague judgment such as “Is the expression good or bad?” The first step is dimension decomposition. A reasonably complete Chinese-language expression evaluation framework usually includes vocabulary, grammar, discourse, register, cultural context, and creativity.

To avoid inconsistent scoring across reviewers, it is recommended to use a unified 1-5 integer scale:

• 1 means clearly unacceptable.
• 3 means basically usable.
• 5 means excellent expression.

If a business team prefers a 0-10 scale, normalization must be completed before weighted aggregation.

Dimension	Observable scoring anchors	1-point behavior	3-point behavior	5-point behavior
Lexical richness and accuracy	Wrong words, repetition, empty phrasing, terminology misuse, unnatural collocations	Many wrong words and obvious repetitive empty phrasing	Basically accurate with minor repetition	Precise vocabulary and natural, rich expression
Grammar and syntactic fluency	Broken sentences, missing components, disordered syntax, readability of long sentences	Multiple grammatical errors and difficult to understand	Minor grammatical issues without affecting understanding	Smooth sentence patterns and natural rhythm
Semantic coherence and discourse logic	Paragraph transitions, contradictory information, clarity of the main thread, consistency of conclusions	Contradictory and broken main thread	Basically coherent with local jumps	Clear logic and natural transitions
Register and style adaptation	Oral style, written style, official document style, script style, poetry, marketing copy, and other style requirements	Style clearly deviates	Mostly aligned but mixed	Highly aligned with the target style
Cultural context and localized expression	Idioms, colloquialisms, internet language, Chinese reader expectations, real-world context	Stiff or misused expression	Basically follows conventions	Idiomatic and natural with a local feel
Creativity	Imagination, originality, avoidance of clichés, novelty	Mechanical copying and severe templating	Some novelty but unstable	Novel expression with reasonable logic

A special note is needed for “diversity.” If only one model output is available, a single-output score is more suitable for measuring creativity than diversity. Diversity usually requires multiple candidates or multiple samples. It is recommended to generate several candidates for the same prompt, such as 3 to 5 outputs, and then calculate additional indicators:

1. Repetition rate: whether candidates contain high-frequency repeated phrases, identical sentence patterns, or highly similar expressions.
2. Semantic diversity: vector similarity or semantic distance between candidates. Greater distance usually indicates higher diversity.
3. Ranking stability: whether the relative ranking of candidates remains stable across multiple scoring rounds, avoiding distorted conclusions caused by a one-off fluctuation.

Therefore, for single-output scoring, it is recommended to measure creativity only. For tasks that require diversity evaluation, multi-candidate indicators should be added separately. Diversity should not be forced into a single-output subjective score.

The value of dimension decomposition is that different business scenarios can adjust the weights accordingly. Marketing copy may place more emphasis on lexical accuracy and style adaptation. Long-form scripts may place more emphasis on discourse logic, character consistency, and dialogue naturalness. Summarization and rewriting may place more emphasis on semantic fidelity and conciseness. Official document writing may place more emphasis on style compliance and cultural context accuracy. Weights are not fixed values, but they must be explicitly declared and kept consistent within the same evaluation version.

3. Building a High-Quality Evaluation Set

The evaluation set is the foundation of the entire evaluation system. If the evaluation set has insufficient coverage, severe sample bias, or is repeatedly used for parameter tuning, subsequent conclusions are likely to be distorted.

For initial rapid validation, 50-100 tasks can be used. For formal evaluation, it is recommended to stratify by business type, difficulty, style, and scenario, with at least 20 samples for each core stratum. If the initial sample size is insufficient, core scenarios should be covered first, and the dataset can be expanded gradually.

The evaluation set should be split into a public set and a held-out set. The public set can be used for development, prompt debugging, and rubric refinement. The held-out set should not be used for prompt tuning or scoring parameter adjustment. It should only be used for final acceptance to avoid evaluation set contamination.

The following is a stratified sampling table template:

Business type	Difficulty	Style/scenario	Initial samples	Public set	Held-out set
Summarization	Low	News, reports, long articles	20	14	6
Rewriting and polishing	Medium	Oral-to-written, written-to-oral	20	14	6
Complex instruction generation	High	Multiple constraints, format restrictions, banned words	30	21	9
Open-ended creation	Medium-high	Copywriting, scripts, fiction fragments	30	21	9
Dialogue generation	Medium	Customer service, role-play, plot dialogue	20	14	6

The evaluation set should cover the real business distribution. Creative tasks can include character biographies, episode outlines, dialogue, scene descriptions, and headline copy. Generation tasks can include summarization, rewriting, polishing, expansion, and style transfer. For the same intent, multiple prompt phrasings should be prepared to avoid rewarding a model only because it happens to match a predictable prompt. Complex instructions should also be covered, such as word-count limits, style requirements, banned words, character settings, format constraints, and output structure requirements.

Reference answers should be handled according to task type. For summarization, rewriting, and error correction, reference texts can be prepared as comparison baselines. For open-ended creative tasks, a single standard answer should not be imposed. Instead, scoring points should be clarified, such as whether character motivation is preserved, whether plot progression is completed, whether the required register is followed, and whether factual conflicts exist.

The evaluation set requires version management, including prompt version, rubric version, sample addition/removal records, and annotation change records. After each model upgrade, the same evaluation ruler should be used to retest the model before judging whether capability has improved or declined.

Common errors include:

• Sample size is too small, causing unstable conclusions.
• Only popular prompts are tested.
• Reference answers are too strong, suppressing reasonable creativity.
• Lack of difficulty stratification causes strengths and weaknesses to be masked by averaging.
• The evaluation set is repeatedly inspected, leading to leakage.

It is recommended to establish a deduplication mechanism and clean samples with semantically duplicated prompts, highly similar outputs, or severe templating.

4. Scoring Methods: Combining Human Review, Automatic Metrics, and Model Review

Scoring methods are usually divided into three categories: human scoring, automatic metrics, and model-based review. In practice, a combined approach is often used.

All scores must be normalized to a common scale before entering the aggregation layer. It is recommended to use a 1-5 integer score for each dimension. During weighted aggregation, normalize to a 0-1 range using:

(score - 1) / 4

If a 0-10 scale is used, normalize using:

score / 10

This makes scores comparable across different reviewers, scoring tables, and model outputs.

Human scoring is performed by people familiar with the business, who assign scores by dimension. Its advantage is that it is close to real-world perception. Its disadvantage is high cost and subjective variance. A clear rubric should be defined, and multiple reviewers should cross-score the same samples. Reviewer agreement should be calculated, for example using Krippendorff’s alpha or Cohen’s kappa. If agreement is too low, the scoring anchors may not be clear enough, and reviewer training or rubric revision is needed.

Automatic metrics include lexical similarity, semantic similarity, and fluency-related indicators. Before calculation, the Chinese tokenization method must be clarified, such as word-level, character-level, or subword-level segmentation. Different tokenization choices can significantly affect metric results.

ROUGE-L is more suitable for tasks with reference texts, such as summarization, rewriting, and polishing. BERTScore is more suitable for measuring semantic similarity. Perplexity can be used to roughly observe fluency, but it is affected by tokenization, corpus domain, and the autoregressive characteristics of the model.

The advantage of automatic metrics is low cost and batch execution. The disadvantage is limited discriminative power for open-ended creative writing. They cannot determine whether characters feel vivid, whether the plot has tension, or whether cultural expression is fully natural. Therefore, for open-ended creative tasks, automatic metrics should only be used as auxiliary signals.

Model-based review, also known as LLM-as-a-judge, can significantly reduce human review cost, but the judge model must be calibrated.

It is recommended to prepare at least 200 human-annotated samples as the calibration set. For high-risk business scenarios, the calibration set can be expanded to 300-500 samples. Calibration samples should use stratified random sampling and cover different business types, difficulty levels, styles, lengths, and prompt variants. The average absolute error, correlation coefficient, or rank correlation coefficient between model review scores and human scores should be calculated.

A reasonable experience baseline is:

• On a 1-5 scale, MAE should be no greater than 0.5.
• Spearman or Pearson correlation coefficient should be no lower than 0.7.

However, these thresholds are not universal standards and must be judged according to the specific task.

Applicable conditions include:

• The calibration set contains at least 200 samples.
• The task type is relatively closed, such as summarization, rewriting, polishing, or constrained creation.
• Human re-review uses the same scoring table and scoring anchors.
• Review results are based on double-blind review or cross-review.
• Confidence intervals or sampling error are reported when presenting results.

If the calibration set has fewer than 200 samples, or the task is highly open-ended creative writing, these thresholds should be used cautiously, and small-sample human re-review should be prioritized.

If the calibration target cannot be reached, the evaluation prompt should be adjusted, scoring anchors should be added, ambiguous expressions should be reduced, or the task should be returned to human re-review.

Common biases in model review include:

• Position bias: preferring candidates that appear earlier or later.
• Length bias: preferring longer or more complex outputs.
• Self-preference bias: preferring text similar to the judge model’s own style.
• Style bias: preferring ornate or forceful expression.
• Difficulty bias: unstable scoring on complex tasks.

Recommended mitigation measures include random ordering, pairwise comparison, fixed scoring templates, sampled re-review, and bias monitoring.

A possible LLM judge prompt template is as follows:

You are an expert reviewer of Chinese-language expression. Please score the model output according to the given scoring anchors.
Score range: integers from 1 to 5.
Dimensions: lexical accuracy, grammar and fluency, semantic coherence, style suitability, cultural context and localization, creativity.
Scoring requirements:
1. Output JSON only. Do not output explanations.
2. Every dimension must be an integer from 1 to 5.
3. Do not increase the score because the output is longer.
4. Do not increase the score because the output is more ornate.
5. If there are clear factual conflicts or cultural misuse, lower the score in the corresponding dimension.
Output format:
{"lexical_accuracy": 1-5, "grammar_fluency": 1-5, "semantic_coherence": 1-5, "style_suitability": 1-5, "cultural_context_localization": 1-5, "creativity": 1-5}

The following is an example of score aggregation code. Note that diversity requires multi-candidate indicators and should not be forced into single-output scoring.

from statistics import mean
WEIGHTS = {
    "lexical_accuracy": 0.15,
    "grammar_fluency": 0.15,
    "semantic_coherence": 0.15,
    "style_suitability": 0.15,
    "cultural_context_localization": 0.20,
    "creativity": 0.20,
}
MIN_SCORE = 1.0
MAX_SCORE = 5.0
def validate_score(value):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise TypeError("judge_fn must return a numeric score")
    if value < MIN_SCORE or value > MAX_SCORE:
        raise ValueError("Score must be within the 1-5 range")
    return float(value)
def normalize_score(value):
    value = validate_score(value)
    return (value - MIN_SCORE) / (MAX_SCORE - MIN_SCORE)
def score_one(item, model_output, judge_fn):
    raw = {}
    for dim in WEIGHTS:
        raw[dim] = validate_score(
            judge_fn(item["prompt"], model_output, dim, item.get("reference"))
        )
    weighted = sum(normalize_score(raw[d]) * WEIGHTS[d] for d in WEIGHTS)
    return {
        "dimensions": raw,
        "weighted_score": round(weighted, 4),
    }
def evaluate_dataset(dataset, model_call, judge_fn):
    results = []
    for item in dataset:
        output = model_call(item["prompt"])
        results.append(score_one(item, output, judge_fn))
    return {
        "count": len(results),
        "avg_score": mean(r["weighted_score"] for r in results),
        "details": results,
    }

The key points of this code are:

• judge_fn must return scores in the same scale.
• Scores must be validated against the allowed range.
• Normalization must be completed before weighted aggregation.
• Dimension-level details should be retained to make issue localization easier.

If the weighted total score drops while a specific dimension rises, the weight configuration, score scale, or dimension definitions should be checked for conflicts.

5. Additional Evaluation for Creative Writing Scenarios

For long-form creative writing such as scripts, novels, short dramas, and brand stories, general dimensions are not enough. Additional attention should be paid to character consistency, plot tension, dialogue naturalness, and long-range dependencies.

Character consistency can be checked using a small-sample, high-density method. For each character, sample 3-5 dialogue lines and action descriptions, then cross-check whether personality, motivation, and speaking style remain stable. If a character is rational and calm in earlier scenes but suddenly becomes irritable later without motivation or event support, the character should be judged as inconsistent. This can be quantified as a character fact consistency rate, such as the proportion of character settings, relationships, abilities, and motivations that remain consistent with the original text or previous context.

Plot tension evaluation should focus on conflict, suspense, reversal, and pacing. During scoring, check whether:

• The conflict is clear.
• The goal is clear.
• Obstacles escalate.
• Reversals are foreshadowed.
• The ending resolves the main thread.

Dialogue naturalness should assess whether the lines match the character’s identity, age, relationship, and scene, and whether they sound like a person speaking rather than mechanical exposition of settings.

Long-range dependency evaluation is used for multi-turn or multi-chapter generation scenarios. The focus is whether the model can remember early settings and maintain them consistently in later content.

The following quantitative fields are recommended:

1. Early-setting hit rate: whether later content correctly reuses earlier settings.
2. Character fact consistency rate: whether character identity, relationships, abilities, and motivations remain consistent.
3. Foreshadowing payoff rate: whether clues planted earlier receive reasonable responses later.
4. Dialogue humanness score: whether lines are natural and match the character’s tone.
5. Long-range consistency score: whether obvious breaks occur across multiple chapters or dialogue turns.

A deep review table can include the following fields:

• Task ID
• Model version
• Character ID
• Sampled fragment
• Character consistency judgment
• Plot tension judgment
• Dialogue naturalness judgment
• Long-range dependency judgment
• Main issues
• Recommendation
• Revision suggestions

This converts creative evaluation from “feeling-based” judgment into traceable, discussable, and reusable review records.

6. Engineering the Evaluation Workflow

Evaluation should not be a one-off activity. It should be accumulated into a sustainable process.

The workflow can be divided into six steps:

1. Configuration freeze
2. Task submission
3. Asynchronous execution
4. Result validation
5. Report generation
6. Retrospective and archiving

Configuration freeze includes fixing prompts, scoring rubrics, random seeds, model versions, and evaluation set versions.

After task submission, tasks enter a message queue and execute asynchronously to avoid large-scale evaluation blocking the main workflow. Results are written to object storage. Log services retain review records. Monitoring and alerting track failure rates, latency, and cost anomalies.

A recommended engineering checklist includes:

• Model version
• Evaluation set version
• Prompt version
• Score scale
• Weight configuration
• Random seed
• Failure retry count
• Rate-limit threshold
• Timeout duration
• Abnormal sample persistence
• Cost estimation
• Least-privilege access control
• Log desensitization
• Held-out set isolation
• Reproducible results

For large-scale evaluation, failure retries and rate limiting should be configured to avoid cost overruns caused by high-frequency calls. For content involving user data or creative materials, desensitization should be applied, and the least-privilege principle should be followed.

For infrastructure selection, container platforms can be used to run evaluation tasks. Message queues can implement asynchronous dispatch. Object storage can store batch results. Log services can retain review records. Monitoring and alerting can track failure rates and latency. This allows teams to focus on evaluation set design, scoring anchor refinement, and model comparison instead of being blocked by infrastructure work.

7. Common Pitfalls and Troubleshooting Suggestions

First, do not rely only on general leaderboards for Chinese-language model selection. Leaderboards can be used as references, but they must be supplemented by Chinese business evaluation sets.

Second, do not let reference answers suppress reasonable creativity. Open-ended creation should be evaluated with scoring points rather than a single standard answer.

Third, do not replace human judgment with automatic metrics. ROUGE-L, BERTScore, and similar metrics are suitable for closed tasks, but they are not sufficient for judging creative quality alone.

Fourth, do not use LLM judge scoring without calibration. Model-based review must be calibrated with human annotations and continuously monitored for bias.

Fifth, do not repeatedly use the held-out set for parameter tuning. Once the held-out set is used for optimization, it loses its acceptance value.

Sixth, do not ignore cultural context. Chinese expression is not only grammatically correct. It also depends on whether idioms, colloquialisms, context, reader habits, and local cognition feel natural.

8. Conclusion

There is no universal answer to the question of which model has the best Chinese-language expression. A more reliable approach is to return to the actual business scenario and use an evaluation system that is clearly decomposed, quantifiable, and reproducible.

Dimension decomposition makes evaluation evidence-based. Scoring anchors allow different reviewers to align on standards. Evaluation sets ensure that conclusions are credible. Combining human review with model-based review balances efficiency and real-world perception. Engineering practices allow evaluation to iterate continuously.

When model selection no longer depends on vague impressions but is built on a stable and verifiable evaluation ruler, teams can make more reliable technical decisions in creative and generation-related businesses.