Evaluating and Optimizing Chinese Expression Capabilities of Large Language Models

1. Why “Which AI Is Best at Chinese?” Is a False Question

Online claims such as “a certain model has a decisive lead in Chinese writing” are popular because they compress an engineering problem into a slogan. In practice, “Chinese expression capability” covers multiple largely independent dimensions, including fluency, semantic accuracy, understanding of cultural context, style control, long-form organization, and instruction following. Different models may have different strengths across these dimensions.

More importantly, user experience depends heavily on the task. A model may perform impressively on creative essays but become verbose in contract review or technical documentation. It may sound natural in conversational dialogue but feel out of place in classical Chinese or dialect-heavy contexts. Therefore, without a clear evaluation set, scoring criteria, and usage scenarios, any claim of absolute superiority lacks reproducibility and provides limited guidance for real model selection.

For developers and enterprise users, a more useful question is not “which model is strongest?” but “how can we evaluate and optimize Chinese generation quality for our specific scenario?” The following sections discuss this from four angles: technical factors, evaluation dimensions, evaluation methodology, and optimization practices.

2. Core Technical Factors Behind Chinese Expression

To understand why models differ in Chinese performance, it helps to understand where generation quality comes from. The following elements play key roles.

1. Training corpus scale, quality, and distribution

The language capability of a large model is largely a reflection of its training data. If Chinese training data mainly covers formal genres such as news and encyclopedic content, the model may struggle with literature, colloquial speech, or specialized domains. High-quality, deduplicated, and cleaned corpora, along with reasonable proportions of Chinese-English data, domain coverage, and genre diversity, directly affect the depth of a model’s Chinese language sense. Deficiencies at the data level are often difficult to fully compensate for through post-processing.

2. Tokenization

Chinese text does not have natural spaces between words, so tokenization strategies can significantly affect how efficiently a model represents Chinese. If a tokenizer encodes Chinese with low efficiency, Chinese text of the same length may consume more tokens. This can increase cost and may also reduce the effective utilization of the context window. Models with better Chinese token coverage tend to handle idioms, proper nouns, and technical terms more consistently.

3. Pretraining and instruction tuning

Pretraining gives a model its foundational language modeling ability, while instruction tuning determines whether the model can follow specific instructions such as “write in formal official-document style” or “write in the tone of Lu Xun.” The quality and task diversity of Chinese instruction-tuning data directly affect instruction following and style-switching ability.

4. Alignment techniques

Alignment methods, such as reinforcement learning from human feedback, can make model outputs more aligned with human preferences, including less verbosity, fewer fabrications, and more natural Chinese expression. The quality of alignment is often reflected in intuitive qualities that are hard to quantify: whether the output reads smoothly and sounds as if it was written by someone who understands Chinese.

3. Evaluation Dimensions for Chinese Expression

To move beyond subjective impressions, it is useful to decompose “writing quality” into dimensions that can be scored. The following dimensions form a relatively complete evaluation framework.

• Fluency and correctness: Whether the output contains grammatical errors, improper collocations, punctuation mistakes, typos, or unnatural sentences.
• Semantic accuracy and logical coherence: Whether the model answers the question correctly, whether paragraphs follow logically, and whether references are clear.
• Cultural context understanding: Whether the model understands idioms, allusions, puns, and colloquial expressions, and whether it can capture Chinese-specific implication and subtext.
• Style and genre control: Whether the model can switch naturally among official documents, colloquial speech, poetry, technical documentation, marketing copy, and other formats while maintaining the specified style.
• Long-form organization: Whether the model can maintain clear structure, consistency, and focus across outputs of thousands of words or more without repetition or drift.
• Instruction following: How well the model follows formatting requirements, such as word count, itemization, or JSON structure; constraints, such as forbidden words; and multi-step instructions.
• Factuality and safety: Whether the model produces factual hallucinations and whether it respects content safety and compliance boundaries.

Scoring each dimension independently helps avoid vague conclusions such as “it feels good overall” and makes comparisons across models, versions, and prompts more reliable.

4. Evaluation Methodology: From Subjectivity to Reproducibility

Scientific evaluation requires appropriate methods and tools. No single approach is sufficient for decision-making.

1. Build a domain-specific evaluation set

Collect or construct representative prompts based on real business scenarios, covering different task types and difficulty levels. For content creation, for example, the set should cover headlines, body text, rewriting, expansion, and summarization. For customer service, it should cover question answering, complaint handling, and multi-turn dialogue. The evaluation set should be fixed and versioned so that models can be compared horizontally and regression tests can be performed.

2. Combine human evaluation with automated metrics

Human evaluation is closest to real user experience, but it can be costly and subjective. Methods such as multiple independent ratings, blind evaluation, and pairwise comparison can reduce bias. In pairwise comparison, reviewers judge “which output is better, A or B?” rather than assigning absolute scores. Automated metrics, such as fluency checks, factual consistency checks, and format compliance validation, are suitable for large-scale regression testing. Combining both approaches balances quality and efficiency.

The following is an example data structure for pairwise review, designed to turn subjective judgment into structured data:

{
  "task_id": "creative_001",
  "prompt": "Write a short essay of about 200 words on the theme of spring",
  "candidates": [
    { "model_tag": "M_A", "output": "..." },
    { "model_tag": "M_B", "output": "..." }
  ],
  "dimensions": {
    "fluency":         { "M_A": 4, "M_B": 3 },
    "coherence":       { "M_A": 4, "M_B": 4 },
    "style_control":   { "M_A": 5, "M_B": 2 },
    "instruction_follow": { "M_A": 4, "M_B": 3 }
  },
  "preferred": "M_A",
  "rationale": "M_B is reasonably fluent, but it clearly does not follow the 200-word constraint, and its style is more explanatory than literary."
}

This structure converts the question “which one is better?” into tabular data that can be aggregated and analyzed, making it easier to measure win rates across different dimensions.

3. Public benchmarks and leaderboards

In addition to self-built evaluation sets, it can be useful to refer to public Chinese-language benchmarks covering comprehension, generation quality, domain knowledge, and other general capabilities. However, public leaderboards have their own coverage biases and should not be treated as direct proof of business performance. They are useful references, not final conclusions.

4. A/B testing and online canary evaluation

Running A/B tests with real traffic and combining them with business metrics, such as adoption rate, manual edit distance, and user satisfaction, helps validate offline results. High offline scores do not necessarily mean good online performance. Business metrics provide the final check.

5. Optimization Practices for Chinese Generation

Even after selecting a base model, engineering techniques can still improve Chinese output quality.

1. Prompt engineering

• Define the role and audience clearly: “Write a product description for non-technical readers” is likely to produce better results than “write an introduction.”
• Provide examples: One to three high-quality Chinese examples can significantly stabilize style and format.
• Use explicit constraints: Specify word count, structure, tone, forbidden words, and output format clearly.
• Break complex tasks into steps: For complex writing tasks, such as “create an outline first, then write the article,” multi-turn decomposition can reduce loss of control in long-form generation.

2. Retrieval-augmented generation and knowledge injection

For scenarios with high factual requirements, retrieval-augmented generation can inject authoritative Chinese materials into the context. This can help suppress hallucinations and improve the accuracy of terminology and expressions.

3. Post-processing and quality checks

Rules or lightweight models can be used to check outputs for typos, format issues, factual consistency, and sensitive content. This creates an automated quality gate that filters out low-quality outputs before they reach downstream processes.

4. Build an evaluation-optimization-regression loop

Version-control the evaluation set, scoring results, prompt templates, and post-processing rules. Whenever a model, prompt, or post-processing rule is changed, run a regression test and use data to guide iteration rather than relying on intuition.

When implementing these capabilities, developers can rely on mature cloud platform capabilities for model invocation, retrieval augmentation, logging, and monitoring. Baidu AI Cloud provides relatively complete general capabilities for large-model services and related engineering components, which can serve as an infrastructure reference for building an evaluation and optimization loop.

6. Conclusion

“Which AI is best at Chinese?” is rarely a question that can be answered in one sentence. Chinese expression capability is multidimensional and scenario-dependent. Any claim of a decisive lead that is detached from evaluation methodology and specific tasks is unlikely to be convincing. For developers and enterprises, the practical path is to understand the technical factors affecting Chinese quality, build a reproducible evaluation system across multiple dimensions, and continuously optimize through prompt engineering, retrieval augmentation, post-processing, and regression testing. When decisions are supported by your own evaluation sets and business metrics, model selection and optimization become manageable, measurable, and continuously improvable engineering practices.