Why LLM Responses Are Often Weaker in Chinese Than English: Causes and Optimization

1. The phenomenon: why LLM answers in Chinese are often weaker than in English

In real development work, many engineers encounter a similar issue: when the same requirement is submitted to the same large language model in Chinese and in English, the English response is often better in factual accuracy, structural completeness, fluency, and instruction following.

Many teams simply attribute this to the model “preferring English” or “not liking Chinese.” From an engineering perspective, this explanation is not precise.

A more reasonable explanation is that a quality gap between Chinese and English responses may indeed exist, but it comes from multiple technical layers, including training data, tokenization, data-cleaning rules, and post-training alignment strategies. Understanding the causes of this gap not only helps explain the phenomenon, but also helps developers design more targeted optimization methods, including prompt engineering, retrieval-augmented generation, data governance, model selection, and evaluation system design.

The goal of this article is not to create anxiety. Instead, it provides a verifiable, quantifiable, and executable path for diagnosing and improving LLM performance in Chinese.

2. Root causes: where the Chinese-English response gap comes from

2.1 Differences in corpus scale and distribution

The capability ceiling of an LLM is largely determined by its training corpus. English has a longer history and a more complete ecosystem in technical documentation, academic papers, open-source code comments, standard question-answer pairs, and product documentation. As a result, the proportion of high-quality English samples is often higher.

The total volume of Chinese text on the internet is very large, but the relative proportion of “structured, logically rigorous, factually accurate, and fully validated” high-quality data may be lower.

This leads to a predictable result: the model is exposed to more high-quality samples in English, allowing it to estimate probabilistic relationships between words, sentences, and paragraphs more accurately. In Chinese scenarios, if high-quality samples are insufficient, the model may perform worse on complex reasoning, long-form summarization, and professional question answering.

It is also worth noting that this effect is related to model size. Generally, larger models have stronger capacity to digest training data, so differences in data quality become more pronounced. Smaller models may be more affected by corpus distribution and training strategy.

2.2 Tokenizer efficiency differences

Tokenizers are a critical but often overlooked factor in LLM performance for Chinese.

Most mainstream models use subword tokenization. Their vocabularies are usually built from statistics of the training corpus. If English accounts for a higher proportion of the training data, the vocabulary will provide better coverage for English. A common English word may occupy only a few tokens, while Chinese characters carry high information density and combine flexibly, so the same semantic content may be split into more tokens.

This creates several downstream effects:

• Shorter effective context: If Chinese consumes more tokens for the same number of characters, fewer pieces of information can fit into a fixed context window.
• Higher computational cost: Both training and inference costs are related to token count, so long Chinese text may have a higher marginal cost.
• More fragmented semantic segmentation: After Chinese words are split into multiple subword or character-level tokens, the model needs stronger context modeling ability to reconstruct the intended meaning.

Below is an illustrative code framework. Note that this is pseudocode and cannot be run directly, because encoding results differ across models, tokenizers, and versions.

# Pseudocode: cannot be run directly; results may vary by model, tokenizer, and version.
# tokenizer = load_tokenizer(model_name="<your_model>", revision="<model_version>")
text_zh = "Please summarize the core conclusions of this technical document."
text_en = "Please summarize the key conclusions of this document."
# tokens_zh = tokenizer.encode(text_zh, add_special_tokens=False)
# tokens_en = tokenizer.encode(text_en, add_special_tokens=False)
# print(len(tokens_zh), len(tokens_en))

It is not advisable to draw a conclusion from a fixed number, such as “Chinese always uses X% more tokens than English.” A more rigorous approach is to record the following information:

Comparison item	Chinese sample	English sample	Engineering note
Model name	Must be recorded	Must be recorded	Do not mix different models
Tokenizer version	Must be recorded	Must be recorded	Version affects tokenization
Special tokens	Must be noted	Must be noted	Affects token counting
Text length	Character count	Character count	Useful for cross-language comparison
Token count	Measured	Measured	Track input and output separately
Cost basis	Unit price × tokens	Unit price × tokens	Separate input and output pricing

In some tokenizers, Chinese may be split into smaller pieces, causing the same semantic content to consume more tokens than English. In other tokenizers, the difference may be smaller. Therefore, engineering decisions should be based on actual measurements.

2.3 Differences in data cleaning and quality tiering

High-quality models usually rely on strict data governance processes, including deduplication, noise removal, harmful content filtering, quality scoring, and domain balancing.

The problem is that these rules and quality-scoring models may be designed primarily around English corpora. When the same strategies are applied to Chinese data, two types of bias may appear:

• False rejection: Valid Chinese text is classified as low quality and removed.
• Missed filtering: Some low-quality, repetitive, or noisy Chinese content remains in the dataset.

If high-quality Chinese data entering training is reduced, or if the proportion of low-quality Chinese data is too high, model performance on Chinese tasks will be constrained. Data-side problems eventually show up in generation quality.

2.4 Language bias in instruction tuning and alignment data

The usability of a model depends heavily on the post-training stage, including instruction tuning, human feedback alignment, and safety alignment.

If this stage is dominated by English instructions and high-quality English responses, the model is more likely to learn response patterns for English scenarios, such as how to structure bullet points, follow format requirements, control length, and complete multi-step reasoning.

If Chinese alignment data is insufficient, the model may show the following behavior: it can answer questions, but the answers are not well organized; it can understand instructions, but format following is unstable; it can generate content, but factual accuracy and structural quality are weaker. This is also an important reason why many teams feel that “Chinese answers are acceptable, but not quite good enough.”

3. How to quantify the Chinese-English gap in LLMs scientifically

To determine whether the Chinese-English response gap really exists, subjective impressions are not enough. A reproducible, comparable, and auditable evaluation method is needed.

3.1 Build a parallel test set

It is recommended to prepare semantically equivalent Chinese and English question pairs covering the following tasks:

• Factual question answering
• Logical reasoning
• Long-document summarization
• Code generation
• Instruction following
• Format output
• Multi-turn dialogue
• Domain-specific question answering

Suggested sample sizes:

• Quick validation: 30–50 pairs per task type.
• Regular evaluation: 100–200 pairs per task type.
• Formal acceptance testing: At least 1,000 pairs in total, or coverage of core business scenarios.

A sample field template for a parallel test set is shown below:

id: unique identifier
language: zh / en
task_type: qa / summary / code / instruction_following
prompt: input prompt
reference: reference answer or key points
rubric: scoring items
expected_format: expected output format
notes: notes

3.2 Fix the scoring rubric and weights

A unified scoring standard with explicit weights is recommended. For example:

Scoring dimension	Weight	Description
Factual accuracy	40%	Whether the answer is correct and free of factual errors
Completeness	25%	Whether the answer covers key points
Relevance	20%	Whether the answer stays on topic without unnecessary content
Format compliance	10%	Whether the output follows list, JSON, Markdown, or other requirements
Safety and compliance	5%	Whether the output contains risky or non-compliant expressions

Each dimension can be scored from 0 to 5, and a weighted total can be calculated. For critical business scenarios, veto items can also be added. For example, a severe factual error can make the output directly unacceptable.

3.3 Control variables

Variables must be controlled during evaluation. Otherwise, configuration differences may be mistaken for language differences.

The following parameters should be kept consistent:

• Same system prompt
• Same temperature setting
• Same top_p or top_k
• Same maximum output length
• Same context window limit
• Same decoding strategy
• Same model version

If business conditions allow, it is recommended to use a low or fixed temperature during testing to reduce randomness.

3.4 Add statistical significance and confidence intervals

Comparing only average scores may hide fluctuations. The following methods are recommended:

• Calculate average scores for Chinese and English separately for each task.
• Use paired comparison, because Chinese and English questions are semantically equivalent.
• Use bootstrap methods to calculate 95% confidence intervals.
• If resources allow, combine paired t-tests or non-parametric tests.
• Define a minimum practical difference. For example, a Chinese-English average score difference greater than 0.3 with a 95% confidence interval that does not include zero may be considered engineering-meaningful.

3.5 Evaluate quality, cost, and token efficiency together

LLM optimization for Chinese should not focus only on quality. Cost must also be considered.

The following formulas can be used:

Total cost = input token count × input unit price + output token count × output unit price + compute or cloud resource cost
Unit quality cost = total cost / weighted score
Token efficiency = weighted score / total token count
Chinese optimization gain = optimized unit quality cost / baseline unit quality cost - 1

The evaluation process can be represented as pseudocode:

# Pseudocode: cannot be run directly
records = []
for case in parallel_testset:
    out_zh = model.generate(case.prompt_zh, **fixed_params)
    out_en = model.generate(case.prompt_en, **fixed_params)
    score_zh = rubric.evaluate(out_zh, case.rubric)
    score_en = rubric.evaluate(out_en, case.rubric)
    cost_zh = estimate_cost(case.prompt_zh, out_zh)
    cost_en = estimate_cost(case.prompt_en, out_en)
    records.append({
        "id": case.id,
        "score_zh": score_zh,
        "score_en": score_en,
        "cost_zh": cost_zh,
        "cost_en": cost_en,
        "diff_score": score_zh - score_en,
        "diff_cost": cost_zh - cost_en,
    })
report = analyze_records(records)

4. Does Chinese corpus improve or reduce model capability? A conditional conclusion

Regarding whether Chinese corpus improves or reduces model capability, absolute claims should be avoided. A more rigorous statement is as follows:

High-quality Chinese corpora usually help improve a model’s Chinese capabilities, and may also provide some cross-lingual transfer through multilingual training. Whether English performance improves depends on model size, training ratio, data quality, training strategy, and task type. Low-quality Chinese corpora may provide limited benefit or even dilute overall data quality.

Therefore, the real question is not whether to include Chinese corpus, but how to increase the density and quality of effective Chinese data.

For business teams, prioritizing high-quality Chinese instruction data, domain knowledge bases, question-answer pairs, evaluation sets, and filtering rules is usually more valuable than simply increasing the amount of Chinese text.

5. Engineering optimization path: what to do first and how to choose

5.1 Start with prompt engineering and retrieval-augmented generation

In most business scenarios, the recommended optimization order is as follows:

1. Prompt engineering first: Clearly define role, task, constraints, output format, examples, and prohibitions.
2. Then apply RAG: Inject authoritative Chinese knowledge into the context to reduce issues caused by insufficient internal Chinese knowledge.
3. Then improve data governance: Clean low-quality Chinese corpora and build high-quality Chinese instruction and question-answer data.
4. Finally consider fine-tuning or model selection: If evaluation shows that a specific task continues to lag, consider fine-tuning, switching models, or hybrid calling.

The advantage of this order is that prompt engineering and RAG usually produce faster results with controllable costs, while data governance and fine-tuning require longer cycles and higher investment.

5.2 Chinese prompt engineering checklist

• Are structured prompts used instead of a single vague instruction?
• Is the output format clearly specified, such as JSON, Markdown, tables, or lists?
• Are a few high-quality examples provided?
• Are answer length and irrelevant content constrained?
• Is the model asked to list key points before generating conclusions?
• Are requirements provided for technical terms, citation sources, and factual evidence?

5.3 Chinese knowledge enhancement checklist

• Is an authoritative Chinese knowledge base available?
• Has the knowledge base been deduplicated, denoised, and version-controlled?
• Is retrieval, reranking, and citation traceability supported?
• Can external knowledge be combined with prompt templates?
• Can retrieval quality be monitored?

5.4 General cloud capabilities can be used as infrastructure

When implementing the above pipeline, generic cloud capabilities can often be reused. For example, object storage can host corpora and knowledge bases, message queues can support data processing pipelines, container platforms can provide elastic scaling for training and inference, and logging, monitoring, and alerting services can help observe task status.

The above are illustrative capability descriptions only. They do not constitute a commitment to specific product selection or commercial endorsement. Developers should evaluate choices based on their own business needs, cost, compliance requirements, and operational capabilities.

6. Common mistakes teams make during evaluation

6.1 Inconsistent temperature and generation parameters

If Chinese testing uses a temperature of 0.8 while English testing uses 0.2, the comparison is meaningless. Generation parameters must be fixed.

6.2 Inconsistent prompt templates

If Chinese and English prompts differ in wording, the task difficulty is different. Semantically equivalent prompt templates with consistent structure should be used.

6.3 Missing statistical significance

Comparing only one generation result or a small number of samples is easily affected by randomness. Sufficient sample size should be used, and confidence intervals or significance judgments should be reported.

6.4 Looking only at quality, not cost

Chinese may consume more tokens, which can increase unit quality cost. Evaluation should record quality, tokens, and cost at the same time.

6.5 Relying only on subjective impressions

“Chinese answers feel a bit worse” can be a clue, but it should not be the final conclusion. A rubric, test set, and report template should be established.

7. Practical recommendations for developers

First, evaluate before drawing conclusions. Use parallel test sets to quantify LLM response quality in Chinese and the Chinese-English gap, rather than making technical decisions based on impressions.

Second, distinguish capability problems from cost problems. Higher Chinese token consumption may indicate a quality issue, a cost issue, or both. Evaluation should record quality scores, token counts, and fees together.

Third, prioritize data and retrieval improvement rather than blindly switching models. In many scenarios, improving prompts, building a Chinese knowledge base, introducing RAG, and cleaning high-quality Chinese instruction data offer better value.

Fourth, make language adaptation part of the standard workflow. It is recommended to integrate multilingual evaluation into continuous integration, and continuously monitor changes in Chinese and English performance after model versions, prompt templates, knowledge bases, or corpora are updated.

In summary, the fact that LLM output is often weaker in Chinese than in English is not a simple model preference. It is an engineering problem caused by corpus, tokenization, data governance, and alignment strategies. By building a quantifiable evaluation system, defining a clear cost baseline, and implementing optimizations in priority order, teams can gradually improve LLM Chinese response quality in real business scenarios, bringing multilingual capabilities to a level that is usable, effective, and maintainable.