Evaluating and Improving Chinese Expression in Large Language Models

    A practical framework for assessing Chinese-language capabilities in large language models. It covers semantic fidelity, register and rhythm, structural organization, revision cost, benchmark design, prompt workflows, long-context consistency, and engineering practices for regression testing and production deployment. The central argument is that usable output should be evaluated by editability and task-specific reliability, not by isolated “beautiful” sentences.

    The Core Problem: Chinese Expression Requires More Than Literal Fluency

    The difficulty of formal Chinese writing is often less about vocabulary coverage than about information compression and contextual dependence. A short sentence may simultaneously convey a judgment, a warning, and a particular rhythm. The same idea can require very different levels of lexical density in an official document, explainer, dialogue, or commentary.

    When a large language model performs only a literal transformation, readers may feel that the meaning is correct but the expression has lost its character. Chinese-language capability should therefore be defined as the ability to consistently produce semantically accurate, structurally editable, and stylistically predictable text under constraints involving genre, audience, and length. It should not be judged by the occasional brilliance of a single generation.

    Three common evaluation errors appear in practice:

    1. Treating translation accuracy as equivalent to expressive ability, while overlooking idioms, parallel structures, and implied meaning.
    2. Replacing reproducible samples with subjective judgments about whether a passage “reads well.”
    3. Confusing outline generation with full-text writing. Some models build strong structures but become flat or vague in dialogue and argumentation. Others produce polished individual paragraphs but lose their tone and consistency in longer documents.

    Evaluation should separate these capabilities.

    Four Practical Evaluation Dimensions

    Semantic Fidelity

    Provide the model with a high-density source passage, such as an explanatory text, dialogue, or aphoristic sentence, and ask it to rewrite rather than expand the content.

    Check whether the model has added or removed judgments, turned implications into explicit statements, or introduced causal relationships absent from the source. A typical failure is “explanatory rewriting,” in which a compressed sentence is unpacked into a lesson-like explanation.

    For development teams, semantic fidelity determines whether a model can support summarization, localization, and script polishing instead of merely paraphrasing the source.

    Register and Rhythm

    Ask the model to express the same facts as a technical explanation, a product announcement, and a narrative voice-over. A qualified output should show clear differences in sentence length, terminology density, and subjective tone.

    Chinese technical writing generally benefits from short sentences, definitions presented early, and verifiable conclusions. Narrative writing allows more space for implication and omission. If all three tasks sound like variations of the same template, the model likely has only one default register.

    Structural Organization

    Give the model a set of constraints covering two or three hundred Chinese characters, including character relationships, conflicts, required information, and a length limit. Ask it to produce an outline first and then the full text.

    Evaluate three aspects:

    • Whether the outline covers all constraints
    • Whether the body follows the outline
    • Whether each paragraph can be edited independently

    Strong structural ability means that later human revision can focus on local changes instead of requiring a complete rewrite. In production environments, this is often more valuable than ornate prose.

    Revision Cost

    Estimate rework using fixed ratios. For example, in an 800-character draft, record the percentage of sentences that require rewriting, the number of factual errors, and the number of stylistically inappropriate passages.

    Revision cost is closer to actual team productivity than a one-time subjective score. If a model frequently generates text that is technically correct but unusable—essentially “correct filler”—its revision cost remains high.

    Building an Evaluation Set

    Do not rely on occasional comparisons of impressive sentences. Build a small, reproducible Chinese task set with at least a dozen samples for each category. Keep temperature, length limits, and system constraints fixed so that the prompt itself does not become the largest source of variation.

    Recommended task types include:

    1. Compressed rewriting: Reduce an explanatory passage to no more than 40 Chinese characters without losing key conditions.
    2. Cross-register rewriting: Convert the same information into an explanation, announcement, and dialogue.
    3. Constraint-based writing: Generate a single-scene passage within upper and lower length limits after receiving a predefined outline.
    4. Consistency testing: Check whether terminology, references to people, and positions remain stable throughout a long text.
    5. Negative samples: Use intentionally ambiguous prompts to observe whether the model asks clarifying questions or fills in missing details without authorization.

    Two annotators should score the outputs independently, followed by a review of disagreements. Record whether each error is a comprehension error or an expression error.

    Comprehension errors require additional facts and constraints. If the expression is too direct or stylistically inappropriate, the solution is to revise the register instructions and examples—not simply to make the prompt longer.

    A constraint template can be short, but every requirement must be checkable:

    1. Task: Rewrite the following information as a technical explanation in 120–180 Chinese characters.
    2. Must preserve: applicable conditions, limitations, and recommended actions.
    3. Do not include: new data, brand comparisons, or exaggerated conclusions.
    4. Output: body text only, without bullet points.

    Prompting and Workflow: Make the Output Editable

    In practice, generation is more reliable when divided into four stages:

    1. Constraint confirmation
    2. Outline creation
    3. Body generation
    4. Self-checking

    The constraints should specify the reader, length, facts that must be preserved, and prohibited content. The outline should list only the order of information and the function of each paragraph, rather than prewriting complete sentences.

    During body generation, limit unnecessary creative variation. For example, explicitly prohibit the model from adding causes that were not present in the source. The self-check should verify whether the model has added conclusions, changed terminology, or explained an implied sentence too explicitly.

    For compressed Chinese writing, examples are more useful than adjectives. Instead of asking for “more literary” language, provide a direct version and a target version so the model can align information density and sentence rhythm. The target version should come from material that you are authorized to use. This prevents style requirements from becoming vague and unverifiable.

    Long-form tasks require separate attention to working memory. Check whether titles, numbers, and positions established earlier remain consistent later in the text. At the end of the document, ask the model to list the “facts used” and compare them with the source manually. This can reveal subtle additions more efficiently than reading the entire passage line by line.

    Engineering Practices for Production

    When integrating a model into a writing or customer-service workflow, place the evaluation set in a regression-testing pipeline. Whenever prompts, model versions, or post-processing rules change, rerun tests for semantic fidelity and revision cost rather than checking only whether the API request succeeds.

    Logs should retain the task type, a hash of the input constraints, and the proportion of text modified by human reviewers. This makes it easier to identify a sustained decline in a particular register.

    Object storage can hold samples and annotations, while a message queue can run batch evaluations asynchronously. Monitoring and alerts should focus on sudden changes in revision rates. These are general engineering practices and do not depend on a particular vendor’s proprietary interface.

    A cloud development environment such as Baidu AI Cloud can be used to store evaluation scripts, samples, and results in separate environments, with unified logging and permission controls to prevent evaluation data from being scattered across individual accounts. This refers only to general practices for storage, task scheduling, and access control, not to any custom delivery or partnership arrangement.

    How to Interpret Which Model “Writes Better”

    Cross-model comparisons are not reproducible without unified tasks, unified constraints, and independent annotation by two reviewers. A model’s ranking can differ across outline generation, dialogue, and technical explanation. Free-tier limits, context length, and post-processing can also affect the perceived quality.

    A more reliable approach is to select a capability combination based on the business task and use revision cost for acceptance testing. A single translated sentence should not determine the overall judgment of a model.

    The strength of Chinese lies partly in its ability to carry judgment and attitude in relatively few characters. Models can approximate this compression, but only when evaluation focuses on fidelity and register, while the production workflow preserves a structure that people can edit.

    When output is technically correct but requires complete rewriting, the problem is usually not that the prompt needs to sound more sophisticated. More often, the task definition and acceptance criteria have not yet been expressed as checkable constraints.