Quality Evaluation and Engineering Selection Guide for Chinese LLMs

    A practical framework for evaluating Chinese large language models beyond leaderboard scores. It covers seven quality dimensions, layered benchmark design, reproducible execution, judge-model scoring, hard eligibility rules, failure analysis, and scenario-based selection for production use.

    Quality Evaluation and Engineering Selection Guide for Chinese LLMs

    As the number of Chinese large language models continues to grow, one of the most common enterprise mistakes is reducing “quality” to a single leaderboard score. A high-scoring model is not necessarily the right choice for a specific business. Some models handle compliance well but produce unstable structured output. Others have a natural writing style but fluctuate in factual consistency and complex reasoning. A model may perform well in a one-off demonstration yet produce format drift, repetition, or inappropriate refusals when used at scale.

    A more reliable approach is to build a reproducible and auditable evaluation system that can identify specific failure dimensions, then combine the results with business requirements for engineering-oriented model selection.

    1. Define Quality Before Measuring It

    Output quality should be divided into at least seven independent dimensions. These dimensions interact with one another, but they should be evaluated separately when comparing candidate models or conducting internal acceptance testing.

    Dimension What to evaluate Typical failure modes
    Scenario and channel fit Whether the output matches the requirements of a page, support workflow, document, or messaging channel Inappropriate tone, poor information hierarchy, or output that cannot be used directly
    Compliance and safety Whether the model handles sensitive content, boundary requests, refusals, and risk warnings appropriately Answering requests that should be refused, or refusing too broadly and making the task unusable
    Task completion and conversion Whether the response actually addresses the user’s goal and includes necessary steps or next actions Irrelevant answers, missing key steps, or responses that appear complete but cannot be executed
    Decision-making and reasoning Whether the model can compare options, balance constraints, and explain causes consistently Contradictions, unsupported conclusions, or missed constraints in complex tasks
    Factuality and consistency Whether numbers, dates, entity relationships, and citation-like statements remain coherent Hallucinated facts, conflicting values, or contradictions across turns
    Format and constraint following Whether fields, ordering, length, labels, and structure meet the requirements Extra explanations, missing fields, unparseable output, or repeated format drift
    Persona and style stability Whether role, tone, audience adaptation, and multi-turn consistency are maintained Persona drift, abrupt tone changes, or forgotten context

    Compliance and safety should normally be treated as hard gates. A serious violation, unauthorized answer, or inappropriate refusal in a high-risk task should not be offset by high scores in other dimensions.

    Format compliance depends on the usage scenario. Content intended for human readers can tolerate some free-form expression. Structured output consumed by software must be much stricter; otherwise, downstream systems cannot integrate with the model reliably.

    Average scores are useful for overview reporting, but not for final decisions. A candidate may perform well on most dimensions while having a systematic weakness in compliance or format adherence. The evaluation report should therefore include the overall score, dimension-level distributions, worst-performing samples, and failure categories—not just a ranking table.

    2. Build a Task Set with Realistic and Adversarial Cases

    The quality of an evaluation is limited first by the quality of its task set. Start with samples from real business requests, then add boundary and adversarial cases to create a layered benchmark.

    1. Normal task layer: Cover common tasks such as question answering, summarization, rewriting, classification, extraction, solution comparison, and multi-turn consultation. Each task should have explicit acceptance criteria, including required information, prohibited content, length limits, and format requirements.
    2. Boundary task layer: Include insufficient information, conflicting goals, excessive constraints, overlong inputs, interrupted context, and repeated user revisions. Observe whether the model asks appropriate questions, makes unsupported assumptions, and provides risk-controlled answers when information is incomplete.
    3. Adversarial and trap layer: Add leading language, contradictory constraints, easily misread formatting requirements, sensitive requests that require refusal, and factual questions likely to trigger hallucinations. These cases should not dominate the benchmark, but they are essential because real-world risk is often concentrated in a small number of boundary situations.
    4. Multi-turn and state layer: Structure tasks as conversations. Require the model to preserve its role, remember constraints, correct earlier mistakes, and handle changing user requirements.
    5. Hidden and rotating test sets: Keep part of the benchmark hidden from candidate providers and from repeated tuning. Rotate some cases regularly to reduce overfitting to a fixed test set.

    Each task should ideally include structured metadata such as task type, risk level, required fields, acceptance criteria, expected refusal behavior, allowed clarification methods, and scoring anchors. For subjective tasks, provide reference examples and deduction rules instead of simply stating that the answer should be “good.” This improves consistency between judge models and human reviewers.

    3. Execution Architecture: Parallel Calls, Raw Storage, and Resumability

    For reproducible evaluation, separate model invocation from output scoring. The invocation stage should generate and persist raw results. The scoring stage should read a fixed version of those results. This prevents rule changes during execution from making results incomparable.

    A typical workflow is:

    1. Configure the task set, candidate models, prompt version, and sampling parameters.
    2. Invoke candidates in parallel while controlling rate limits, retries, and timeouts.
    3. Persist raw outputs separately by candidate, task ID, turn, and parameter version.
    4. Record the input, system prompt, user messages, output, timestamp, status, and error details.
    5. Retry failed calls idempotently and support checkpoint-based resumption.
    6. Run the scoring stage independently against the stored outputs.
    7. Aggregate rankings, analyze failure dimensions, and produce a shortlist.

    The process can be abstracted as follows:

    1. for candidate in candidates:
    2. for case in benchmark_suite:
    3. for sample_index in range(sample_count):
    4. run_id = stable_id(
    5. candidate.id, case.id,
    6. prompt_version, sample_index
    7. )
    8. if run_exists(run_id):
    9. continue
    10. request = build_request(case, candidate.config)
    11. response = invoke(candidate, request)
    12. persist_raw(run_id, {
    13. "candidate_id": candidate.id,
    14. "case_id": case.id,
    15. "prompt_version": prompt_version,
    16. "request_meta": request.meta,
    17. "response": response,
    18. "sample_index": sample_index,
    19. "status": "done"
    20. })
    21. raw_results = load_all_runs()
    22. scores = independent_scoring(raw_results, rubric_version)
    23. report = aggregate(scores, weights, hard_rules)

    Traceability is critical. Store prompt versions, sampling parameters, candidate configurations, and failure reasons. Without this information, an abnormal output cannot be reliably attributed to model capability, a prompt change, or a timeout that truncated the response.

    Parallel execution should also isolate failures so that one candidate with repeated timeouts does not block the entire batch. Record rate limiting and retry events as well; otherwise, retries may distort the normal sample distribution.

    4. Scoring: Use Judge Models for Auditable First-Pass Evaluation

    A judge model can perform the initial evaluation across the seven dimensions, but it should not be treated as absolute ground truth. A stronger process is to have the judge score each dimension using a common rubric, provide reasons and supporting evidence, and send high-risk, highly subjective, or disputed cases to human review or a second judge.

    Recommended practices include:

    • Use anchored rubrics: Define levels ranging from unusable, usable with significant defects, good, and excellent to ready for deployment. Avoid vague numerical scoring.
    • Require evidence: The judge must cite specific output fragments or identify missing elements rather than simply stating that the response is “good.”
    • Blind and randomize evaluation: Hide candidate identities and randomize presentation order to reduce brand and position bias.
    • Use a second judge and arbitration: Send cases with score differences above a defined threshold to human review.
    • Perform sampled human checks: Manually inspect high-risk dimensions such as compliance, factuality, and formatting to estimate the judge model’s error rate.
    • Version scoring artifacts: Version the rubric, judge prompts, and judge configuration so historical results remain comparable.

    A scoring record can be stored in the following form:

    1. {
    2. "candidate_id": "candidate_x",
    3. "case_id": "case_001",
    4. "dimensions": {
    5. "channel_fit": {"score": 0, "reason": ""},
    6. "compliance": {"score": 0, "hard_fail": false, "reason": ""},
    7. "conversion": {"score": 0, "reason": ""},
    8. "decision": {"score": 0, "reason": ""},
    9. "facts": {"score": 0, "reason": ""},
    10. "format": {"score": 0, "reason": ""},
    11. "persona": {"score": 0, "reason": ""}
    12. },
    13. "evidence": [],
    14. "review_status": "auto"
    15. }

    Aggregation can use a weighted total score, but hard rules and failure distributions must be retained:

    1. weighted_score(candidate) =
    2. sum(weight[d] * mean_score(candidate, d) for d in dimensions)
    3. eligible(candidate) =
    4. compliance_hard_fail_count == 0
    5. and format_critical_fail_count < threshold
    6. and other_hard_rules_pass
    7. report(candidate) = {
    8. weighted_score,
    9. eligible,
    10. dimension_scores,
    11. worst_cases,
    12. failure_clusters
    13. }

    Final ranking should first filter candidates through hard gates and then compare weighted scores.

    Weights should vary by scenario:

    • Content generation should emphasize style stability and factual consistency.
    • Customer support and consultation should emphasize compliance, task completion, and multi-turn consistency.
    • Structured information processing should emphasize format adherence and field completeness.
    • Complex business analysis should emphasize reasoning, decision quality, and factual support.

    5. Interpret Results by Asking “Which Model Fits Best?”

    Ranking is only one output of the evaluation. Failure diagnosis is often more valuable. Reports should include the following:

    1. Overall scores and confidence intervals: When a candidate is sampled multiple times, examine score variation instead of relying on a single result.
    2. Lowest-scoring dimensions and weak samples: A high average score with one substantially weak dimension may indicate that the model is unsuitable for a particular scenario.
    3. Failure clusters: Group failures into categories such as compliance errors, format drift, missing facts, reasoning gaps, persona drift, and channel mismatch. Determine whether each issue is isolated or systematic.
    4. Prompt robustness: Test whether the same task remains stable under different wording, context lengths, and multi-turn orders.
    5. Operational constraints: Evaluate latency, throughput, cost, long-context stability, and error rate alongside quality.
    6. Fixability: Some formatting issues can be addressed with templates, parsers, or post-processing. Compliance and factuality problems are usually more expensive to correct and should be treated differently.

    Selection conclusions should generally use a shortlist plus scenario matching rather than declaring one model the universal winner. Candidates in the Chinese LLM ecosystem often have different strengths in Chinese understanding, structured output, complex reasoning, persona and style control, and operational stability.

    Organizations should define business hard constraints first, validate shortlisted candidates with hidden test sets and limited-traffic pilots, and then use continuous regression testing and production monitoring to decide whether to expand deployment.

    6. Common Anti-Patterns and a Practical Implementation Path

    Avoid the following anti-patterns:

    • Looking only at average scores
    • Replacing internal validation with public leaderboards
    • Keeping the test set unchanged and encouraging overfitting
    • Evaluating only single-turn interactions
    • Having the same judge model generate and evaluate its own output
    • Failing to persist raw outputs
    • Changing scoring criteria throughout the project without version control
    • Ignoring cost and latency
    • Omitting human spot checks
    • Treating one selection exercise as a permanent conclusion

    A more reliable implementation path is:

    1. Establish the seven-dimension rubric and high-risk hard rules.
    2. Build a layered task set and reserve a hidden test set.
    3. Implement batch execution with checkpoint-based resumption and raw-output persistence.
    4. Use a judge model for initial screening and perform sampled human validation.
    5. Report dimension-level results, failure clusters, and cost metrics.
    6. Select two or three candidates for a limited-traffic pilot.
    7. Continue regression testing and production monitoring after launch.

    A high-quality Chinese LLM is not defined by a fixed list of models. The right choice depends on alignment with business goals, risk levels, formatting requirements, operating costs, and maintenance capabilities. Engineering the evaluation process turns model selection from a decision based on demonstrations and leaderboards into a reproducible, auditable, and continuously improvable technical process.