Evaluating Chinese LLMs: A Five-Dimensional Framework and Model Selection Pitfalls
This article presents a five-dimensional framework for evaluating Chinese LLMs: compliance and safety, context adherence, factual accuracy and decision quality, format and structured output, and North Star and role consistency. It explains why a single leaderboard score can hide weaknesses, uses a set of reference scores to show how models differ by dimension, and discusses how to apply the results to model selection and production deployment, with caveats about the reference data.
1. Why Model Selection Needs an Evaluation Method
The capability boundaries of large language models are still expanding rapidly. Early models mainly handled text generation and summarization/extraction. Today’s models also need to handle multi-turn dialogue, tool calling, structured output, and role-playing. The gap between models on complex tasks is becoming increasingly obvious.
For developers and enterprises, model selection should not rely only on parameter scale or release date. The real basis is what the business needs and what constraints the scenario imposes. A reproducible evaluation method that covers multiple dimensions is an important basis for answering “which model is more suitable for my scenario.”
A composite score from a single leaderboard often masks weaknesses. A model with a high total score may be unstable in role consistency and show persona drift in customer service training scenarios. Another model with an average total score may be very stable in format adherence and therefore more suitable for structured data extraction. Model selection should therefore look at specific dimensions, not just one number.
This article does not provide a simple ranking. It first explains the evaluation framework, then uses a set of measured data to illustrate differences between models, and finally discusses how to apply evaluation results to model selection and production deployment.
2. Five-Dimensional Evaluation Framework
The evaluation scheme can be divided into five dimensions, each with a maximum score of 1.0. These five dimensions cover safety boundaries, instruction execution, facts and reasoning, output format, and identity and knowledge boundaries. They are interrelated and together form an evaluation loop from generation quality to engineering usability.
2.1 Compliance and Safety
This dimension evaluates model performance on questions related to safety, ethics, law, and values. An ideal model neither over-refuses normal questions nor outputs harmful advice.
Scoring mainly considers three points: whether the model can identify risk; whether it remains neutral and restrained on sensitive edge topics; and whether it can provide a reasonable alternative direction when refusing to answer. The gap between models is largest in this dimension. In the raw data, some models scored 0.96, while others scored only 0.52, indicating that different teams invest very differently in safety alignment.
2.2 Context Adherence
Context adherence refers to whether a model can strictly answer around the user’s instructions, historical information, and constraints. This includes whether it can correctly understand pronoun references in multi-turn dialogue; whether it can follow hard instructions such as “output only JSON” or “answer in list form”; and whether it can stay consistent with the initial topic during long-form reasoning.
This dimension directly affects whether the model is controllable in engineering. During evaluation, some models with strong reasoning ability performed only moderately on context adherence. Reasoning ability and instruction following are not naturally bound together, so they need to be tested separately during model selection.
2.3 Factual Accuracy and Decision Quality
Factual accuracy measures whether a model’s statements are consistent with objective knowledge, especially when data, dates, and technical terms are involved, and whether it avoids fabricating content. Decision quality measures the quality of the model’s judgment when facing open-ended questions. A correct conclusion is not enough; the derivation process should also be complete and the causal relationships clear.
These two capabilities often appear together, so they are combined into one dimension for scoring. In the raw data, except for Model I, most models scored between 0.80 and 0.88 in this dimension. There is still room for improvement in complex reasoning and rigorous judgment.
2.4 Format and Structured Output
As AI applications move from text chat to data pipelines, model control over output format becomes important. This dimension checks whether a model strictly follows a specified format, for example whether it generates valid JSON, whether Markdown hierarchy is standard, whether list indentation is correct, and whether field names match the agreed convention.
Models with weak overall capability are sometimes very stable in format adherence. Conversely, some highly capable models occasionally “freestyle,” breaking format constraints and increasing the difficulty of downstream parsing.
2.5 North Star Metric and Role Consistency
The North Star metric is a validation question with a pre-set answer in the evaluation set. It is used to detect whether a model is easily misled or produces knowledge hallucinations. Role consistency requires a model to maintain a matching tone, knowledge boundary, and response style when playing a specific role, such as a “senior lawyer” or a “psychological counselor.”
These two items show obvious gaps in the raw data. High-scoring models have a stronger sense of boundaries and are not easily induced. Low-scoring models are more likely to answer off-topic or break out of the role setting.
3. Reference Scores for a Set of Chinese LLMs
The evaluation used the same evaluation set and the same sampling parameters (temperature=0.3, top_p=0.9). Models are represented by letter codes, and the codes do not correspond to specific vendors.
Three points need to be noted. First, the raw records counted the best result per round, which inflates the values. Readers reproducing the evaluation should use multi-round means or medians. Second, this article does not provide the evaluation set sample size, raters, or confidence intervals; the values in the table can only serve as a reference. Third, the data in the table only represent the results of this evaluation set under this set of parameters and cannot be used to directly infer production environment performance.
The “Five-Dimension Average” in the table below is the simple average of the five dimensions, rounded to two decimal places.
| Model Code | Compliance & Safety | Context Adherence | Facts & Decisions | Format Output | North Star & Role | Five-Dimension Average |
|---|---|---|---|---|---|---|
| A | 0.96 | 0.97 | 0.88 | 0.96 | 0.97 | 0.95 |
| B | 0.94 | 0.95 | 0.88 | 0.95 | 0.96 | 0.94 |
| C | 0.93 | 0.96 | 0.83 | 0.94 | 0.92 | 0.92 |
| D | 0.93 | 0.82 | 0.84 | 0.92 | 0.70 | 0.84 |
| E | 0.88 | 0.76 | 0.81 | 0.78 | 0.66 | 0.78 |
| F | 0.90 | 0.81 | 0.80 | 0.73 | 0.74 | 0.80 |
| G | 0.58 | 0.91 | 0.82 | 0.91 | 0.84 | 0.81 |
| H | 0.52 | 0.86 | 0.80 | 0.91 | 0.86 | 0.79 |
| I | 0.55 | 0.78 | 0.38 | 0.28 | 0.86 | 0.57 |
3.1 First Tier: Balanced Across Dimensions (A, B, C)
These three models score high across all five dimensions. None of their scores in any dimension is below 0.83. They usually combine strong reasoning ability with sufficient instruction tuning and safety alignment.
Model A has an average score of 0.95 and a context adherence score of 0.97, the highest value in the table. In multi-turn dialogue or long-document tasks, Model A has a lower probability of drifting. It is suitable for core business functions such as intelligent customer service, complex workflow automation, and multi-step data analysis.
Model B is only 0.01 lower than A in average score. The score gap between the two is very small, and users usually will not notice it in actual business scenarios.
