Evaluating China's Top-Tier LLMs: From User Experience to Engineering
Determining which Chinese large language models belong to the "first tier" requires more than a generalized ranking. This article explores how to define and evaluate top-tier LLMs based on specific tasks, user types, and engineering requirements. It outlines core capabilities such as task correctness, instruction adherence, long-context understanding, coding ability, and uncertainty expression, and provides a framework for building a repeatable evaluation process and making informed model selection decisions for enterprise deployment.
1. Defining the “First Tier”
When discussing which Chinese large language models (LLMs) belong to the first tier, a common mistake is to compress different use cases into a single unified ranking. Some focus on code generation accuracy, others on the convenience of long-text reading, while some prioritize response speed, expression naturalness, or API costs. These criteria are all valid but not necessarily interchangeable.
A more rigorous understanding is that the “first tier” should refer to a group of models that consistently achieve high-quality results under specific times, tasks, and constraints, rather than a fixed list that applies to all scenarios.
This requires three premises:
First, clarify the evaluation target. Base model capabilities, application-level experience, and complete business system performance are not the same. An application can improve its performance through retrieval, document parsing, prompt design, and result validation; therefore, application experience should not be directly equated with underlying model capability.
Second, clarify the evaluation time. Model versions, service strategies, and application features may change. Any conclusion should record the test date and version information; otherwise, it is difficult to reproduce.
Third, clarify task boundaries. Being good at daily Q&A does not mean being suitable for complex code maintenance; being able to process long inputs does not mean accurately understanding all constraints within them.
Therefore, instead of asking “who ranks first,” it is better to ask: “What capabilities does my task require, and how can I prove that the model has them?”
2. Why Developers and Everyday Users Reach Different Conclusions
2.1 Developers value controllability and integrability
For developers, an LLM is not just a chat window to be opened and used; it is a component of uncertainty within a business system. Model outputs need to be consumed by programs, errors need to be identified, and failures need to be recoverable.
For example, a business system might require the model to extract payment milestones from a contract. Developers are concerned not only with whether the answer is fluent but also with whether the fields are complete, whether the amounts correspond to the original text, whether missing information is explicitly marked, whether the output format can be stably parsed, and whether exceptions can be handled via retries or manual review.
Thus, an excellent model in development scenarios must perform reliably in:
- Following clear input and output constraints;
- Distinguishing between known, inferred, and missing information;
- Maintaining acceptable consistency across similar inputs;
- Demonstrating sufficient understanding of code, data structures, and business rules;
- Integrating with retrieval, access control, and validation workflows.
These capabilities determine the engineering cost post-integration, not just the impression during a demo.
2.2 Everyday users value task completion experience
Daily use emphasizes low learning costs and natural interaction. Users typically do not write strict field specifications first; they directly ask to “organize this material,” “explain this content,” or “make this text clearer.”
Here, the model’s ability to understand incomplete expressions, proactively clarify ambiguities, maintain multi-turn context, and provide easy-to-read answers is often more important than structured output.
However, a good experience does not equate to factual correctness. A well-structured, confidently toned answer may still contain unverified information. For medical, legal, financial, and other high-risk issues, the model should be used as an auxiliary analysis tool rather than the final decision-maker.
The difference between these two types of needs shows that the model preferred by developers can differ from the one preferred by daily users, and it does not necessarily mean that either party is wrong.
3. Core Capabilities to Observe for Evaluating the First Tier
3.1 Task Correctness: Does the answer actually solve the problem?
Correctness should be at the center of the evaluation framework. A well-written article does not mean the conclusion holds; code that runs does not mean it meets the requirements.
Different tasks require different standards:
- Information extraction: Check field accuracy, completeness, and correspondence with the source text.
- Document Q&A: Check whether conclusions are supported by materials and whether restrictive conditions are omitted.
- Programming assistance: Check functionality, boundary conditions, exception handling, and maintainability.
- Content generation: Check facts, structure, expression, and adherence to constraints.
Particularly, identify answers that are “partially correct but wrong on key points.” For high-risk fields like amounts, dates, permissions, and execution conditions, an average score may mask severe issues, so separate checks should be set up.
3.2 Instruction Adherence: Does it complete the task as required?
Instruction adherence is not just about following word count or format; it also means avoiding conflicts when executing multiple constraints. For example, if asked to “answer only based on materials, retain uncertainty, use fixed fields, and do not add external information,” the model should meet all these conditions simultaneously.
During testing, requirements can be broken down into independent checks to record:
- Whether mandatory content is omitted;
- Whether forbidden content is added;
- Whether the specified format is altered;
- Whether answers are fabricated when information is insufficient.
Format compliance and factual correctness should be scored separately. A completely correct format for a wrong answer cannot be considered a task success.
3.3 Long-context Understanding: Handling input does not equal comprehensive understanding
Long-context capability encompasses at least input capacity, information localization, cross-segment correlation, and comprehensive reasoning.
In actual evaluations, key facts can be placed at the beginning, middle, and end of a document to observe if the model has positional bias. You can also add clauses with similar wording but different scopes of application to test whether it confuses objects, times, and conditions.
When materials exceed the processable range, consider segmented retrieval, hierarchical summarization, or question-based evidence extraction instead of simple truncation. Note that summarization loses details and segmentation may break cross-segment relationships, so the processing method also needs validation.
3.4 Coding Ability: From generating snippets to verification and modification
Coding tests should not only cover short examples but also common tasks in actual development:
- Understanding existing code and call relationships;
- Locating defects and explaining causes;
- Modifying functionality without breaking existing behavior;
- Adding normal, exception, and boundary tests;
- Identifying obvious security risks.
When evaluating code, it should be run in an isolated environment with manual review. Execution commands, file operations, or permission modification suggestions provided by the model should not be executed directly just because the explanation is detailed.
3.5 Uncertainty Expression: Knowing when not to conclude
Reliability is reflected not only in answering correctly but also in not over-answering when information is insufficient.
The test set should include questions that cannot be answered from the materials, contradictory descriptions, and tasks missing key premises. Observe whether the model can identify gaps, request supplementary information, or provide conditional conclusions.
However, do not judge caution solely based on wording like “possibly” or “for reference only.” Truly valuable uncertainty expression needs to explain what is uncertain and what evidence is needed to continue the judgment.
4. Establishing a Repeatable Evaluation Process
4.1 Build a task set from real business scenarios
Do not only use ideal questions designed to showcase capabilities. A more effective method is to extract representative tasks from real work and perform necessary desensitization.
The task set should cover high-frequency tasks, complex tasks, information-missing tasks, abnormal inputs, and high-risk tasks. A portion of independent test questions should also be retained to avoid getting good results only on familiar questions after repeated prompt adjustments.
For each task, it is best to record the input materials, expected results, scoring criteria, risk level, and allowed processing time. For tasks without a single correct answer, itemized scoring rules can be established instead of forcibly matching a specific standard phrasing.
4.2 Unify testing conditions as much as possible
When comparing, unify the task descriptions, input content, output requirements, and available external information.
If one solution can retrieve materials while another can only answer based on input, the final result reflects the difference in system solutions, not pure model differences. This comparison is not without value, but the object of comparison must be clearly labeled.
Additionally, it is recommended to repeat tests on some tasks to observe output fluctuations and record call failures, timeouts, and format anomalies. A single success does not represent long-term stability.
4.3 Adopt itemized scoring rather than a single impression score
You can separately evaluate correctness, completeness, constraint adherence, evidence support, and readability based on actual use. Weights should be determined by business needs, and it is not advisable to assume a universal ratio for all scenarios.
For high-risk scenarios, thresholds must be set that cannot be offset by average scores. For example, if an unauthorized leak or a critical amount error occurs, even if other items perform excellently, it should not directly enter the production process.
5. Enterprise Deployment Requires Engineering and Security Perspectives
Strong model capabilities do not automatically mean the system is usable. Enterprises must also consider response time, service stability, data protection, permission isolation, and fault recovery.
Response time should distinguish between the wait time for the first token and the time for complete result generation; cost should be calculated based on the total expense of completing a qualified task, not just the single call fee. Retries, manual modifications, retrieval, document processing, and result validation can all affect the final cost.
In knowledge Q&A, a common workflow is:
User question → Identity and permission check → Retrieve authorized materials → Organize necessary context → Model generation → Result validation → Return answer.
The key here is that permissions should be enforced at the retrieval and data access stages, not just by prompting the model “do not leak.” External documents should also be treated as data to be analyzed, not as instructions with the authority to change system rules.
For tasks that trigger actual operations, executable actions should be further restricted, and manual confirmation should be added before high-risk operations.
6. From Evaluation Results to Selection Decisions
After testing, there is no need to force a single champion covering all use cases. A more practical approach is to build a capability matrix based on tasks:
- Daily writing and material organization: Prioritize interaction experience, structural organization, and fact-checking;
- Programming assistance: Prioritize code correctness, modification boundaries, and testing performance;
- Enterprise knowledge Q&A: Prioritize evidence correspondence, permission control, and handling of unanswerable situations;
- Complex business processing: Prioritize constraint adherence, exception recovery, and operation auditing.
If considering combining multiple models, you should also evaluate the additional routing, maintenance, data flow, and troubleshooting costs. A combined solution is not inherently better than a single one; it is only worth adopting if the benefits cover the complexity.
Going live is also not the end of evaluation. You should continuously collect failure samples, re-validate key tasks on updated models, and retain a rollback mechanism.
Conclusion
The “first tier” of Chinese LLMs is better understood as a set of task-validated capability standards rather than a fixed ranking detached from use cases.
From a developer’s perspective, what matters is integrability, verifiability, and controllability; from a daily user’s perspective, natural understanding, ease of operation, and practical results; from an enterprise perspective, security, stability, and total cost must also be added.
A truly reliable choice is not finding a model that leads in all questions, but finding a solution that consistently performs well under your own business constraints through real tasks, unified conditions, and clear standards.
