How to Evaluate Chinese LLMs: From Intent Understanding to Engineering Reliability
Evaluating Chinese large language models for software development requires more than benchmark scores or casual conversation impressions. This article explains why code generation differs from intent understanding, uses a data-and-file management case study to show why "working code" is insufficient, and provides practical guidance on constraint-driven prompting, evaluation metrics, layered model collaboration, and compliance considerations for enterprise adoption.
1. Evaluating Chinese LLMs Starts with Understanding What Problem They Solve
Discussions about the capabilities of Chinese large language models often fall into two extremes. One relies solely on public benchmark scores, assuming that similar scores mean similar user experiences. The other judges a model’s suitability for all development work based on a few conversations.
Both approaches ignore a fundamental fact: software development is not a single code-generation task, but a continuous process that includes requirement clarification, architecture design, implementation, testing, deployment, and maintenance.
A model that quickly writes a function may still fail to recognize business invariants in an enterprise system. A model that generates a complete page may still violate permission, transaction, audit, and compatibility requirements in an existing project. Therefore, the focus of evaluation should shift from “can it produce output” to “can it reliably deliver correct results under constraints.”
The practical value of Chinese LLMs should not be measured by generic competitive slogans either. For developers, what matters is whether the model reduces requirement communication and rework. For enterprises, what matters is whether it can enter production workflows within the boundaries of security, compliance, and cost.
2. Code Generation Is Not the Same as Intent Understanding
2.1 Well-Defined Tasks and Open-Ended Tasks Require Different Capabilities
For tasks with clear boundaries—such as field format validation, simple queries, or fixed template conversion—many models perform well as long as inputs, outputs, and error handling are fully specified.
But in enterprise development, requirements are often incomplete, with constraints scattered across documentation, code, and team conventions.
For example, the instruction “add a lock to archive operations” contains at least the following ambiguities:
- Lock a single file, or the entire directory?
- Lock the archive resource, or the business record it belongs to?
- Restrict only write operations, or deletions as well?
- Is the lock scoped to a single process, or multiple service instances?
- What happens when a business record deletion and an archive upload occur simultaneously?
Models with stronger comprehension are more likely to detect these unstated issues and propose reasonable interpretations based on context. However, “being able to infer user intent” is not the same as “being authorized to confirm requirements on the user’s behalf.” For operations involving data integrity, permissions, or irreversible actions, asking follow-up questions is usually more reliable than generating code directly.
2.2 Parameter Count Is Not a Conclusive Measure of Capability
A larger parameter count may bring stronger knowledge representation and complex task handling, but it is not a sufficient guarantee of engineering performance.
Model effectiveness is also influenced by training data, training methods, context handling, tool-use strategies, and task alignment. A smaller model may outperform a general-purpose large model on well-bounded specialized tasks.
A more rigorous judgment is:
Complex tasks require stronger context understanding and constraint reasoning, but this capability must be validated through actual tasks—it cannot be inferred from parameter count alone.
3. A Data and Archive Management Example: Why “The Code Runs” Is Not Enough
Suppose an enterprise system contains two types of objects: business records and archives. Each archive belongs to one business record. Users can upload, delete, or move archives, and can also delete an entire business record.
Before writing any code, the business invariants must be defined:
- Archives must not be written to a business record that has already been deleted.
- Deleting a business record must not overlook in-progress archive changes.
- Conflicting operations on the same business object must execute serially according to a deterministic rule, or return an explicit conflict.
- After a failure, the system must not be left with an unexplainable association state for an extended period.
These rules are more fundamental than “which lock to use.” A lock is a means of protecting business invariants, not the requirement itself.
3.1 Why a Directory-Level Lock May Be Insufficient
If the model only locks on the archive directory identifier, it can prevent some concurrent conflicts within that directory, but the business record deletion path may not acquire the same lock.
The following interleaving becomes possible:
- Request A locks a directory and begins uploading an archive;
- Request B deletes the business record that owns the directory;
- Request A finishes the upload and proceeds to write archive metadata;
- The system ends up with an archive whose associated record no longer exists, or files that cannot be accessed properly.
This implementation may pass routine functional tests because everything works under sequential execution. The design flaw only surfaces under concurrent interleaving.
3.2 Using the Business Primary Key as the Coordination Boundary
If archives depend on business records, using the business record’s primary key as the shared coordination key usually aligns better with the business consistency boundary.
An illustrative flow:
operateOnArchive(businessRecordId, operationParams):check basic access permissionsacquire the exclusive coordination right for this business recordtry:re-read the business state within the protected scopeverify the record exists, is operable, and permissions remain validexecute the archive changepersist associated metadata and write audit informationfinally:release the coordination right
The entry point for deleting a business record must follow the same coordination protocol. Otherwise, locking only the archive interface cannot provide complete protection.
What truly matters here is not what the lock key is, but: whether all operations that could violate the same business invariant enter the same coordination mechanism.
3.3 A Primary-Key-Level Lock Is Not a Universal Answer Either
Widening the lock scope simplifies reasoning, but it can reduce concurrency. If multiple archives under the same business record can be modified independently, serializing all of them creates unnecessary waiting.
Consider the following when choosing an approach:
- Whether concurrent access to the same business record is frequent;
- Whether uploads, conversions, and similar operations take a long time;
- Whether the business allows retries;
- Whether conflicts can be detected through version checks;
- Whether access spans multiple processes or instances.
A single-process mutex cannot protect multiple service instances. Database row locks generally take effect only within the corresponding transaction scope and cannot automatically constrain external storage operations. With distributed locks, you must also handle lease expiration, holder pauses, mistaken releases, and stale holders continuing to write—adding version or token validation on the write path where necessary.
Therefore, a model that only answers “add a lock” has not yet demonstrated engineering design.
4. When External Files Are Involved, Consistency Extends Beyond Transactions
Enterprise archive systems typically involve both database metadata and file storage. The two are not necessarily in the same transaction, so a file upload may fail even after the database transaction succeeds; conversely, metadata writes may fail after the file upload completes.
Holding a database transaction open while waiting for a large file transfer—just to achieve superficial atomicity—is not advisable. A more reasonable approach is to design an explicit state machine, for example:
- Pending upload;
- Uploading;
- Pending confirmation;
- Available;
- Deleting;
- Deleted.
One workable general flow is: first create a controlled upload task and place the file in a staging area, then—within a shorter transaction or coordination window—check the business record’s state, bind the file, and complete confirmation. On failure, a retryable compensation process cleans up staged files or repairs intermediate states.
The specific solution depends on business requirements, but at minimum three questions must be answered:
- Does repeated execution produce duplicate archives?
- If execution fails midway, how does the system recover?
- When background cleanup and foreground operations occur simultaneously, who has the final say?
A model that proactively identifies these questions demonstrates more engineering value than one that simply produces more code.
5. From “Wishful Programming” to Constraint-Driven Development
Natural language programming lowers the barrier to implementation, but production projects cannot only describe desired outcomes—constraints must be expressed explicitly.
“Implement archive management” is a goal description. “Every archive write must confirm that the owning business record is still in an operable state” is a business contract. “No new persistence dependencies may be introduced” is an architectural constraint. All three need to be provided together.
A reasonably complete task specification should include at least:
- Business goal: what operation the user needs to accomplish;
- Object relationships: which entities depend on each other;
- Consistency rules: which states must never occur;
- Concurrency scenarios: which operations may happen simultaneously;
- Failure semantics: retry, rollback, or compensation after failure;
- Project constraints: existing interfaces, dependencies, deployment model, and coding standards;
- Acceptance criteria: how to prove the implementation is correct.
A model can help fill in omissions, but it cannot be the sole source of requirements. Especially in high-risk areas such as permissions, finance, and data deletion, critical constraints should be confirmed by business and technical owners and固化 into tests.
6. How to Build a Trustworthy Model Evaluation System
6.1 Evaluation Tasks Should Cover the Full Engineering Lifecycle
Comparing only whether a single answer is correct tends to overestimate model capability. More valuable evaluations should include the following:
- When information is insufficient, does the model ask key clarifying questions?
- After reading existing code, can it identify hidden constraints?
- When modifying a localized feature, does it break existing interfaces?
- Facing concurrency issues, can it construct failure timelines?
- When tests fail, can it locate the root cause instead of trial-and-error patching?
- When evidence is lacking, does it explicitly state its uncertainty?
Test sets should come from real projects wherever possible, with sensitive information removed. Different models should be given as consistent as possible context, tool permissions, and completion budgets.
6.2 Evaluate Process Cost, Not Just Results
The true cost of a task can be summarized as:
Total task cost = model usage cost + human clarification cost + review and rework cost + failure risk cost.
A low unit price does not necessarily mean low total cost. If a model keeps going down wrong paths, it consumes not only API fees but also developer attention and delivery time.
Likewise, an expensive model need not be used for every step. Its extra cost is justified only when it genuinely reduces high-value human effort or significantly lowers high-risk errors.
Recommended metrics include first-pass acceptance rate, number of human corrections, end-to-end elapsed time, average task cost, and the count of severe defects. For tasks where results vary across runs, observe stability rather than keeping only the best run.
7. Replace “One Model Does Everything” with Layered Collaboration
Enterprises can allocate model resources according to task risk and uncertainty:
- For requirement clarification, design review, and complex fault analysis, prioritize models with stronger constraint-reasoning capabilities;
- For template generation, well-defined interface implementations, and routine test additions, use lower-cost models that have been validated;
- For permission changes, data deletion, and database schema modifications, retain human confirmation;
- High-risk code must pass independent testing and review—it should not be automatically approved just because the generation process looked smooth.
The key to layered collaboration is handoff material. An upstream step should not output only “implement according to this plan,” but should provide interface contracts, business invariants, failure handling, prohibitions, and acceptance test cases.
In addition, having the same model review itself is not equivalent to independent verification. Deterministic checks, concurrency testing, and human review remain irreplaceable.
8. Compliance and Deployment Fit Are Prerequisites for Enterprise Adoption
Whether an enterprise can use a given model depends on data classification and grading, industry regulations, procurement policies, and internal security rules. It cannot be simplified to “a certain type of organization may always or never use a certain type of service.”
A practical evaluation should cover at least:
- Whether inputs contain personal information, trade secrets, or sensitive code;
- Whether data retention, logging, and deletion mechanisms are clearly defined;
- Whether the service’s deployment location and access scope meet requirements;
- Which tools the model can invoke, and whether least-privilege principles are followed;
- Whether generated code, commands, and data modifications go through approval steps;
- Whether the full pipeline is auditable, traceable, and reversible.
Even when a model is deployed in an environment controlled by the enterprise, it is not inherently secure. Untrusted instructions from documents, code comments, or external content can still interfere with model decisions. Retrieval content must therefore be separated from execution permissions, and high-risk actions must go through independent validation.
9. Conclusion: Verifiable Delivery Capability Is What Truly Matters
Evaluating Chinese LLMs should not remain an abstract debate about superiority, nor should parameter counts, leaderboard rankings, or pricing tiers be treated as the final answer.
For real engineering work, what matters more is whether the model can identify ambiguities in requirements, understand constraints among business objects, propose appropriate concurrency and failure-handling designs, and consistently pass acceptance criteria at a controllable cost.
When a model moves from “quickly generating a piece of code” to “helping the team maintain business contracts, reduce rework, and protect system boundaries,” its engineering value truly materializes. Reliable model selection should be built on exactly this kind of evidence.
