Chinese Coding LLM Selection: Version Comparison and Testing Guide
This guide explains how engineering teams can compare Chinese coding LLMs and their versions without relying on subjective rankings. It covers defining task-specific goals, building reproducible evaluations from historical work items, separating model changes from workflow changes, measuring effective delivery, and rolling out model assistance with task routing and rollback controls.
1. Define “usefulness” first: which step does the model save for the team?
The same coding LLM may be good at suggesting a few lines during inline editing, yet unsuitable for independently modifying cross-module logic. It may also produce a more complete patch for complex tasks, but respond too slowly for frequently triggered inline suggestions. Therefore, “which model is most useful” is first a workflow question, not a single ranking question.
Existing materials may list candidate versions, but they often lack reproducible runtime logs, patches, test results, or human review records. That is not enough to produce a measured ranking, nor can version numbers be used to infer engineering capability. A more reliable approach is to have candidate versions work on the same batch of real tasks and observe whether they can deliver mergeable code under the team’s existing permission, latency, and cost constraints.
The evaluation goals can first be split by usage scenario:
| Use case | Primary observation | Easily overlooked cost |
|---|---|---|
| Inline completion and boilerplate generation | First-response latency, suggestion acceptance rate | Frequent deletions or edits to suggestions may interrupt the coding flow |
| Generating a local feature from requirements | Acceptance criteria coverage, edge-case handling | Only the main flow is implemented, while null values, duplicate requests, or invalid inputs are missed |
| Fixing defects based on a repository | Root-cause localization, patch scope, regression results | Behavior is not fixed; only test expectations are changed or errors are hidden |
| Cross-module refactoring | Caller coverage, compatibility, reviewability of the patch | Public interfaces are changed casually, increasing migration and rollback costs |
| Adding tests and technical documentation | Assertion quality, consistency between documentation and implementation | More tests are added, but existing errors are still not detected |
These metrics are not interchangeable. An accepted inline suggestion does not mean the generated code will remain in the final change. A passing regression suite does not prove that the patch stayed within the intended change boundary. Teams should first decide which tasks are worth assigning to the model, and then decide what to measure for those tasks.
2. Build a reproducible evaluation from historical work items
1. Fix repository snapshots and avoid giving the model the answer
Prefer sampling from completed historical work items. For each task, fix the repository snapshot from before the fix and retain the requirements, error logs, interface conventions, and related documentation that developers could see at the time. Merged patches and their acceptance tests should not be provided directly as model input; otherwise, the evaluation may become a test of whether the model can restate an existing answer.
Historical patches also do not need to be the only standard. A new solution can be considered valid as long as it satisfies behavioral constraints, passes independent validation, and has reasonable maintenance cost.
Tasks should cover local implementation, defect localization, cross-file changes, and cases where failure is expected. For example, some work items lack a key business rule. The correct behavior is to point out the ambiguity and ask for clarification, rather than guessing a default value. Including such tasks helps distinguish whether a model truly recognizes an information gap or is merely good at generating code when the task statement is complete.
2. Turn “done correctly” into checkable items
Consider an illustrative task: a list query currently returns an error when no data matches. The requirement is to change it to return a normal empty result while preserving the pagination fields and their meaning.
This task can define four acceptance criteria in advance:
- The empty result follows the existing response structure.
- Pagination behavior remains unchanged when data exists.
- Data-read exceptions still follow the existing handling path.
- The patch does not change public interfaces or rewrite code unrelated to the problem.
For the third criterion, separate validation cases should be added. Otherwise, a model that converts all exceptions into an empty result may pass tests that only cover the normal path.
The value of this sample task is not the amount of code involved. It can verify whether the model distinguishes between “no data found” and “read failure.” It also shows why comparing two versions with a vague instruction such as “fix the list query exception” is insufficient: if the acceptance boundary is unclear, reviewers may end up comparing only how each model guessed the requirements.
3. Keep records sufficient to support conclusions
For each task, at least record the task ID, repository snapshot, model version, input materials, available tool permissions, elapsed time, generated patch, validation results, and reasons for human intervention.
Reviewers can categorize failures into several types:
- The model did not find the relevant code.
- The model found the right location but misunderstood the business rule.
- The model made the correct modification but missed callers.
- Validation was insufficient.
- The change exceeded the intended scope.
When possible, hide version identifiers during human review to reduce preconceptions about older or newer versions.
A convincing trial report should be traceable from “a certain type of task performed better” to specific patches, failure categories, and rerun conditions. Showing only a response that looks reasonable cannot prove that a model can reliably complete engineering delivery.
3. Compare new and old versions: separate model changes from workflow changes
When evaluating a version upgrade, the important question is not whether the answers are longer, but whether the failure modes for the same types of tasks have changed. For example, if an older version often locates the wrong modification point, while a newer version locates it accurately but introduces more unrelated changes, it should not be judged as an overall upgrade. For teams that handle frequent small fixes, the cost of a larger change scope may outweigh the benefit of better localization.
A two-round comparison is recommended:
- Fixed conditions: compare the model itself. Use the same tasks, repository snapshots, prompt templates, input materials, tool permissions, and runtime limits. When conditions allow, run the same task repeatedly to see whether results are stable. Timeouts, environment failures, or validation failures caused by factors outside the model should be recorded separately and not directly attributed to code quality.
- Actual configuration: compare the full workflow. Allow each version to use the intended context retrieval, stepwise prompting, or automated validation approach, and record the entire process from receiving the task to obtaining a reviewable patch. If the benefit comes from more accurate file retrieval or better test feedback, the result should be attributed to the whole workflow, not only to the model version.
When comparing long-context capability, the same task can also be prepared with two sets of inputs: one containing only the located relevant files, and another adding callers, specifications, and other background materials. If errors decrease significantly after key callers are added, improving the supply of materials should be the first priority. If the key information is already complete and the model still repeatedly modifies public behavior, then there is reason to point to code understanding or constraint following. Receiving more context does not mean the model can accurately identify the conventions that must be followed.
4. Measure “effective delivery,” not the first response
Before evaluation, the definition of “effective delivery” should be fixed: the patch passes necessary validation, does not violate hard constraints, and requires human effort within a range preset by the team. This avoids the situation where a model outputs a half-finished product, developers complete it, and the result is still counted as a success.
Core metrics can be divided into three groups:
- Quality metrics: first-pass effective delivery rate, effective delivery rate within a limited number of retries, regression pass rate, and number of out-of-scope changes. First-pass and retry results should be reported separately so that multiple attempts do not hide first-pass quality.
- Efficiency metrics: first-response time, total time from submission to receiving an effective patch, and human review and rework time. For interactive scenarios, slow requests should also be tracked, not only average latency; a small number of long waits can noticeably interrupt developers’ workflow.
- Cost metrics: inference resources, validation resources, and human effort. If the three can be converted into the same currency unit, divide the total input for a batch of tasks by the number of effective deliveries, and include failed attempts in the numerator. If reliable conversion is not possible, report resource costs and human minutes separately. If there is no effective delivery, report directly that the batch produced no effective results and do not calculate unit cost.
Human baselines should also be set for each task type. For example, developers may already locate a small issue quickly on their own. Even if the model responds faster, if developers need to read a long patch, there may be no net benefit. In contrast, cross-module changes may take longer to generate, but if they reduce the time spent finding call relationships and repeatedly filling gaps, they may still be worth adopting. When reporting conclusions, list the sample size and the number of successes and failures. When the sample is small, avoid presenting differences observed in a few work items as stable advantages.
5. From offline results to rollout: route tasks and preserve fallback paths
After offline results meet expectations, the model can first be trialed on a small number of real work items, with developers deciding whether to adopt its suggestions. Start with tasks that have clear boundaries, sufficient validation, and easy rollback, such as local test additions and defect fixes involving non-public interfaces.
For changes involving public contracts, data migration, or core business rules, require an impact scope and validation plan first. Only after confirmation should the model generate the patch, rather than assuming it will complete the entire change in one pass.
The integration environment should follow least privilege: by default, provide the model only with the repository content needed to complete the task, and avoid including secrets or unrelated business data in the input. Generated patches should be validated in an isolated environment and should not be written directly to protected branches or automatically enter production. After rollout, track human rework, regression failures, and timeouts by task type. If a new version continues to regress on high-frequency tasks, roll back the configuration for those tasks first, rather than forcing a full upgrade because other tasks improved.
The final selection conclusion should ideally be written as an executable rule: which version is used by default for which task type, what conditions trigger a switch to another version or a return to human review, and which signals trigger a rollback. A conclusion that can be rerun with the same batch of work items, explained with actual patches, and validated by delivery cost is more reliable than a leaderboard detached from the team’s workflow.
