Choosing and Upgrading Chinese Coding LLMs: A Practical Evaluation Guide

    A reliable coding model is not the one that produces the most fluent answers, but the one that consistently delivers patches that pass tests, follow existing conventions, and can be safely merged. This guide explains how to evaluate coding LLMs by task type, build reproducible version comparisons, measure quality and total cost, and roll out model upgrades without disrupting established engineering workflows.

    1. Define “Useful”: A Shippable Patch Matters More Than a Fluent Answer

    When evaluating coding LLMs, teams often overvalue the first impression: fast responses, detailed explanations, and plausible examples. In real engineering work, however, developers need patches that pass tests, follow existing conventions, and can be safely merged. A model that generates code quickly but repeatedly edits the wrong files, introduces unrelated dependencies, or requires developers to rewrite critical logic may not save time at all.

    Model selection should therefore start with the team’s actual workload rather than with a universal ranking. A team maintaining a mature business system may prioritize defect localization and narrowly scoped changes. A team building a new service may care more about API implementation, test generation, and adherence to engineering standards. Even when using the same model version, these teams may reach different—and equally reasonable—conclusions.

    Without verifiable test records, it is difficult to claim that a particular vendor or version is “the best.” A credible evaluation should answer four questions:

    • What task was performed?
    • What information was available to the model?
    • How was the patch accepted?
    • How much time did the developer spend completing the work?

    2. Evaluate Engineering Tasks Separately to Avoid Hidden Weaknesses

    Scenario Example evaluation tasks What to observe
    Local completion and routine changes Add parameter validation, convert fields, complete exception branches Whether the first output can be adopted directly and whether latency disrupts the editing workflow
    Defect localization and repair Fix business logic based on error logs and a minimal reproduction case Whether the model identifies the root cause instead of swallowing exceptions or bypassing failed checks
    Cross-file changes Change API fields and update callers, data transformations, and tests Whether all affected paths are covered while compatibility requirements are preserved
    Constrained refactoring Split a long function or consolidate duplicate logic while preserving external behavior Whether the changes are necessary and whether key behavior remains consistent before and after refactoring
    Test completion Add tests for boundary values, retry failures, and state transitions Whether the new tests fail when the defect exists rather than merely validating the current implementation

    It is also important to separate model capability from capability provided by the integration. For example, one candidate may be able to search the entire repository while another can access only the current file. Results on cross-file tasks cannot be attributed solely to the underlying model in that situation.

    If the goal is to compare model versions, keep the context, tool permissions, and modification budget as consistent as possible. If the goal is to decide which complete solution should be deployed, it is reasonable to compare the actual configurations, but the conclusion should clearly apply to the entire workflow rather than to the model alone.

    Long-context performance should not be judged only by how much text a model can accept. A more useful method is to prepare three input levels for the same task:

    1. The necessary files
    2. The necessary files plus related modules
    3. The necessary files plus a larger amount of irrelevant material

    Then observe whether the generated patch still follows interface constraints and accurately limits the modification scope. In engineering work, the valuable capability is finding and applying important constraints within a large input—not simply accepting more text.

    3. How to Build Reproducible Version Comparisons

    3.1 Build the Evaluation Set from Historical Issues

    Start with closed issues that can be reproduced and have clear acceptance criteria. Use the code snapshot from before the fix as the starting point. Provide the model with the issue description, representative logs, and the context required to complete the task, but do not provide the final patch.

    The historical patch can help reviewers understand the original design, but the model should not be required to reproduce it line by line. Any implementation that satisfies the same behavioral constraints should be considered valid.

    The evaluation set should not contain only typical success cases. Include tasks with complex dependencies, unclear error signals, and strict compatibility requirements. Record results by task type, affected module, and risk level. A high pass rate on simple changes cannot offset a change that breaks data compatibility on a critical path.

    3.2 Fix the Environment and Separate First Attempts from Iterative Repairs

    Use the same code snapshot, prompt template, dependency environment, available tools, and execution permissions for both old and new versions. Record the identifiable version, test date, context scope, retry count, and time budget.

    When a task involves automated editing or test execution, start each run in a clean isolated environment. This prevents files or state left by an earlier attempt from affecting the next one.

    It is useful to report two categories of results separately:

    • First output: Measures how well the model understands the task and follows constraints without feedback.
    • Final output within the allowed limit: Measures whether the model can effectively correct its work after receiving compiler, test, or review feedback.

    Looking only at the final result can hide the amount of rework required. Looking only at the first attempt can underestimate versions that perform well in interactive collaboration. For tasks with substantial output variation, run the evaluation multiple times and report the number of successful runs along with typical failure causes. Do not use the best run as the representative result.

    3.3 Replace Impressionistic Scores with Explicit Acceptance Criteria

    Each task should ideally define requirements in three categories:

    • Behavior: What the software must do
    • Scope: What must remain unchanged
    • Engineering quality: How the change should be validated and maintained

    Consider the requirement that repeated submissions of the same business request must not create duplicate records. Behavioral acceptance should cover the first request, sequential duplicate submissions, and concurrent duplicate submissions. Scope acceptance might require unrelated API response formats to remain unchanged. Engineering-quality acceptance should verify that the added tests can reliably reproduce the defect before the fix.

    This example also shows why passing existing tests is not enough. A patch may prevent duplicates during sequential retries while still writing duplicate records when two requests arrive at the same time. For concurrency, state consistency, and API compatibility tasks, critical counterexamples should be mandatory checks. They should not be offset by high scores for code style or explanation quality.

    4. How to Measure Efficiency, Quality, and Cost

    Quality should first be measured by what passes. At a minimum, distinguish among:

    • Whether the project builds
    • Whether the task-specific acceptance criteria pass
    • Whether existing regression checks pass
    • Whether the patch passes human review

    For failed samples, record the primary cause, such as misunderstanding the requirement, uncontrolled modification scope, omitted callers, insufficient test coverage, or unrelated dependencies. Failure reasons are more useful than a single pass rate when planning improvements. Missing context may indicate that the input workflow needs adjustment, while repeatedly ignored constraints may indicate that the model’s suitable scope should be narrowed.

    Efficiency should distinguish waiting time from human effort. For inline editor completion, generation latency directly affects usability. For complex defect repairs, the more important measure is the end-to-end time from task assignment to a reviewable patch, as well as the time developers spend preparing context, checking logic, and correcting the result.

    A candidate that responds more slowly but delivers an accurate patch in one attempt may require less human effort than a faster candidate that needs frequent correction. If possible, establish a manual baseline using comparable tasks without model assistance. This helps prevent improvements caused by environment tuning or developer familiarity from being incorrectly attributed to the model.

    Calculate cost per completed task. For the same evaluation set, add model calls, retries, and additional verification costs, then divide the total by the number of tasks that ultimately pass acceptance. Failed attempts must be included in the numerator.

    Human review and rework can be listed separately or converted using a consistent team-wide method. This makes it possible to explain why a solution with a lower per-call price may cost more on complex tasks. When the sample size is small, report both the number of passing tasks and the total number of tasks. Do not interpret a difference from a few samples as stable leadership.

    5. Model Upgrades Do Not Require an Immediate Full Migration

    A new version will not necessarily outperform the old version on every task type. An upgrade report can be organized by:

    • Task type
    • Old and new pass results
    • Changes in human effort
    • Newly observed failure modes

    Pay particular attention to cases where the old version passed but the new version failed. If the new version tends to expand the modification scope, ignore existing interface conventions, or rewrite tests to accommodate an incorrect implementation, it may not be suitable as a direct replacement—even if its overall pass rate improves.

    For deployment, begin with offline replay using historical tasks, then pilot the new version on a small number of low-risk issues. For local completion and routine changes, prioritize response time and direct adoption rate. For cross-module changes, require the model to list the affected scope and validation plan before generating a patch. For changes involving data migration, permission checks, or critical state transitions, retain human review at all times.

    The model may run approved checks in an isolated environment, but it should not merge code or deploy independently. Before sending context to the model, remove credentials, personal information, and internal materials unrelated to the task.

    Define pause and rollback conditions before the pilot begins. Examples include failures in critical regression tests, a sustained increase in review effort, or file modifications beyond the agreed scope. Keep the old configuration and evaluation records available. This makes it easier to determine whether a problem comes from the model change, prompt template, code retrieval, or execution environment—and to quickly restore a validated workflow.

    The final selection does not need to be a decision to use one model across the entire organization. A more practical strategy may be to use a fast, high-adoption option for routine low-risk work, assign complex changes to the model that performs most consistently on the relevant acceptance set, and retain human approval for critical modifications.

    When evaluations are reproducible, rework is included in cost calculations, and upgrades can be rolled back, the question of which model is “more useful” becomes a verifiable engineering decision rather than a matter of subjective impression.