A Practical Guide to Selecting and Evaluating Chinese Programming LLMs

    This guide provides a practical framework for evaluating and selecting Chinese programming Large Language Models (LLMs) for software development. It breaks down the misconception of simply looking for the "best" model, categorizes models by core capabilities, and proposes a unified test set covering code generation, debugging, and cross-file functionality. It also outlines key metrics to observe during actual use, offers selection recommendations for different team sizes, and provides a recommended engineering workflow and internal scoring rubric to help engineering teams make reliable, practical decisions.

    1. Why You Shouldn’t Just Look for the “Best” Model

    In real-world development, the strengths and weaknesses of a programming LLM are rarely determined by a single metric. One model might excel at generating scripts but may not be suitable for maintaining large backend systems; another might respond quickly but perform averagely in cross-file modifications, complex dependency analysis, and exception localization. Therefore, “best to use” should be broken down into several specific questions:

    1. Is the generated code accurate?
    2. Can it understand the existing project structure?
    3. Does it maintain the original logic when modifying code?
    4. Can it autonomously fix compilation errors?
    5. Can it handle long contexts?
    6. Are the output speed and invocation costs suitable for long-term use?
    7. Does it support private deployment, permission control, and audit requirements?

    From an engineering perspective, model selection should not rely on marketing materials or a single trial experience, but should be validated against the team’s tech stack, code scale, and delivery goals.

    2. How Mainstream Models Can Be Categorized by Capability

    Currently, Chinese programming LLMs can be roughly categorized into four types based on their positioning.

    The first category is General Flagship Models. These models typically possess strong natural language understanding, complex problem analysis, and multi-turn reasoning capabilities. They are suitable for architecture design, difficult troubleshooting, technical solution reviews, and cross-module refactoring. However, their response speed and usage costs are often inferior to lightweight models. They are best suited for solving high-value problems rather than handling all completion tasks.

    The second category is Code-Specific Models. These models are primarily optimized around code generation, completion, test writing, bug fixing, and code explanation. They usually offer good efficiency for common languages, standard frameworks, and repetitive development tasks. They are suitable for integration into editors, code review processes, or automated R&D platforms.

    The third category is High-Speed Lightweight Models. The advantages of these models are fast response, low invocation costs, and suitability for high-frequency use. Daily code completion, comment generation, simple query transformations, interface boilerplate code, and initial unit test drafts can be assigned to them. However, when dealing with large projects, complex business rules, and implicit dependencies, manual review is still required.

    The fourth category is Long Context Models. They emphasize reading a large amount of code, documentation, logs, and configuration files at once, making them suitable for project-level analysis. However, supporting a longer context does not necessarily mean it can accurately understand all the content. If the input materials lack directory structures, module boundaries, and key constraints, the model may still miss the point.

    3. Recommend Using a Unified Test Set for Version Comparison

    If you want to compare models of different generations or different positioning, it is recommended to establish a test set sourced from real scenarios with layered difficulty, containing at least the following tasks.

    1. Basic Code Generation

    Prepare several clear requirements, such as implementing data validation, pagination queries, file processing, and permission checks. Focus on whether the code can run directly, whether boundary conditions are omitted, and whether it complies with the project’s existing coding standards.

    2. Legacy Code Understanding

    Provide a set of existing modules and only ask the model to explain the execution flow, key dependencies, and potential risks, without requiring immediate code modification. This step helps determine if the model truly understands the code rather than guessing conclusions based on keywords.

    3. Defect Localization and Repair

    Prepare code containing null handling errors, concurrency issues, unreleased resources, and missing exception branches. Ask the model to first explain the cause of the problem, then provide a minimal modification plan. Excellent models usually distinguish between “surface errors” and “root causes” and do not indiscriminately rewrite the entire file.

    4. Cross-File Feature Development

    Ask the model to add a complete feature, such as a new query interface, involving routes, business logic, data access, configuration, and test files simultaneously. The evaluation focus is not the volume of code, but whether the modifications are complete, whether naming is consistent, and whether existing interfaces are broken.

    5. Automated Test Generation

    Have the model supplement unit tests and exception scenario tests based on existing implementations. It is necessary to check whether the tests actually cover business branches rather than just generating a few superficial success cases.

    6. Log and Fault Analysis

    Input desensitized runtime logs, configuration snippets, and related code, and ask the model to provide a troubleshooting path. This task reflects the model’s ability to process unstructured information, establish causal relationships, and propose validation steps.

    4. Key Metrics to Observe During Actual Use

    Code correctness is the primary metric, but you shouldn’t just look at “whether it runs.” You should also focus on the following aspects:

    First, the scope of modification. Does the model only adjust necessary files? Does it arbitrarily change frameworks, directory structures, or key dependencies? For enterprise projects, making fewer erroneous changes is often more important than generating more code.

    Second, constraint adherence. Real-world requirements usually contain many restrictions, such as not changing the interface format, ensuring backward compatibility, or preserving transaction boundaries. Whether the model can consistently adhere to these constraints directly impacts deployment efficiency.

    Third, explanation and verification capability. Can the model explain the reasoning behind its solution, proactively point out uncertainties, and suggest further confirmation through testing, logging, or static checks? Answers that only provide conclusions without a verification path have limited engineering value.

    Fourth, long-conversation stability. After continuous multi-turn modifications, does the model forget previous requirements, show contradictions, or repeatedly reintroduce already fixed issues? Stability is usually more important than a single stunning response.

    Fifth, Chinese requirement understanding capability. Chinese development teams extensively use Chinese requirements, internal abbreviations, and business terms. Therefore, the model must not only know how to write code but also accurately identify the meaning of roles, states, permissions, and processes in the context.

    5. Selection Recommendations for Different Development Scenarios

    Individual developers focus more on speed, ease of use, and comprehensive capabilities. Daily work can adopt a combination of “lightweight models for completion, general models for complex problems,” avoiding assigning all tasks to high-spec models.

    Small R&D teams should prioritize solutions with stable code generation, strong debugging capabilities, and compatibility with common tech stacks. At the same time, they should establish code review rules to clarify what can be auto-merged and what requires manual confirmation.

    Large teams should focus on permission isolation, data protection, invocation auditing, private deployment capabilities, and R&D process integration. Even if a model provides high-quality answers, if it cannot meet code security and compliance requirements, it is not suitable for direct integration into core repositories.

    For legacy systems, long context and code understanding capabilities are often more important than pure generation speed. For repetitive scripts and interface boilerplate, high-speed lightweight models may offer a higher return on investment.

    A reliable usage workflow can be divided into six steps:

    Step 1: First, let the model summarize the project structure without immediately asking it to modify code.
    Step 2: Clarify task goals, immutable constraints, input/output formats, and acceptance criteria.
    Step 3: Require the model to provide an implementation plan first, then modify in phases, avoiding rewriting large amounts of files all at once.
    Step 4: Execute compilation, static checks, and automated tests immediately after each modification.
    Step 5: Feed error logs and actual results back to the model, allowing it to continue fixing based on facts rather than speculation.
    Step 6: Developers complete the final review, focusing on security, exception handling, resource release, concurrency control, and compatibility.

    It must be emphasized that models are suitable for “analysis, generation, and assisted verification,” but they should not replace code review where clear boundaries of responsibility exist.

    7. How to Build an Internal Team Scoring Rubric

    A 100-point scale can be used for evaluation: code correctness (30 points), complex task understanding (20 points), debugging and repair (15 points), test generation (10 points), long context capability (10 points), response speed and cost (10 points), and security and governance capability (5 points).

    When scoring, record three results: whether the initial answer is usable, whether it is usable after one round of feedback, and how much final manual modification is required. Pay special attention to the “manual modification amount,” because if a model seemingly generates a large amount of code but developers still need to rewrite it extensively, the actual benefit is not high.

    Ultimately, there is no single answer for Chinese programming LLMs that applies to all teams. A more reasonable approach is tiered usage based on task type: pursue speed for simple tasks, pursue reasoning and stability for complex tasks, pursue context understanding for project-level refactoring, and place security governance and process integration on equal footing for enterprise applications. Only through real code, unified standards, and continuous re-testing can reliable conclusions be drawn that fit your specific R&D scenario.