Selecting and Evaluating Chinese Programming LLMs: A Practical Guide for Engineering Workflows

    This article presents a practical framework for selecting and evaluating Chinese programming LLMs in real software engineering workflows. It defines seven evaluation dimensions, proposes a repeatable test process with four task types and measurable metrics, examines selection priorities for backend development, debugging, refactoring, and frontend work, and discusses cost, private deployment, common pitfalls, and a final selection strategy.

    Chinese programming LLMs have advanced rapidly in code generation, program understanding, automated debugging, and software engineering assistance. Unlike early comparisons focused on whether a model could generate a code snippet, developers now care about more practical questions: Can the model understand an existing project? Can it complete multi-file changes reliably? Can it reduce rework? Can it meet data security and cost constraints?

    For these reasons, judging whether a programming LLM is genuinely useful cannot rely only on public benchmarks or a single question-and-answer session. A more reliable approach is to place the model inside a real development workflow and evaluate it continuously across requirements analysis, code implementation, test writing, troubleshooting, and code review.

    1. Define What “Useful” Means

    Different users have different expectations for programming models.

    Individual developers often prioritize response speed, ease of use, code completion quality, and low-cost API calls. Enterprise engineering teams place more emphasis on project-level context understanding, permission control, audit capabilities, and team collaboration. Large organizations or public-sector users must also consider intranet deployment, data isolation, local hardware and software compatibility, and long-term operations and maintenance costs.

    “Useful” can be broken down into seven dimensions:

    1. Code correctness: Whether the generated code compiles, runs, and satisfies the requirement.
    2. Requirement understanding: Whether the model can identify implicit constraints, boundary conditions, and exception flows.
    3. Engineering completeness: Whether it can update configuration, interfaces, data structures, tests, and documentation together.
    4. Complex problem reasoning: Whether it can provide verifiable solutions for concurrency, transactions, performance, and security issues.
    5. Context retention: Whether it forgets key constraints across multi-turn conversations and larger codebases.
    6. Interaction efficiency: Whether response speed, modification granularity, and communication cost fit daily development.
    7. Deployment and governance: Whether it supports permission control, log auditing, data masking, and private operation.

    Code correctness is only a baseline metric. A model that produces elegant code but frequently omits exception handling or breaks existing interfaces may be less valuable in a real project than a model with fewer features but greater stability.

    2. Build a Consistent Practical Evaluation Process

    For version comparison or solution selection, avoid relying on scattered online reviews. Build your own test set instead.

    2.1 Prepare Four Types of Test Tasks

    The first type is short code generation. Examples include a data conversion function, a validator, pagination logic, or a simple API. This helps observe syntax correctness and basic coding style.

    The second type is module-level development. Give the model an interface definition, data structures, and business rules, and ask it to complete the service layer, persistence layer, and unit tests. This type of task reflects the model’s understanding of multi-file relationships.

    The third type is legacy code modification. Provide an old module with defects and require the model to fix the problem without changing the public interface. Focus on whether the model refactors arbitrarily, removes compatibility logic, or introduces new side effects.

    The fourth type is complex fault localization. Provide logs, call chains, key code snippets, and reproduction conditions, and ask for root cause analysis, a fix plan, and verification steps. This often reveals more engineering value than code generation alone.

    2.2 Use Consistent Input Constraints

    When testing different models, keep requirement descriptions, code snippets, output formats, and tool permissions as consistent as possible. Otherwise, the results may reflect differences in prompts rather than differences in model capability.

    It is advisable to require the model to:

    • State its understanding of the requirement first;
    • List the modification scope and potential risks;
    • Then output code or a patch;
    • Finally provide a test method;
    • Mark assumptions where uncertain instead of fabricating details.

    This workflow reduces answers that look complete but are unusable in practice.

    2.3 Record Quantifiable Metrics

    A scoring table can be established as follows:

    Evaluation Item Focus
    Compile or run success rate Whether the code passes basic verification
    First-pass success rate Whether multiple rounds of modification are needed
    Test coverage completeness Whether normal, exception, and boundary scenarios are covered
    Modification accuracy Whether only necessary files are changed
    Defect introduction rate Whether existing functionality is broken
    Explanation verifiability Whether reasoning can be verified through logs or tests
    Response efficiency Number of rounds and waiting time required to complete the task
    Security and compliance Whether sensitive information, injection risks, or unsafe implementations exist

    For teams, first-pass success rate and defect introduction rate are often more important than single-generation speed, because rework costs can quickly cancel out the efficiency gains delivered by the model.

    3. Selection Priorities Across Engineering Scenarios

    3.1 Enterprise Backend Development

    In enterprise backend projects, the model needs to understand API protocols, database schemas, transaction boundaries, permission models, and exception conventions at the same time. Generating controller code alone is not difficult. The real difficulty is maintaining consistency across layers.

    During evaluation, focus on whether the model can:

    • Add new APIs following existing code style;
    • Handle parameter validation and exception mapping correctly;
    • Understand transaction commit and rollback boundaries;
    • Avoid duplicate queries and unnecessary remote calls;
    • Add executable tests for core logic;
    • Update callers when an interface changes.

    In practice, ask the model to work in small commits. Analyze the impact scope first, modify one feature point, run tests, and then continue. Asking the model to rewrite an entire business module in one pass often increases uncontrolled changes.

    3.2 Algorithms, Low-Level Systems, and Difficult Troubleshooting

    For concurrency, memory, networking, compiler, and performance issues, the challenge is not writing code but building the correct causal chain.

    A high-quality troubleshooting process should generally include:

    1. Distinguish reproducible issues from intermittent ones based on symptoms;
    2. Extract evidence from logs, metrics, and call chains;
    3. Propose multiple possible causes;
    4. Design minimal verification experiments;
    5. Explain the side effects of the fix;
    6. Provide regression testing and monitoring recommendations.

    If the model jumps directly to a conclusion without explaining the evidence, developers should be cautious. For production incidents, the model is better used as an analysis assistant than as a replacement for human decision-making.

    3.3 Large-Scale Refactoring and Code Audit

    Large projects are most vulnerable to “locally correct, globally out of control” problems. A model may optimize a function while breaking interface compatibility, configuration loading, or historical data processing.

    When refactoring, the task should be split into:

    • Mapping module dependencies;
    • Identifying public interfaces;
    • Establishing a behavior baseline;
    • Writing regression tests;
    • Replacing implementations in small scopes;
    • Comparing performance and error rates;
    • Cleaning up deprecated code last.

    Code audits should focus on permission checks, input filtering, sensitive information handling, deserialization, file access, SQL concatenation, and log leakage. A model’s statement that “a risk exists” cannot be treated as a conclusion. It must be combined with static analysis, testing, and human review.

    3.4 Frontend and Full-Stack Development

    Frontend tasks often involve strong visual and interaction requirements. Models can quickly generate page structures, but they often miss details in responsive layout, state management, accessibility, and complex interactions.

    During evaluation, provide the page structure, interaction flow, and design constraints, and require the model to output:

    • A component breakdown plan;
    • A state flow description;
    • Key event handling logic;
    • Empty states and error states;
    • Mobile adaptation strategy;
    • Basic test cases.

    If screenshot reproduction is involved, do not compare visual similarity alone. Also check whether the page remains stable across different resolutions, data volumes, and error states.

    4. Trade-Offs Between Low-Cost Calls and Private Deployment

    For high-frequency, low-complexity tasks, choose lightweight models with fast responses and lower resource consumption, such as code completion, comment generation, simple test writing, and format conversion.

    For cross-file modification, complex debugging, and architecture analysis, prioritize reasoning capability and context stability. In real systems, a common approach is layered routing:

    • Simple completion goes to a lightweight model;
    • Module-level tasks go to a general code model;
    • Complex faults and architecture questions go to a high-reasoning model;
    • High-risk changes must enter a human review process.

    Private deployment requires additional evaluation of GPU memory requirements, concurrency capacity, model update methods, log retention, access permissions, and failure recovery. Do not compare only model quality. Calculate the total cost of hardware, operations and maintenance, monitoring, and security governance.

    If using Baidu AI Cloud or other general cloud capabilities, decide the invocation method based on data sensitivity. Ordinary code can use cloud assistance. Content containing keys, customer information, or core business rules should be masked first or processed in a controlled environment.

    5. Common Pitfalls

    Pitfall 1: Looking Only at Public Benchmarks

    Benchmarks usually reflect performance on standard tasks and may not indicate whether a model fits your technology stack and coding standards. The most reliable conclusions come from your own codebase and real tasks.

    Pitfall 2: Treating Output Length as Capability

    Longer answers do not mean deeper analysis. A good engineering answer should have clear boundaries, executable changes, and verifiable risks.

    Pitfall 3: Ignoring Testing and Review

    Model-generated code must go through compilation, unit testing, integration testing, and security checks. Code involving payments, permissions, data deletion, and concurrency control must not be deployed directly.

    Pitfall 4: Opening the Entire Codebase at Once

    More context is not always better. Use directory summaries, relevant file filtering, and staged loading to reduce interference from irrelevant information while controlling the exposure of sensitive data.

    Individual developers can prioritize solutions with stable responses, natural code completion, and controllable usage costs, and use fixed prompt templates to improve output consistency.

    Small development teams should establish a shared evaluation set and record each model’s success rate, rework count, and defect situation in projects, rather than relying on subjective impressions.

    Medium and large enterprises should incorporate model selection into engineering governance. This should include at least permission management, data classification, operation auditing, code scanning, human approval, and effectiveness review. A model is not a standalone tool but an intelligent node in the development process.

    Overall, Chinese programming LLMs are moving from “assisting with a few lines of code” toward “participating in the software engineering process.” The best choice is not necessarily the model with the highest score on a single capability, but the one that balances the target technology stack, team workflow, security boundaries, and cost budget. Only by placing models into real projects and validating them continuously with consistent tasks can individuals and enterprises reach conclusions that are genuinely useful for decision-making.