Year-End Review: Evaluating the Engineering Capabilities of China’s Domestic Coding LLMs

    By the end of 2025, coding ability in general-purpose large language models can no longer be measured by whether a model can generate a correct code snippet. Real projects require models to understand requirements within limited context, organize directories, follow interface contracts, run build and test commands, read error logs, and complete multiple rounds of fixes. A more practical evaluation perspective for China’s domestic coding LLMs is therefore engineering-loop capability: whether a model can take a project from requirements to a runnable, testable, maintainable, and deliverable result, with every step verifiable. This article abstracts common findings from a cross-model evaluation without focusing on specific products or ranking brands. It explains what engineering teams should evaluate, how to design tests, and how to introduce domestic general-purpose LLMs into software development workflows.

    1. Evaluation Perspective: From Code Generation to Engineering Delivery

    A single-file code snippet can be judged by whether it appears correct. Engineering delivery requires a much broader set of metrics. At a minimum, evaluation should cover the following dimensions:

    1. Requirements understanding and task decomposition: Can the model break natural-language requirements into pages, routes, data structures, interaction states, and acceptance criteria instead of simply stacking components?
    2. Page and information-architecture completeness: Does it implement the header, main content, footer, sitemap, copyright information, empty states, error states, and loading states, rather than focusing only on the most visible part of the first screen?
    3. Responsiveness and usability: Is the interface usable on mobile devices? Does it avoid horizontal scrolling, undersized touch targets, incorrect image cropping, and text overflow?
    4. Defect-fixing ability: When static analysis, type checking, tests, or builds fail, can the model identify root causes, perform multiple rounds of fixes, and avoid introducing regressions?
    5. Maintainable project structure: Does the project go beyond a simple three-layer structure of pages, components, and styles? Does it include data access, type definitions, configuration, tests, scripts, and documentation?
    6. Tool use and automated execution: Can the model perform dependency preparation, builds, tests, and log analysis in a controlled environment, rather than merely generating a dependency manifest and leaving the rest to a human?
    7. Security and compliance boundaries: Does it respect file-access scope, command allowlists, secret protection, and dependency-introduction policies?

    Together, these dimensions determine whether model output can enter a team’s workflow. Looking only at one-time code correctness can overestimate delivery capability. Looking only at whether the project directory appears well organized can overlook actual runtime behavior and regression validation.

    2. Test Design: Fixed Environments, Fixed Tasks, and Dual-Track Acceptance

    For reproducible results, evaluations should use fixed task cards and fixed runtime environments. A task card should specify the page list, data models, interface contracts, styling constraints, mobile breakpoints, browser compatibility range, prohibited operations, and acceptance commands. Each test should record the model version, invocation date, sampling parameters, generated files, command logs, and final diff. Results from different versions should not be mixed in the same comparison.

    Acceptance should combine automated and manual evaluation:

    • Automated checks: Dependency preparation, formatting, static analysis, type checking, unit tests, end-to-end tests, and build validation.
    • Manual checks: Information-architecture completeness, visual consistency, information density on detail pages, mobile usability, code readability, and coverage of exceptional states.
    • Diff checks: Whether the model changed only the files related to the task, accidentally removed existing capabilities, bypassed tests, or weakened validation.

    A model should not be allowed to rewrite acceptance scripts during testing. Generating a dependency manifest must not be treated as equivalent to making a project runnable. Acceptance must be based on successful builds, passing tests, and inspection of the actual pages.

    3. Common Findings: Significant Progress, but Gaps in the Engineering Loop

    The cross-model evaluation showed that China’s domestic general-purpose LLMs already have strong capabilities in generating initial drafts and fixing local issues. However, stable engineering delivery remains challenging.

    3.1 Page Completeness Often Stops Halfway

    A common pattern is a completed header paired with a footer that lacks a sitemap or copyright information. Images may not have fixed aspect ratios or mobile cropping rules. Detail pages may lack sufficient information density, while product cards on the homepage may have inconsistent spacing, hierarchy, and content priorities.

    The root cause is usually not that the model cannot write components. Rather, the model tends to prioritize visible local content and pay less attention to the page-wide information architecture and global design constraints. A better approach is to express requirements as an auditable page checklist covering the header, main content, footer, sitemap, copyright information, empty states, error states, loading states, mobile folding rules, and image-size rules. The model should also be required to explain how each item was completed.

    3.2 Mobile Experiences Are Generally Usable, but Need Dedicated Checks

    Mobile compatibility was generally acceptable in testing, which is encouraging. However, “it opens” does not mean “it is easy to use.” Evaluators should still check viewport configuration, horizontal overflow, touch-target size, image ratios, long-text wrapping, obstruction caused by fixed-position elements, and keyboard accessibility.

    Mobile acceptance should be performed independently from desktop acceptance. Simply shrinking a desktop browser window is not sufficient.

    3.3 Defect Fixing Is Improving, but Patch-Based Fixes Require Review

    In one test involving a migration from a dynamically typed language to a statically typed language, more than twenty type-related and boundary issues were exposed. The fixes were completed relatively smoothly, indicating that the model performed reasonably well in local diagnosis, error explanation, and iterative correction.

    However, patch-based fixes can introduce hidden problems, including excessive type assertions, incomplete error handling, and duplicated logic. After every fix, teams must rerun type checks, unit tests, and end-to-end tests, while also reviewing changes along critical paths.

    3.4 Project Structures Are Often Too Simple for Continued Iteration

    Some generated projects contain only pages, components, and styles. This may be sufficient for a demo, but not for continuous development. Production-oriented projects should at least consider data access, type definitions, configuration, utility functions, tests, scripts, and documentation.

    An overly simple structure increases the cost of future changes and can blur responsibilities. Pages may directly handle data requests, components may take on business logic, styles may become scattered, and tests may be difficult to add later.

    A directory structure should not be complex for its own sake, but its boundaries must be clear. Pages should handle composition and routing; components should provide reusable interfaces; the data-access layer should handle requests and adaptation; the types directory should define contracts; and the test directory should support regression validation. Small projects can reduce the number of layers, but they should still preserve clear responsibility boundaries.

    3.5 Tool Use Is the Biggest Differentiator

    One of the most visible weaknesses in testing was limited tool-operation ability. In some cases, the model could not directly operate scripts to generate a complete project. Instead, it expressed dependencies indirectly by generating a dependency manifest. The manifest has value, but it only describes dependencies. It does not complete the loop of preparing the environment, running the build, reading logs, and continuing repairs based on errors.

    A more mature setup places the model in a controlled sandbox with read-only directory access, a restricted command allowlist, log collection, and rollback support. The model can be allowed to prepare dependencies, run builds and tests, and perform static checks, while network access, sensitive-file access, and high-risk commands remain restricted.

    Only an execution-observation-repair-revalidation cycle gives a model genuine engineering-agent capability.

    3.6 Version Freshness Must Be Part of the Evaluation

    Models evolve quickly, and any cross-model result represents only a specific point in time. Earlier versions may be weaker in tool use, generation consistency, or support for newer engineering practices, but that should not be generalized into a long-term conclusion.

    Teams should maintain their own evaluation baselines and continuously record model versions, dates, tasks, and outcomes. This prevents outdated findings from being used for model selection.

    4. Deployment Approach: Constrain Model Output with Specifications, Decomposition, and Quality Gates

    When introducing a domestic coding LLM into a development workflow, a five-step approach is recommended.

    Step 1: Write the specification before generating code. The specification should include the page list, route table, data models, interface contracts, global style variables, component boundaries, empty and error states, mobile breakpoints, acceptance commands, and prohibited actions. The closer the specification is to an auditable acceptance checklist, the more stable the model output is likely to be.

    Step 2: Decompose the work instead of generating everything at once. Build the skeleton first, then implement shared components, complete page composition, and finally add data, exceptional states, tests, and documentation. One-shot generation of a large project often produces locally complete but globally inconsistent results.

    Step 3: Provide strongly constrained context. This should include existing directory conventions, naming conventions, sample code, interface types, error-handling policies, dependency allowlists, and output formats. The purpose of context is not to let the model improvise freely, but to help it make reviewable changes within clearly defined boundaries.

    Step 4: Establish a secure tool sandbox. Allow the model to run formatting, static analysis, type checks, unit tests, end-to-end tests, and build validation. Restrict arbitrary network installation, access to sensitive configuration, and destructive commands. Command logs must be traceable, and failed changes must be reversible.

    Step 5: Set quality gates. A failed check should block merging. Required gates can include formatting, static analysis, type checking, unit tests, end-to-end tests, build validation, mobile critical-path checks, image and asset checks, and accessibility checks.

    5. Reference Project Structure and Quality-Gate Script

    To prevent generated projects from remaining at demo quality, teams can adopt a reference structure such as:

    1. project/
    2. ├── src/
    3. │ ├── pages/ Page and route composition
    4. │ ├── components/ Shared interface components
    5. │ ├── features/ Business feature modules
    6. │ ├── services/ Data access and API adaptation
    7. │ ├── data/ Local and mock data
    8. │ ├── types/ Type contracts
    9. │ ├── styles/ Global styles and design variables
    10. │ └── utils/ Utility functions
    11. ├── tests/ Unit and end-to-end tests
    12. ├── scripts/ Validation and helper scripts
    13. └── docs/ Requirements and change records

    A quality gate can be represented with pseudocode. The key points are the execution order and the failure policy:

    1. # Quality-gate pseudocode
    2. Run dependency preparation
    3. Run formatting checks
    4. Run static analysis
    5. Run type checks
    6. Run unit tests
    7. Run end-to-end tests
    8. Run build validation
    9. If any check fails:
    10. Block the merge
    11. Save command logs and the change diff
    12. Return an error summary to the model for narrowly scoped fixes

    The acceptance checklist can include page completeness, image dimensions and alternative text, no horizontal scrolling on mobile, keyboard-accessible critical actions, correct routing and form validation, complete loading and error states, data access kept out of page components, successful tests and builds, and complete change documentation.

    6. Conclusion: Evaluate Model Capability Together with the Engineering System

    By the end of 2025, China’s domestic coding LLMs had become practically useful for initial code generation, local defect fixing, page construction, and code explanation. They can significantly reduce the time required for boilerplate projects and repetitive development. However, information-architecture completeness, maintainable project structure, closed-loop tool use, and automated acceptance still require support from the surrounding engineering system.

    The real differentiator is not simply whether a model can write code. It is whether the model can execute, validate, and continuously repair that code in a controlled environment.

    For engineering teams, the more reliable strategy is to build an evaluation set independent of any specific product, use standardized task cards, provide a rollback-capable tool sandbox, and enforce strict quality gates. Model output should be treated as an engineering draft that requires review, not as a finished artifact exempt from inspection. This approach can improve development efficiency while keeping maintainability, security, and delivery risks under control.