Evaluating Code Generation in Open-Source Chinese LLMs: From Prototype Output to Semantic Understanding

    This article presents a practical evaluation of code generation capabilities in open-source Chinese large language models, based on a real test where a model received a requirements document for a three-module web project (login, product list, and form pages). It proposes a reusable four-level assessment framework—covering project organization, structural fidelity, semantic interpretation, and constraint handling—and discusses the strategic importance of open-source models approaching the performance of closed-source counterparts. Developers will gain actionable criteria for model selection and prompt design without relying on benchmark scores alone.

    1. Background: From “Generating Code” to “Understanding Requirements”

    Over the past two years, large language models (LLMs) have made visible strides in code generation. Early systems could only complete function fragments; today, they can turn a requirements document into a runnable front‑end project. This shift has made “engineering‑delivery capability” a key measure of practical value.

    However, one easily overlooked fact is that “code runs” and “requirements are correctly reproduced” are two different things. Many models can generate well‑structured, visually neat pages but miss filtering conditions, validation rules, or field dependencies that are explicitly stated in the document. This is the “depth of understanding” issue that matters most when testing code‑generating LLMs.

    This article is based on a real test: the model was given a requirements document covering three modules — a login page, a product list page, and a form page. No specific technology stack was mandated (the model chose React). The goal was not to score the model, but to extract a reusable evaluation framework that helps developers judge which model suits which task.

    2. Overall Experience: Solid Engineering Completeness, but Slowness Remains a Drawback

    The most notable outcome: the model did not return scattered code snippets, but a complete, directly accessible project URL. This is a meaningful achievement. For a developer, the difference between “can open it and see the result” and “must set up an environment and run it yourself” is enormous. The former means the model handled end‑to‑end project organization: directory structure, dependency management, build configuration, page routing — the “dirty work” that many generators skip.

    Two issues stood out:

    1. Concurrency and queue limits. The free tier required multiple attempts before access was granted, reflecting that inference‑resource cost remains a core bottleneck. For enterprise users, stability and concurrency often matter more than single‑shot generation quality when deciding whether to pay.

    2. Slow generation speed. The whole project took more than half an hour. For “quickly validating an idea,” this delay significantly weakens iteration productivity. An ideal prototype generator should work at minute‑level speed — letting you see a first version while your thoughts are still fresh, then enter a “edit, view, edit” loop.

    3. Scenario‑by‑Scenario Breakdown: Capabilities Are Layered

    3.1 Login Page: Structural Reproduction Is the Model’s Comfort Zone

    Login pages are typical “patterned requirements”: forms, input fields, buttons, carousels. These components appear frequently in training data, so the model can almost generate a decent version by pattern matching.

    The test confirmed this: the carousel requirement from the document was correctly implemented, the page was fully functional, and nothing obvious was missing. The result was at the level of “a few tweaks away from production.”

    This reveals a rule: the closer a requirement is to common patterns in training data, the better the model performs. Login pages, 404 pages, and basic navigation bars usually come out with high quality.

    3.2 List Page: The First Crack in Semantic Understanding

    Problems appeared on the list page. The document explicitly specified filter conditions and interaction logic, but the generated output omitted the filter controls entirely — and this was not an isolated incident, but a systematic gap across multiple list pages.

    The page’s table structure, layout, and pagination styling were correct. Only the “functional logic that is stated in text but not directly visible in the page structure” was dropped. This suggests that the model’s current attention mechanism or requirement‑parsing flow has weak capability for capturing non‑visual requirements. A filter condition is essentially a piece of interaction logic plus state management; unlike “put a button here,” it cannot be directly mapped to UI structure. The model must first extract semantics from the document and then translate that into component behavior — this translation step is a weak point in many current models.

    The lesson for developers: structural and visual requirements can be safely delegated to the model, while logical and behavioral requirements must be explicitly emphasized, ideally broken into independent items and verified one by one after generation.

    3.3 Form Page: A Test of Reproduction Under Complex Constraints

    Form pages are where models diverge most. Forms involve field definitions, required‑field validation, format checks, inter‑field dependencies (e.g., “show field B only when option A is selected”), and post‑submission state transitions. The model’s generated form ran, but many field details, validation rules, and interaction constraints were not implemented as specified in the document.

    The core issue here is not whether the code is syntactically correct, but whether the semantic mapping from requirements document to behavioral specification is complete. When a clearly written document still fails to be fully reproduced, the bottleneck lies on the model side, not the document side. Complex constraint systems (N fields × M rules) are a real stress test for a model’s instruction‑following ability.

    3.4 Visual Style: Engineering Usability Takes Priority over Polish

    The overall UI style was average — better than most rough prototypes, but still behind mature component‑library templates. The design feel was “engineering‑first,” lacking eye‑catching details. This positioning is reasonable: the model’s primary goal at this stage is correctness; visual refinement is a secondary priority. Developers can accept that the model produces the skeleton, then designers or design systems apply the final skin.

    4. A Reusable Evaluation Framework

    Abstracting from the observations above, you can assess a code‑generating LLM along four progressive levels:

    Level What to Check Typical Scenario
    L1 – Engineering Organization Can it produce a complete runnable project rather than fragments? Directory structure, build config, routing
    L2 – Structural Fidelity Can it correctly generate page scaffolds and visual components? Login pages, static pages
    L3 – Semantic Understanding Can it extract non‑visual requirements from documentation? Filtering logic, state management
    L4 – Constraint Handling Can it handle complex constraint combinations and interaction details? Form validation, field dependencies

    Most current models have solidly passed L1 and L2; the real differentiator lies in L3 and L4. When choosing a model, run a fixed requirements document through a few candidates and focus on the reproduction rate of filters, validations, and dependencies — not just how “good” the pages look.

    5. The Significance of the Open‑Source Route: Why “Closing the Gap” Matters

    Back to the opening question: when an open‑source Chinese model approaches the code‑generation capability of top closed‑source models, what does that mean for the industry?

    First, open source breaks the black box of capability. Closed‑source models can only be evaluated through black‑box API testing. Open‑source models allow the community to inspect architecture, training methodology, and inference implementation, making capability claims more trustworthy and reproducible. For enterprises, self‑hostable open‑source models also address data‑security and compliance concerns that are common blockers to cloud adoption. This makes “closely matching performance” far more valuable than a benchmark score alone.

    Second, open source accelerates capability diffusion and iteration. Improvements to a closed‑source model benefit only one company. Improvements to an open‑source model are inherited by the entire ecosystem: the community fine‑tunes it, adapts it to vertical domains, integrates it into toolchains, and in turn produces high‑quality feedback data. Code generation, which depends heavily on real‑world engineering feedback, gains especially from this virtuous cycle.

    Third, “closeness” itself signals that the competitive moat is weakening. When open‑source models are already on par at L1/L2 and narrowing the gap at L3/L4, the premium that closed‑source vendors can charge shrinks, and the industry as a whole moves toward capability democratization. Developers ultimately benefit from lower costs, more choices, and more transparent pricing.

    It is also important to stay realistic: this test shows that in semantic understanding depth and response speed, the open‑source model still had a perceptible gap from top closed‑source models, especially in complex constraint handling (L4). The value of the open‑source path does not lie in having caught up today, but in the speed of progress and the transparency of the path — these two factors matter more for the long‑term health of the domestic model ecosystem than any single score.

    6. Conclusion

    The biggest takeaway from this hands‑on test is not “model X works / doesn’t work,” but a recognition: the core challenge of LLM code generation is shifting from “can it generate” to “how deeply does it understand.” Structural reproduction is no longer a problem; what truly determines usefulness is the model’s ability to faithfully translate logical and constraint‑based content in requirement documents.

    Three practical suggestions for developers:

    1. Build your own evaluation baseline with a fixed requirement document and verify results level by level (L1–L4).
    2. Explicitly split logical requirements into separate items and check them one by one; do not assume the model “reads and instantly understands.”
    3. Include open‑source models in your shortlist, and weigh them together with private‑deployment needs and community ecosystem.

    The gap between open and closed source still exists — but the speed at which it is closing deserves the attention of every practitioner.