Evaluating Domestic LLMs for Engineering-Grade Code Generation and the Value of Open-Source Ecosystems

1. Background: When “Benchmarking Against the Best” Becomes Routine, Evaluation Should Return to the Engineering Floor

Over the past year, domestic large language models have frequently used “performance on par with top international models” as a launch highlight, and the score gaps on various benchmark leaderboards have been steadily narrowing. But for frontline developers, there is often a considerable gap between leaderboard scores and the actual experience of writing code. Whether a model can take a requirements document and fully deliver it as an accessible, runnable, maintainable front-end project matters far more than a few extra points on a multiple-choice benchmark.

This article uses a real front-end project generation task as a sample, thoroughly documenting the performance of a domestic open-source model across three typical page types—login page, list page, and form page—and providing a lateral reference against the handling level of a commonly used high-capability technical solution in the industry. It attempts to answer three questions:

1. Is the current code generation capability of open-source models at the “demo-usable” or “engineering-usable” stage?
2. Where do capability shortcomings concentrate—syntax, structure, or requirements understanding?
3. What does this capability level mean for the open-source ecosystem and AI-assisted development practices?

It should be noted that this article discusses observations at the level of general technical capability and does not constitute promotion or selection recommendation for any specific product.

2. Evaluation Design: A “Closed-Book Exam” Using a Real Requirements Document

2.1 Task Setup

The evaluation task is drawn from the front-end requirements of a real project, comprising three pages:

• Login page: includes a carousel, account/password form, login button, and basic interactions;
• Product list page: includes multi-condition filtering, list rendering, pagination, and filter-linked logic;
• Form page: includes multi-field input, inter-field linkage, validation rules, and submission interactions.

Requirements are provided as a natural language document, explicitly specifying filter conditions, field structures, and interaction logic, without limiting the technology stack. The model selects its own framework (the actual output was a component-based front-end framework solution), and the final deliverable requirement is an accessible project address, not scattered code snippets.

2.2 Observation Dimensions

To avoid vague “feels okay” evaluations, this assessment scores from four dimensions:

Dimension	Observation Points
Functional completeness	Whether all functions explicitly required in the document are implemented
Requirements fidelity	Whether details such as fields, validation, and linkage match the document
Engineering usability	Whether the project can run directly and whether the structure is reasonable
Generation efficiency	Time from submitting requirements to producing an accessible project

2.3 A Preliminary Observation: Resource Scheduling

In actual use, prompts such as “too many users, please queue or pay” appeared multiple times, and only after repeated attempts did the generation process begin. This phenomenon itself indicates: there is tension between the popularity of open-source models and actual inference resources, and availability under free quotas is not stable. This point also affects the efficiency comparison discussed later.

3. Page-by-Page Results: Structure Is Right, but Understanding Starts to Lag

3.1 Login Page: The Most Complete Portion

The login page performed best among the three pages. The carousel required in the document was correctly implemented, form elements were complete, the page was interactive overall, and there were no obvious functional gaps. From an engineering perspective, this page is at the level of “ready to ship after minor style tweaks.”

This result is not surprising. The login page is the most repeatedly implemented page type in front-end development, with abundant similar samples in training corpora. The model’s prior knowledge of “what a login page should look like” is strong enough that even with a brief requirements description, it can fill in a reasonable structure.

3.2 List Page: Systematic Discount in Functional Understanding

Upon entering the list page, problems began to surface in a concentrated manner. The filter conditions and interaction logic explicitly stated in the document were not implemented in the actual output. More notably, this gap was not occasional: it appeared repeatedly across multiple list-type page generation results, representing a common shortcoming at the capability level rather than a single random failure.

Specifically:

• The page skeleton was correct—the list area and pagination area were present;
• But filter controls were missing or had only static appearance, not linked to the data flow;
• The “filter—list—pagination” three-way linkage logic described in the document was not reproduced.

The essence of this problem is not that the model “cannot write a filter component,” but that it failed to translate the conditional descriptions in the document into a data flow design. Filtering requires the model to understand the chain of “state → query parameters → data request → view update,” whereas the model tends to downgrade it to an isolated UI element.

3.3 Form Page: Insufficient Detail Fidelity

The form page was runnable overall, but field completeness, validation logic, and interaction details deviated from the document description. Typical problems included:

• Some fields listed in the document were not generated;
• Validation rules were simplified—for example, boundary conditions for required/non-empty and format validation were not covered;
• Inter-field linkage relationships (such as one field’s value affecting another field’s available options) were not implemented.

It should be emphasized that the requirements document itself was clearly written. The problem lies in the model’s ability to parse and reproduce relationship-intensive requirements such as complex forms: the more fields and the more cross-cutting constraints, the more pronounced the decline in fidelity.

3.4 Overall Style: Engineering Usability First, Design Aesthetics Weak

In terms of UI style, the output was better than most low-fidelity prototypes but still fell short of mature component library templates. The design language was simple and engineering-oriented, lacking fine visual polish. Its positioning can be summarized in one sentence:

Engineering usability > product aesthetics.

For internal tools and prototype validation scenarios, this orientation is acceptable; for end-user-facing product pages, manual design intervention is still needed.

4. Lateral Reference: The Core Gap Is in “Depth of Understanding,” Not “Can It Write”

Submitting the same set of requirements to a commonly used high-capability technical solution in the industry yields a set of meaningful comparisons:

Page	Open-Source Model Performance	High-Capability Solution Performance
Login page	Near-complete	High completeness
List page	Filter items missing, linkage not implemented	Filter and field structure understanding complete
Form page	Fields and validation partially missing	Field relationships, validation logic, and interaction details accurately reproduced
Time	Over half an hour (under free quota)	Minute-level

Two conclusions deserve elaboration:

First, the gap is not in the basic ability to generate code, but in the precision of requirements understanding. Both solutions can write syntactically correct, structurally reasonable component code. The real dividing line is: given a natural language document, can the conditions, constraints, and linkage relationships within it be fully extracted into an executable design? The shortcomings in the list page and form page are essentially “reading the words but not the relationships.”

Second, efficiency gaps amplify understanding gaps. The longer generation takes, the higher the probability that developers will interrupt, retry, or manually supplement, and the actual comprehensive cost far exceeds the difference in single generation time. Of course, queuing and rate limiting under free quotas affect the time data, and this should be viewed objectively rather than directly attributed to model inference capability.

5. What This Means: Three Judgments on Open-Source Attributes and AI Development

5.1 The Value Anchor of Open-Source Models Is Shifting from “Can Generate” to “Can Be Trusted”

As basic code generation gradually becomes a general capability, the competitiveness of open-source models will be increasingly reflected in the completeness of requirements reproduction. The premise for developers to entrust tasks to a model is that the output can be trusted, reviewed, and incrementally modified. At the current stage, high-prior scenarios like login pages are approaching trustworthiness, while relationship-intensive scenarios like list pages and form pages still require human fallback. The position of this “capability fault line” is the main battlefield that open-source models must conquer in the next phase.

5.2 Open-Source Attributes Bring Not Just Cost Advantages, but Auditability

For enterprises and developers, one of the core values of open-source models is private deployment, auditability, and customizability. In AI-assisted development workflows, this is particularly important: generated code needs to enter code review, security scanning, and compliance processes, and the explainability and reproducibility of model behavior directly affect whether it can enter the production pipeline. Capability shortcomings can be compensated through engineering means (such as structured requirements templates and post-generation automated validation), while closed-source solutions have less room for maneuver in this regard.

5.3 Implications for Development Practice: Treat “Document Quality” as Part of Model Capability

One of the most valuable findings from the evaluation is: model performance is highly dependent on the degree of structuring in the requirements document. Based on this, several immediately actionable practice recommendations can be offered:

1. Replace prose-style requirements with structured templates. Break pages into a three-part description of “field list + interaction rules + boundary conditions,” and explicitly list linkage relationships. This can significantly improve fidelity.
2. Decompose relationship-intensive pages into subtasks. List pages and complex forms should not be generated as a whole page in one shot; instead, split into “data model → filter logic → view rendering” for step-by-step generation and progressive validation.
3. Always perform automated validation after generation. Use type checking, unit tests, and static scanning as the first gate for generated results, upgrading “can run” to “can merge.”
4. Clearly define human-machine division of labor. High-prior pages (login, empty states, basic layouts) can be boldly delegated to the model; relationship-intensive and design-sensitive pages should retain human leadership, with the model handling drafts and repetitive work.

6. Conclusion

A single real-project evaluation can only illustrate limited points, but it points to a clear direction: domestic open-source large models have crossed the “usable” threshold in basic code generation capability, and the next competitive focus is depth of requirements understanding and completeness of engineering delivery. For developers, rather than agonizing over leaderboard rankings, it is better to build your own small-sample evaluation set—take three to five pages from your real projects, run them through, and see whether the output can enter your codebase. This is closer to the truth than any “explosive” review. The long-term value of the open-source ecosystem is also gradually realized through this reproducible, auditable, and improvable cycle.