Evaluating Domestic Open-Source LLMs for Complex Frontend Code Generation: The Real Gap from Requirement Understanding to Engineering Implementation

1. Evaluation Background: The Gap Between Claimed Capabilities and Actual Performance

Recently, the claimed capabilities of domestic large language models (LLMs) in code generation have continuously reshaped our understanding. Following the release of a certain domestic open-source model, tech communities frequently saw evaluations stating that its “capabilities rival top international closed-source models” and that “domestic alternatives are finally mature.” While such remarks are encouraging, as a frontline developer, I firmly believe that a model’s true capability must be validated with real-world projects, rather than relying on benchmark scores or enthusiastic community reviews.

This article documents a complete frontend implementation process based on a real-world project requirement. I provided the same requirement document to both a domestic open-source LLM and a mainstream international closed-source model, conducting a systematic comparison across four dimensions: functional completeness, requirement understanding precision, interaction restoration, and generation speed. It should be noted that the domestic model used in this evaluation was the free version, which may involve queuing and computing quota limits. Therefore, the speed comparison is for reference only, with the focus placed on understanding depth and engineering usability.

---

2. Evaluation Methodology: Based on Real-World Project Requirements

2.1 Requirement Document Design

The evaluation utilized a frontend requirement document containing three typical pages:

Page	Core Requirements	Complexity
Login Page	Carousel, form validation, responsive layout	Medium
Product List Page	Multi-condition filtering, sorting, pagination, card/list toggle	High
Form Page	Multi-field types, cascading validation, dynamic field visibility, submission logic	High

The document was written to be as clear as possible: filtering conditions were explicitly listed, field relationships were explained using tables, and interaction logic was described step-by-step. Such a document is considered “sufficiently detailed” in a real-world development scenario.

2.2 Evaluation Dimensions

• Functional Completeness: Whether all functionalities explicitly required in the document were implemented.
• Requirement Understanding Precision: The degree of restoration regarding field relationships, validation rules, and interaction details.
• Engineering Usability: Whether the generated result can run directly and provides an accessible preview URL.
• Generation Speed: Total time from submitting the requirements to completion.
• UI Design Quality: Whether the visual style meets engineering usability standards.

---

3. Actual Performance of the Domestic Model

3.1 User Experience: Threshold and Speed

Upon first use, I encountered a prompt stating “too many users, please queue or upgrade,” and had to refresh multiple times before accessing the generation interface. This reflects the reality of strained computing resources in the free version—the inference cost of an open-source model does not disappear simply because it is “open”; instead, it may be exacerbated by an influx of users competing for resources.

No specific tech stack was mandated, and the model autonomously chose React. From start to finish, the total time exceeded 30 minutes. The generated result provided a directly accessible project URL rather than just code snippets, which is commendable—being runnable and previewable is the fundamental prerequisite for engineering implementation.

3.2 Login Page: Acceptable Completeness

The implementation of the login page basically covered the document’s requirements:

• Carousel component rendered normally
• Form fields were complete
• Basic validation logic was functional
• No obvious misalignment in responsive layout

Overall evaluation: It is at the level of “ready to go live with minor tweaks.” The requirements for this page were relatively independent and logically simple, resulting in stable model performance.

3.3 List Page: Noticeable Decline in Understanding

Entering the product list page, problems began to surface. The document explicitly specified filtering conditions and interaction logic, but in the generated results:

• Filtering conditions were unimplemented: This occurred across multiple list pages and was not an isolated incident.
• Page structure was correct, but functional understanding was compromised: The layout skeleton was present, but interaction logic was missing.
• The cascading logic for sorting and pagination was incomplete.

This indicates that the model lacks sufficient depth in parsing the requirement document when handling multi-condition combinations and state cascading. It can identify the structural information that “the page needs a filtering area,” but it cannot accurately extract semantic-level requirements such as “what the filtering conditions are, how they combine, and what behaviors they trigger.”

3.4 Form Page: Insufficient Restoration of Complex Logic

The form page is where the gap was most obvious in this evaluation:

• Field types were mostly correct
• However, field cascading logic was not implemented (e.g., a field’s value determining the visibility of another field)
• Restoration of validation rules was incomplete (custom validations explicitly written in the document were not generated)
• Submission logic and error message handling were crude

Even though the document clearly described field relationships and validation rules in a tabular format, the model still failed to restore them completely. This demonstrates that current domestic open-source models have obvious shortcomings in complex state management and conditional logic.

3.5 Overall UI Style: Engineering-Ready, but Lacks Design Polish

• Better than interfaces generated by most prototyping tools
• But still lags behind the default effects of mature component libraries (like Ant Design or Tailwind templates)
• The design leans towards simple engineering, belonging to the “function-first” category
• Its positioning can be summarized as: Engineering Usability > Aesthetic Product Design

---

4. Comparison with Mainstream International Models

The same set of requirements was previously implemented using a mainstream international closed-source model. The comparison results are as follows:

Dimension	Domestic Open-Source Model	Mainstream International Closed-Source Model
Login Page Completeness	Close	Close
List Page Filtering & Interaction	Obviously insufficient	Complete understanding, accurate implementation
Form Page Field Relationships & Validation	Low restoration	Accurately restored field relationships, validation logic, and interaction details
Generation Speed	Over 30 minutes (Free version)	~3 minutes
Requirement Understanding Precision	Primarily structural understanding	Semantic-level understanding, higher precision

The core gap is not about “whether a page can be generated,” but “how deep the understanding goes.”

Mainstream international models can extract dependency relationships between fields, conditional branches for validation, and trigger timing for interactions from the document, translating this semantic information into runnable code. Domestic open-source models are currently better at recognizing “what components a page needs,” but they still fall significantly short at the level of “how components collaborate.”

---

5. Implications for Open Source and AI Development

5.1 Open Source Does Not Mean Free Computing Power

The value of an open-source model lies in being auditable, customizable, and privately deployable, but inference costs always exist. The queuing and rate limiting in the free version are direct manifestations of limited computing resources. For enterprise users, the actual cost of an open-source model should include the investment in inference infrastructure, rather than simply comparing API call prices.

5.2 The Core of the Code Generation Gap is the Gap in “Requirement Understanding”

The competitive focus of code generation models has shifted from “whether syntactically correct code can be generated” to “whether complex requirements can be accurately understood.” This involves:

• Long document understanding: Extracting structured requirements from multiple paragraphs and tables.
• State and condition reasoning: Understanding field cascading, validation branching, and interaction timing.
• Engineering constraint perception: Generating code that conforms to the project’s tech stack and specifications.

Domestic open-source models have basically met the standard for the first step of “structural recognition,” but they still require continuous optimization in the second and third steps.

5.3 Practical Advice for Developers

1. Do not blindly trust benchmark scores: The complexity of real-world projects far exceeds that of evaluation sets; hands-on validation is necessary.
2. Make requirement documents “machine-readable”: Use tables, lists, and clear conditional descriptions to minimize ambiguity.
3. Divide and conquer: Split complex pages into independent components and generate them separately to reduce the semantic load of a single inference.
4. Human fallback is indispensable: Generated results must undergo code review and interaction testing, especially for conditional logic.
5. Pay attention to inference costs: The free quota for open-source models is limited; enterprise deployment needs to assess the Total Cost of Ownership (TCO) for private deployment.

5.4 Implications for the AI Tech Ecosystem

The gap between domestic open-source models and top international closed-source models is narrowing, but the nature of the gap is changing: from “can it be used” to “is it easy to use,” and from “generating code” to “understanding requirements.” This shift places higher demands on model training:

• More training data for multi-turn interactions and requirement clarification is needed.
• Reasoning capabilities for complex conditional logic need to be strengthened.
• Constraints for engineering specifications and best practices need to be introduced.

The advantage of the open-source community lies in transparency and collaboration. If a high-quality evaluation set oriented towards real-world project requirements can be established, and the model’s capability in requirement understanding is continuously iterated, domestic open-source models have the potential to match or even surpass the practical utility of closed-source models in specific scenarios.

---

6. Conclusion

This evaluation demonstrates that domestic open-source LLMs possess engineering usability for simple page generation, but there remains a significant gap compared to mainstream international closed-source models in scenarios requiring deep requirement understanding, such as complex list filtering, form cascading, and validation logic. This gap is not insurmountable, but it must be acknowledged—the core competitiveness of code generation lies not in generation speed or code volume, but in the precise restoration of requirement semantics.

For developers, the most pragmatic strategy at the current stage is: use domestic open-source models for prototyping and simple page generation, while complex interaction logic should still be implemented manually or assisted by models with stronger understanding capabilities. Simultaneously, actively participate in the open-source community’s requirement feedback and evaluation set construction to drive the continuous evolution of models in real-world scenarios.

The significance of open source lies not only in visible code but also in iterable capabilities and co-buildable ecosystems. The process of domestic models catching up to top international levels is fundamentally a marathon about “depth of understanding,” not a sprint.