Testing an Open-Source Chinese LLM on Frontend Project Generation: Results and a Replication Guide

    This article describes how to test a Chinese open-source large language model on generating a frontend project from a real business requirements document. It compares the output with a leading closed-source model and covers the test setup, judging criteria, results for three page types, trade-offs between the two models, and steps for reproducing the test.

    1. Why Reproducible Evaluation Matters

    AI-assisted programming has moved beyond code snippet completion to generating complete projects. Many evaluations only conclude whether a model “works” or “doesn’t work.” Readers cannot learn the test conditions or repeat the test themselves. This article breaks the testing process into parts that readers can check one by one: what the requirements document says, which environment the model ran in, and what standards determined the results. The credibility of any conclusion depends on whether this information is public.

    This article tests a Chinese open-source large language model against a leading closed-source model under the same conditions. Below, the open-source model under test is called “the open-source model,” and the comparison model is called “the closed-source model.”

    2. Test Method and Environment

    2.1 Scope of the Requirements Document

    The test uses one real frontend business requirements document covering three types of pages:

    • Login page: account and password form, input validation, and a carousel.
    • Product list page: multiple filter conditions. Each condition follows a defined interaction rule, and the filtered results update as conditions are combined.
    • Form page: asynchronous validation for required fields, and linkage between fields.

    The requirements document does not specify a frontend framework, so the model can choose its own stack. The same document can therefore produce different outputs under different frameworks. Fix the framework when reproducing the test.

    2.2 Runtime Environment and Records

    Before reproducing the test, record the following items and store them with the results:

    • Name, version, access method, and parameter settings of the model under test;
    • The full original prompt, including the version of the requirements document;
    • The Node.js version, package manager version, and dependency versions used for the generated project;
    • The test date, network conditions, and the start and end time of each generation.

    Only with complete records can a third party re-verify the results.

    2.3 Evaluation Criteria

    Judge each page on the four dimensions below. Each verdict must be “Pass,” “Partial pass,” or “Fail,” and must include the relevant code location or screenshot.

    Dimension Criteria
    Functional completeness Whether all feature points in the requirements document are implemented
    Interaction logic Whether states transition correctly, for example, whether results update after filter conditions are combined
    Validation fidelity Whether validation rules, asynchronous validation, and field linkage match the document
    Runnability Whether the project starts without modification and completes the main operation flow

    3. Results: Generation by Page Type

    The model produced a complete project that could be accessed directly, not scattered code. The verdicts for each page are below.

    Page Requirements Generated result Verdict
    Login Form, input validation, carousel Complete page structure and a working carousel. No obvious logic gaps found. Basic functions pass
    Product list Multiple filter conditions and their interactions Filters were not implemented effectively. The same problem appeared repeatedly across similar list pages. Complex interaction fails
    Form Asynchronous validation, field linkage Asynchronous validation for required fields was reduced to frontend regex validation. Field dependencies were ignored. Business constraints fail

    The login page is a conventional page with a clear structure and simple logic. The model handled it well, so such pages make a good starting point for generation.

    The problems on the product list page deserve more attention. The model generated the page skeleton but did not convert the filter rules into correct state transition code. The cause may relate to context length or to multiple overlapping constraints. However, this test did not control the number of filter conditions or the context length, so the cause remains an unverified hypothesis. To verify it, add filter conditions step by step and observe where failures begin.

    The form page shows a problem similar to the list page. The model tended to use common patterns from its training data, such as replacing asynchronous validation with regular expressions, instead of implementing the business constraints as the document specified.

    Visually, the generated pages had clear layouts, but their visual hierarchy and detail were weaker than pages built with a mature component library. This test did not include a quantitative visual evaluation, so this observation reflects only the overall impression of layout and styling.

    4. Comparison with the Closed-Source Model Under the Same Conditions

    Both models generated their output from the same requirements document.

    On the login page, the two reached similar levels of completion, and both met the usable standard. On the product list and form pages, the closed-source model reproduced the filter combinations and field validation dependencies more completely.

    For timing, the closed-source model took about 3 minutes from input of the requirements to an accessible project. The open-source model took more than 30 minutes, which may have been affected by compute allocation for free-tier calls. These figures come from a single test run. The test did not average multiple runs or control network conditions, so the numbers serve only as a directional reference.

    The two options involve clear trade-offs. A closed-source model usually reproduces complex logic more completely and returns results faster, but it carries costs and requires sending code and requirements to an external service. An open-source model lets deployment and data stay in your own environment, but complex business logic requires more manual verification.

    5. The Value and Limits of Open Source

    Open-source licensing lowers the barrier to entry. Small and mid-sized teams and independent developers can obtain the model directly and fine-tune it on their own business data. This reduces repeated trial and error and lets more people contribute to improvements.

    The feedback loop in the open-source ecosystem also has value. Developers can report error cases found in real projects to the model team through compliant channels, which can help improve later versions. This depends on whether projects are willing to share data and whether the license permits that use. It does not happen automatically.

    Stay realistic about the current state, too. In this test, the problems on the list page and form page appeared in the current version, but nothing guarantees that the next version will fix them. When evaluating whether an open-source model fits a project, base the decision on test results for the current version, not on expectations for future versions.

    6. Adoption Checklist

    Before using a generated project, developers can check it with the following steps:

    1. Break the requirements document into numbered feature points, and tie each point to a verifiable criterion.
    2. Write test cases for each filter rule on the list page and each validation rule on the form page.
    3. Review login authentication, input filtering, and sensitive information handling by hand. Do not adopt generated code for these parts directly.
    4. Pin dependency versions, and archive the generation records with the code.
    5. Run regression tests on each page that passes, to confirm that later changes do not break existing functions.

    Four risks also need attention:

    • Data compliance: Before procurement or integration, confirm whether requirements documents and business code will be uploaded to an external service.
    • License terms: Read the terms on commercial use, modification, and redistribution clause by clause.
    • Cost: Free quotas, queue waits, and compute allocation all affect delivery timelines. Budget based on measured generation times.
    • Long-term maintenance: Whether the generated code has a clear structure and adequate comments determines whether a later team can take it over.

    7. Replication Steps

    1. Fix the version of the requirements document, and confirm that the feature points for the three page types match this article.
    2. Record the model version, call parameters, and original prompt.
    3. Generate from the same requirements three times, and record the completion time and run result of each attempt.
    4. Score each page using the criteria in Section 2.3, and keep the code locations or screenshots.
    5. Run the generated project, and execute the test cases from the checklist in Section 6.

    8. Conclusion

    This test shows that the open-source model can complete basic generation for structured pages such as login. For multi-condition filtering on the list page and asynchronous validation on the form page, its performance still trails the closed-source model. For developers, the safer approach is to treat the model as a basic code generator and have people verify complex business logic.