LLM Code Generation Evaluation: Open-Source vs. Closed-Source Models on a Real Frontend Project

    This article evaluates large language model code generation using a real frontend project rather than standardized benchmark tasks. A structured requirements document covering login, product list, and form pages was used to compare an open-source model with a top closed-source baseline, focusing on requirement fidelity, functional completeness, UI quality, generation speed, and engineering usability. The results show that simple page generation is roughly comparable, while complex requirement understanding remains a clear gap. The article also discusses why the open-source route matters and offers practical guidance for developers.

    1. Background: The “Last Mile” Problem in LLM Code Generation

    Over the past year, large language models have advanced rapidly in code generation. From completing a few lines of code to generating complete projects end to end, the ceiling of model capability keeps rising. Yet a practical problem remains: strong benchmark scores do not equal real-world engineering usability.

    Scores on evaluation leaderboards often come from standardized tasks with clear boundaries. Requirements documents in real projects, however, are full of implicit conventions, ambiguous wording, and context dependencies. Whether a model can accurately translate a business requirements document into runnable engineering code is the key test of its depth of understanding.

    Recently, a new version of an open-source model developed in China was released, claiming to match top international closed-source flagship models on multiple benchmarks. This raises a question worth examining: what does it actually mean for developers and the industry ecosystem when open-source models catch up with closed-source models? This article does not stop at benchmark scores. It starts from a real frontend project, reconstructs the model’s real-world performance, and then returns to the value of the open-source route.

    2. Evaluation Method: Real Projects Instead of Idealized Tasks

    To make the evaluation results close to real usage scenarios, I designed a test plan that reflects daily development work:

    • Input: a structured requirements document containing three typical page types: a login page, a product list page, and a form page, with explicit filter conditions, field validation logic, and interaction details.
    • Requirement: the model must output a complete, accessible project in one pass, rather than scattered code snippets.
    • Control group: the same requirements had previously been completed by a top international closed-source model and served as a baseline reference.
    • Evaluation dimensions: requirement fidelity, functional completeness, UI quality, generation speed, and engineering usability.

    The core idea behind this method is: do not test a toy task like “write a counter,” but test a real capability like “turn a document into a system.”

    3. Hands-On Results: Breaking Down Model Performance

    3.1 Login Page: Close to Launch-Ready

    The login page was completed satisfactorily. The carousel required in the requirements document was correctly implemented, the page functions were complete, the layout was reasonable, and there were no obvious omissions. The level of this page can be summarized as “ready to launch after minor adjustments,” which falls within the engineering-usable range.

    This shows that the model already has stable restoration capability for pages with clear structure and fixed patterns. Design patterns for pages like login pages appear extremely frequently in training corpora; the model is effectively performing a highly patterned generation task.

    3.2 List Page: Requirement Understanding Starts to Fail

    On the list page, problems emerged. The requirements document explicitly specified filter conditions and interaction logic, but the filter items were entirely missing from the generated result, and the issue was systematic rather than occasional.

    The page structure itself was correct, but the “document to function” translation step fell short. This exposes a key issue: the model’s information extraction and instruction-following ability on requirements documents is weaker than its code template generation ability. In other words, the model “can write pages” but “did not understand the document.”

    3.3 Form Page: Insufficient Restoration of Complex Logic

    The form page showed the clearest gap. The page runs, but many field definitions, validation rules, and interaction details were not implemented according to the document.

    Complex forms involve multi-layer dependencies: field linkage, conditional validation, error message timing, and more. Such requirements demand multi-step reasoning and fine-grained understanding of long documents, rather than simple template application. This is exactly the dimension where the gap with top closed-source models is largest.

    3.4 UI Quality and Generation Efficiency

    • Style: overall conventional, better than most prototype output, but still with a quality gap compared with templates built on mature frontend component systems. It is positioned as “engineering usability before product polish.”
    • Speed: generating the complete project took noticeably longer, while the closed-source baseline model completed it within minutes. Part of the speed difference is related to access method and service quotas, but the overall efficiency gap is real.

    3.5 Comparison with the Closed-Source Baseline

    Dimension Tested open-source model Closed-source baseline model
    Simple page restoration Roughly on par Roughly on par
    Complex requirement understanding (filters/validation) Clearly missing Fully restored
    Generation speed Slower Fast
    UI quality Engineered, simpler More refined details

    The conclusion can be summarized in one sentence: open-source models are close in “generation,” but still show a generational gap in “understanding.”

    4. The Significance of the Open-Source Route: Why This Gap Is Worth Closing

    Returning to the question in the title: what is the significance of open-source models benchmarking against closed-source flagships?

    First, break the capability monopoly and reshape cost structures. Closed-source flagship models are usually available only through paid APIs, and their output behavior is controlled by vendor policies. Open-weight models can be deployed privately, allowing enterprises to keep inference costs at the infrastructure layer and keep data within their own domain. This is decisive for highly regulated industries such as finance and healthcare.

    Second, promote capability accessibility and ecosystem prosperity. Open source means the community can fine-tune, distill, quantize, and turn models into agents on top of them. A model is no longer a black-box service, but a composable foundational component. The evolution from operating systems to databases has shown that open ecosystems ultimately generate more active innovation.

    Third, force the entire industry to iterate faster. When open-source models are only one step behind closed-source models, closed-source vendors must widen the next gap faster. End users benefit directly from this competition, and the overall model capability curve moves upward.

    Fourth, strategic value for domestically developed AI. Code generation capability is directly linked to software production efficiency. If domestically developed open-source models can break through the last mile of “understanding depth,” they will substantially lower the barrier to intelligent development for domestic enterprises and form an autonomous and controllable technology foundation.

    5. Practical Recommendations for Developers

    Based on this evaluation, here are several actionable recommendations:

    1. Use models by task layer: simple pages and boilerplate code are highly cost-effective with current open-source models; for complex business logic and multi-constraint requirements, it is still advisable to use models with stronger understanding or to manually decompose requirements before generation.
    2. Structure requirements documents: instead of writing long natural-language paragraphs, write filter conditions and validation rules as structured lists (such as YAML), which can significantly improve model fidelity. Example:
    1. # Structured requirement example: improve model adherence
    2. list_page:
    3. filters:
    4. - field: status
    5. type: select
    6. options: [all, active, archived]
    7. - field: keyword
    8. type: search
    9. behavior: debounce_300ms
    10. table:
    11. columns: [name, status, updated_at]
    12. interactions:
    13. - filter_change -> reload with query params
    1. Human-model collaboration loop: treat model output as a “draft” rather than a “finished product,” and use code review processes to catch risks of missing logic.
    2. Focus on the nature of the gap, not benchmark scores: when selecting a model, focus on testing its instruction-following rate on your own business documents. This is more informative than public leaderboard scores.

    6. Conclusion

    This hands-on evaluation reveals a clear fact: current open-source large models are already no less capable than top closed-source models in the “patterned generation” part of code generation, but there is still a perceptible gap in “deep requirement understanding.” This is both a shortcoming and the main direction for the open-source route going forward.

    The significance of open-source models continuing to approach closed-source flagships goes far beyond a benchmark comparison: it means a lower barrier to intelligent development, more active ecosystem innovation, and more autonomous technology choices. For developers, the right attitude toward technology selection is to treat the “catching up” narrative rationally and validate capability boundaries with real projects. It is foreseeable that as the last mile of “understanding depth” is gradually conquered, the gap between open source and closed source will continue to narrow, and the ultimate beneficiary will be the entire software industry.