Evaluating Open-Source LLMs in Code Generation: Capabilities, Limitations, and Industry Impact

I. Test Background: Why “Real-World Projects” Are the True Litmus Test

With the proliferation of AI coding assistants, benchmark scores can often be dazzling, but performance in real-world development scenarios remains the true “litmus test” for a model’s capability. For this test, a frontend project requirement encompassing three typical pages—login, list, and form—was specifically selected. These scenarios cover over 70% of the CRUD page construction work in daily development. The test did not restrict the tech stack, leaving the model to choose its implementation approach, ultimately completing the project using React. This open-ended prompt is better at reflecting a model’s engineering decision-making capabilities than a fixed template, as it requires the model to independently determine component splitting, state management strategies, and styling solutions.

Notably, the documentation for this test included clear business logic: the login page required support for a carousel display, the list page needed multi-condition combined filtering, and the form page contained complex field linkage validations. These requirements are neither overly simple—which would only test basic code generation—nor so complex as to exceed the current boundaries of AI capabilities. They perfectly distinguish the gap between code that “runs” and code that is “usable.”

II. User Experience: A Threefold Test of Queuing, Speed, and Availability

The first hurdle encountered during actual use was not technical, but resource scheduling: multiple consecutive attempts returned a “service busy, please queue” message, and the generation process was only successfully initiated during off-peak hours. This phenomenon indirectly reflects the temporary imbalance between the旺盛 (robust) user demand for such capabilities and the available supply. In terms of generation time, producing the complete project took approximately 35 minutes, a noticeable gap compared to the minute-level response times of similar services. While compute priority under free tiers might affect speed, the “waiting anxiety” experienced by users is an optimization direction that domestic tools need to address.

On a positive note, the generation result was not an isolated code snippet, but rather provided an independently accessible runtime link, offering great convenience for rapid result verification. This “full-stack” delivery model is evolving from “writing code” to “delivering functionality,” aligning with the core developer demand for AI tools to “reduce context switching.”

III. Feature Breakdown: The Triangular Trade-off of Completeness, Comprehension, and Design

1. Login Page: The Baseline of Fundamental Skills

The login page is the “facade” of the system and the part where the model demonstrated the highest completeness. The generated page not only restored basic elements like forms and buttons but also correctly implemented the carousel logic required in the documentation, with well-proportioned visuals. From an interaction state perspective, details such as the button’s loading state and the input field’s focus styling were all covered. This page reached a level where it could be “directly connected to business APIs and launched,” proving that in low-complexity, highly standardized scenarios, open-source models already possess actual productivity.

2. List Page: The First Decline in Requirement Comprehension

Upon entering the data-intensive list page, issues began to surface. Three core filtering conditions explicitly stated in the documentation (e.g., by status, by time range, by keyword) were either truncated or merged into a single search box in the generated results. More critically, the pagination logic was simplified into a “Load More” button rather than standard page number navigation. This deviation was not an isolated incident but a systemic issue across all three list sub-pages—the model tended to “generate a seemingly reasonable default structure” rather than “precisely replicate the interaction defined in the documentation.” This reflects that the current model’s understanding of requirement constraints remains at the level of “shallow keyword matching” and has not yet reached the height of “semantic-level instruction following.”

3. Form Page: Structural Shortcomings in Complex Logic Reproduction

The form page is the “touchstone” for testing AI comprehension depth. The test designed a complex scenario including cascading selection, dynamic field addition/removal, regex validation, and cross-field logical judgments. Although the generated result executed the basic data submission process, multiple validation rules were silently skipped: for instance, the conditional logic “if the contact type is enterprise, a tax number must be filled in” was not executed; the 11-digit validation rule for mobile numbers was generalized to “non-empty + minimum length limit.” The root cause of these issues is that the model “reasonably” omitted uncommon business rules, lacking the corresponding pattern matching in its training data. This means that code generation for complex form systems currently still requires significant human intervention and supplementation.

4. Visual Style: Sufficient Engineering, Insufficient Design

From an aesthetic perspective, the overall visual was akin to “early admin dashboard templates”: standardized spacing, restrained colors, but lacking design highlights. Compared to the visual effects of directly applying mature component libraries (such as the industry-common Ant Design or Tailwind UI), the generated page fell short in information hierarchy, whitespace rhythm, and the application of rounded corners and shadows. This reveals that in the current model’s training data, the proportion of “function-priority B2B pages” is too high compared to “visually exquisite marketing pages,” causing its default aesthetic orientation to lean towards pragmatic engineering. If positioned as a “prototype accelerator,” it is fully competent; but if “design delegation” is expected, human injection of design tokens is still required.

IV. Horizontal Comparison: Where Lies the “Gap” in Comprehension Depth

Coincidentally, the author had previously used a leading overseas closed-source model to complete the exact same requirements. Placing the two outputs side-by-side yields interesting observations:

• Login Page: The completeness of both was nearly on par, with both implementing all functionalities.
• List Page: The closed-source model performed comprehensively better in the completeness of filter options, the handling of the “no results” state after filtering, and the dynamic show/hide logic of table columns.
• Form Page: The gap expanded exponentially. The closed-source model not only implemented all validations and linkages but also proactively supplemented best-practice details not explicitly stated in the documentation, such as error prompt icon styles and debounce mechanisms to prevent duplicate submissions.

The essence of this gap can be attributed to the “depth of instruction following”—the closed-source model seems better at understanding the “implicit intent” of the documentation, whereas the open-source model is reliable when handling explicit rules but falls short in deducing logic that is “not written by the author but inherently necessary.” Of course, the latter took about one-eighth the time of the former, and this efficiency difference must also be factored into comprehensive considerations.

V. The Open-Source Attribute: Accelerator and Double-Edged Sword from “Usable” to “Useful”

The core variable behind the widespread attention garnered by the open-source model tested is its open-source nature. This holds dual significance for the evolution of the AI industry:

Positive Aspects:

1. Democratization of Technology: Open source returns top-tier model weights, previously monopolized by a few tech giants, to the developer community. This allows small and medium-sized teams to perform domain-specific fine-tuning on them. For example, vertical domain enterprises in healthcare or law can build privatized models based on open-source foundations without training from scratch, significantly lowering the barrier to AI application innovation.
2. Ecosystem Feedback: Open source is not a one-way output. The usage feedback, bug reports, and optimization patches from global researchers effectively constitute continuous free testing and enhancement for the main model. Community-contributed LoRA fine-tuning modules, quantized deployment solutions, and inference acceleration kernels are reverse-engineering the practicality and performance ceiling of the base model at an astonishing pace.
3. Trust Building: For enterprise and government clients, the auditability of code and weights means security and control. Enterprises can deploy them in intranet-private environments, mitigating the risk of sensitive data leakage—a compliance advantage that closed-source API models can never provide.

Practical Challenges:

1. Commercialization Loop Dilemma: Open-source models lack direct commercial returns, and their continuous iteration relies heavily on the parent company’s strategic resolve to “trade open source for ecosystem.” Once a capital winter intensifies, the sustainability of R&D investment becomes questionable.
2. Limited Capability Ceiling: As seen in this test, open-source models still have a gap in complex instruction following. Behind this is a comprehensive difference in training resources, data quality, and tuning experience. If limited open-source community resources are overly concentrated on base-layer replication, it may delay the pace of exploration into inference planning and multimodal fusion.

VI. Implications for AI Development: The Competitive Landscape is Shifting from “Parameter Race” to “Engineering Efficacy Race”

Through this comparison, a clear signal emerges: the gap in model performance is no longer purely about network depth and parameter count, but is reflected in data proportioning, alignment strategies, and tool usage capabilities. Future core competitiveness lies in two areas:

First, frontend code generation is shifting from “static splicing” to “dynamic interaction reproduction.” Whoever can first break through the bottleneck of “generating interactive prototypes directly from design drafts/PRDs” will truly reshape the software development process.

Second, the competition between open-source and closed-source models will coexist long-term. Closed-source models provide extreme performance and out-of-the-box readiness, while open-source models offer flexible customization and data security. For developers, the right strategy is not to blindly chase the “strongest” model, but to establish a hybrid routing mechanism based on task complexity assessment—routing simple tasks to open-source models to reduce costs, and complex logic to closed-source models to ensure quality.

Ultimately, whether open-source or closed-source, the ultimate value metric for AI coding tools has never changed: can it free developers’ time from repetitive coding so they can invest it in more creative architectural design and user experience polishing? From this test, a good start has been made on this path, but the vision of “completely liberating productivity” still requires the unremitting joint efforts of the tech community and the industry.