Evaluating Domestic LLMs in Real-World Frontend Development: Capability Gaps and the Value of Open-Source Ecosystems
This article presents a hands-on evaluation of a Chinese large language model on a real frontend project covering login, product listing, and form pages, comparing its output with leading international models. It analyzes where the gap lies—not in code generation but in requirement comprehension—and discusses what the model's open-source nature means for engineering verifiability, toolchain maturity, and vendor lock-in risk. Developers will find practical suggestions for writing executable specs, validating output in stages, and positioning LLMs as accelerators rather than replacements.
Evaluating Domestic LLMs in Real-World Frontend Development: Capability Gaps and the Value of Open-Source Ecosystems
1. Background: When “Benchmarking Against” Becomes a Stock Phrase
Over the past two years, LLM launch events have settled into a familiar formula: a new model’s capabilities are described as “benchmarking against” international state of the art. Comparisons proliferate—benchmark scores, real task performance, multimodal understanding, code generation. For developers, impressive evaluation numbers are eye-catching, but what actually determines whether a model can enter a production workflow is its engineering usability in real projects: Can it reliably understand requirements? Can it fully implement features? Can it deliver usable results within an acceptable timeframe?
This article does not discuss parameter counts or stack up benchmark scores. Instead, it starts from a complete frontend project requirement, documents a hands-on test of LLM-assisted development, and extends the discussion to a broader question: as domestic models gradually approach the international frontier, what do their open-source attributes—and the overall AI landscape—mean for developers?
2. Test Design: Validating Capability with a Real Project
2.1 Task Setup
The test task was drawn from a real frontend requirement comprising three typical pages: a login page, a product listing page, and a form page. The requirements document specified interaction logic, filter conditions, field validation rules, and visual style requirements, but did not mandate a tech stack. The model ultimately chose to implement the pages in React.
These three pages were chosen because they cover three core frontend capabilities:
- Login page: tests the completeness of basic layouts and common components such as carousels;
- Listing page: tests understanding and reproduction of filter conditions and data interaction logic;
- Form page: tests precise implementation of field relationships, validation rules, and interaction details.
2.2 Process Notes
In actual use, the first obstacle was a resource queue notification; it took multiple attempts before the generation process began. Generation itself was slow—roughly half an hour or more from start to finish. This needs to be viewed objectively: queuing and rate limiting under free or shared resource pools are common phenomena, and the elapsed time does not fully represent the model’s own inference efficiency.
On the positive side, the output was not scattered code snippets but a complete, directly accessible project URL. This is very friendly for verification and secondary development—developers can see the running result without first spending time scaffolding a project.
3. Page-by-Page Analysis
3.1 Login Page: High Completeness
The login page performed well overall. The carousel requirement mentioned in the document was correctly implemented, the page was fully functional, and nothing was obviously missing. The overall level can be summarized as “usable after minor adjustments.” This indicates that the model has stable generation capability for highly structured, pattern-driven pages.
3.2 Listing Page: Comprehension Degrades
Problems began to surface on the product listing page. The filter conditions and interaction logic explicitly stated in the document were not implemented in the actual output. More critically, this problem recurred across multiple listing pages, making it a systematic shortcoming rather than an occasional miss.
The page skeleton was correct—layout and component structure were basically reasonable—but the “functional comprehension” layer was clearly compromised. In other words, the model could build a page that “looks right” but failed to translate “the logic written in the document” into code.
3.3 Form Page: Insufficient Fidelity
The form page ran, but many fields, validation logic, and interaction details were not fully implemented as described in the document. Even with a relatively clear requirements document, the model’s comprehension and reproduction capability for complex form requirements remained noticeably lacking. Forms are precisely the page type with the highest detail density: field interdependencies, validation trigger timing, and error message presentation—any missing piece affects usability.
3.4 Overall Style: Engineering Usability First
On the visual level, the overall UI style was conventional: better than most low-fidelity prototypes, but still short of directly applying a mature design system template. The design feel was simple and engineering-oriented, positioned as “engineering usability over product aesthetics.” For internal tools or rapid validation scenarios, this level is acceptable; for end-user-facing products, design intervention is still needed.
4. Where the Gap with International State of the Art Lies
After giving the same requirements to a leading international model for comparison, the gap was concentrated in depth of comprehension rather than “whether it can do it at all”:
| Dimension | Domestic Model | International State-of-the-Art Model |
|---|---|---|
| Login page | Close in completeness | Close in completeness |
| Listing page | Correct structure, missing filter logic | More complete understanding of filters and field structure |
| Form page | Runnable, insufficient detail fidelity | More accurate reproduction of field relationships, validation logic, and interaction details |
| Generation speed | Slower (affected by resource pool) | Noticeably faster |
The conclusion can be summed up in one sentence: the gap is not in “generating code” but in “understanding requirements.” When a requirements document contains multiple layers of conditions, implicit dependencies, and boundary rules, the model must map natural language descriptions to precise program logic—and the precision of that mapping determines whether the output can be used directly.
5. What This Means for Open-Source Attributes
5.1 The Value of Open Source Is Not “Free” but “Verifiable”
When a model’s capabilities approach the international frontier and it also carries open-source attributes, the most direct value is verifiability. Developers can reproduce evaluations in their own environments, fine-tune on their own business data, and audit whether model behavior meets compliance requirements. No matter how strong a closed-source model is, enterprises still cannot answer the question “why did it produce this output?”
5.2 Open Source Drives Maturity of Engineering Tooling
Once model weights are open, an ecosystem grows naturally around inference optimization, quantized deployment, and fine-tuning toolchains. For developers, this means running models in private environments at controllable cost and embedding LLM capabilities into existing CI/CD, code review, and test generation workflows rather than depending on external APIs.
5.3 Open Source Reduces “Capability Lock-In” Risk
When core R&D workflows depend on an external model API, interface changes, price adjustments, and capability deprecations all translate into business risk. Open-source models provide a fallback: even if the primary solution changes, teams can switch to a self-hosted instance and maintain workflow continuity.
6. Broader Implications for AI Development
First, the focus of competition is shifting from “benchmark scores” to “engineering deployment.” When multiple models trade wins on benchmarks, the real dividing line is comprehension precision on complex requirements, stability on long tasks, and fit with actual R&D workflows. The hands-on test in this article is a microcosm of that trend.
Second, open source and closed source complement rather than replace each other. Closed-source models often lead in general capabilities, while open-source models offer irreplaceable advantages in customizability, auditability, and private deployment. Only in parallel can they cover the full spectrum from rapid validation to deep customization.
Third, the developer’s role is shifting upward. When models can handle repetitive work such as generating page skeletons, developer value is increasingly reflected in requirement decomposition, logic design, quality control, and system integration. Knowing how to use a model is not the endpoint; expressing requirements as specs a model can execute accurately is the core competency.
7. Practical Advice for Developers
- Write requirements documents as “executable specs.” List fields, validations, and interactions item by item to reduce ambiguity—model comprehension precision improves markedly.
- Validate in stages; do not deliver an entire site in one shot. Have the model generate a single page skeleton first, verify it, then layer on logic—this lowers the cost of locating problems.
- Review generated output structurally. Focus on three high-frequency gaps—filter conditions, field interdependencies, and validation rules—rather than only checking whether the page opens.
- Position the model as an “accelerator,” not a “replacement.” Skeleton generation, boilerplate code, and test case generation are currently the highest-value entry points.
- Pay attention to self-hosting and fine-tuning paths for open-source models. For teams with data compliance requirements or deep customization needs, this is key to long-term cost and risk control.
8. Conclusion
That domestic models are approaching the international frontier is a positive development worth acknowledging, but “benchmarking against” is only the starting point. What truly determines their value is whether they can deliver reliably in real projects, whether their open-source attributes can form a verifiable and customizable engineering system, and whether they can push the entire industry from competing on scores to competing on deployment. For developers, maintaining a habit of hands-on testing and building your own evaluation yardstick is more meaningful than chasing any single launch event.
