Evaluating LLMs for Frontend Project Generation: Capability Limits and Engineering Practices
A practical, reproducible evaluation of how well large language models generate complete, runnable frontend projects. It explains why whole-site generation is a fundamentally harder problem than function-level completion, presents an evaluation method that turns subjective impressions into repeatable conclusions across display, list, and form pages, and analyzes why capability decays as requirement density rises. It then lays out an engineering pipeline—machine-readable specs, phased generation, an automated validation matrix, and design tokens—to compensate for the model's comprehension gaps, discusses what open-source models offer enterprises, and closes with four priorities for production adoption.
1. Where it starts: why whole-site generation is a new dividing line
The benchmark for code generation has shifted twice in the past two years: from single-line completion to function-level generation, and then from function-level generation to multi-file, runnable project delivery. The second shift did not bring a linear improvement in capability—it caused a step change in problem difficulty, for three reasons.
The state space explodes. A login page fundamentally only handles a few rendering branches. A list page with filtering and pagination, by contrast, has to stay behaviorally consistent across a multidimensional combination of “query conditions × page number × sorting × empty state × loading state × error state.” The complexity of generated code does not grow with the number of files—it grows with the number of state combinations.
Implicit constraints multiply. A single line in a requirements document—“support multi-condition filtering”—translates in the implementation into six or seven places that must all be correct at once: URL query parameter sync, condition restoration, reset logic, coordination with pagination, and final request parameter assembly. Human developers fill in these implicit chains from experience; the model has to think of all of them within a single forward pass.
There is no immediate feedback loop. The code is not auto-compiled and no tests run automatically, so the model has to “get it right in one shot,” with no chance to self-correct. This is fundamentally different from the high-frequency “write—error—fix” loop a human developer has in an IDE.
So evaluating a model by “can it generate a page that runs” no longer works. A more explanatory question is: as requirement density increases, how fast does the model’s implementation fidelity decay, and at which stages does the decay concentrate?
2. Evaluation method: turning subjective experience into reproducible conclusions
Many online claims that “model X is strong” come from a single simple Q&A. To make a conclusion hold up, you need to control at least the following variables.
- Input side: don’t give a one-sentence prompt—give a structured requirements document (page inventory, field definitions, interaction rules, acceptance criteria) and observe how well the model restores the long-text constraints.
- Output side: require a complete, runnable project with an accessible preview URL, not scattered code snippets—the latter cannot reveal whether “multi-file coordination holds up.”
- Dimension side: break it into at least five observable items rather than a vague “is it good or not.”
- Functional completeness: are all the capabilities listed in the document implemented;
- Document consistency: do the implementation details match the described fields, rules, and edge cases;
- Code structure quality: are directory organization, component decomposition, and type definitions reasonable;
- Delivery latency: time from submission to runnable;
- Maintainability: how many files a single requirement change touches.
- Reproducibility: run the same requirement two or three times across multiple sessions to distinguish “occasional failure” from “systematic weakness.”
One limitation must be stated plainly: this is a subjective test on a single sample, and the conclusions are only a directional reference, not a quantitative benchmark. True quantification requires a fixed evaluation set, fixed scoring scripts, and cross-model comparisons—which the later sections address.
3. Layered observation: three steps of capability decay
Running a full set of frontend requirements through the above method (covering typical pages such as login, list, and form) produced results with clear tiered differences.
| Page type | Completeness | Main problem | Verdict |
|---|---|---|---|
| Display (login page, with carousel) | High | No obvious gaps | Ship-ready after edits |
| List (filtering, pagination) | Medium | Filters explicitly stated in the doc were not implemented, and this recurred across multiple list pages | Structural weakness |
| Form (field linkage, validation) | Low | Fields, validation rules, and interaction details were not fully implemented per the doc | Significant gap |
Three key judgments:
First, display pages are no longer the hard part. Visual restoration, basic component assembly, and common animations—these task patterns are extremely dense in training data, and the model performs stably here. The discussion value of this layer is declining.
Second, the list-page problem is “systematic omission,” not “random error.” This matters: if a filter were dropped on just one page, you could attribute it to randomness; but multiple list pages consistently skip the filtering logic, which shows the model has a stable comprehension gap around the “rule-based requirements in the document,” not just bad luck.
Third, UI appearance is a bonus above the passing line. The generated interfaces are more polished than a rough prototype, but still fall short of a finished product built on a mature component library or an atomic styling approach—overall engineering-oriented and lacking design sense. The conclusion is clear: engineering-usable > product-beautiful. This is basically enough for internal enterprise systems; for consumer-facing (C-facing) products it needs an extra layer of design investment.
Compared against a reference point—the same set of requirements had previously been completed by a closed-source flagship model—the differences concentrate on “depth of understanding” rather than “generation ability”: the two were close on the login page, while the reference implementation restored field relationships and validation logic noticeably more completely on the list and form pages, and did so faster (though free tiers, queuing mechanisms, model tiers, and other factors may all affect this metric, so it should not be attributed directly to model capability).
4. Cause analysis: why “understanding slows down” midway
Breaking down the phenomena above, you can locate four technical causes.
1. Effective context gets diluted. After a long requirements document is input, the model does not weight all paragraphs equally. Constraints that appear early and are strongly task-relevant are more easily retained, while detailed rules in the middle and later sections tend to get “averaged out.” Context window length is not equal to effective utilization—this is one of the core tensions in current long-text understanding.
2. There is no verifiable closed loop. Without compilation, type checking, and test feedback, the model cannot know what it missed. When writing a list page, a human developer first defines the query parameter types, then aligns item by item; the model skips this step and directly produces an interface that “looks reasonable.”
3. Cross-module consistency is hard to maintain. Filter conditions, pagination, request parameters, and state management live in different files. Once a field changes, you have to synchronize edits across four places: type definitions, form components, the request layer, and restoration logic. Propagating this kind of global constraint is an order of magnitude harder than generating a single component.
4. The training signal is biased toward the visual and structural. A large amount of high-quality corpus comes from “good-looking component examples,” while corpus on “how business rules are accurately implemented” is relatively sparse. This explains why the model is good at producing a structurally correct page skeleton but discounts on rule restoration—its prior distribution is itself skewed toward the former.
5. Engineering completion: turning one-shot generation into a pipeline
Since the bottleneck is the “lack of constraint propagation and a validation loop,” the solution should not be to swap in a stronger model—it should be to change how the model is used.
5.1 Making requirements machine-readable
Natural-language descriptions are inherently ambiguous. Extracting the key parts into a structured config gives the model a definite alignment target:
{"page": "product_list","filters": [{"field": "category", "type": "enum", "multi": true, "syncToUrl": true},{"field": "priceRange", "type": "range", "syncToUrl": true},{"field": "keyword", "type": "string", "trim": true}],"pagination": {"mode": "server", "defaultPageSize": 20, "resetOnFilterChange": true},"states": ["loading", "empty", "error", "no_result"],"acceptance": ["URL parameters update synchronously after a filter is selected","Page number resets to 1 when filters change","Filter conditions are restored after refreshing the page"]}
The acceptance field is the key to the whole method: it makes implicit constraints explicit and can naturally serve as input for later automated validation.
5.2 Phased generation instead of one-shot output
Generating the whole project in one shot forces the model to juggle data models, routing, state, styling, and interaction at once, thinning out its attention. A reasonable order is:
- First fix the types and interface contracts (data models, request/response structures);
- Then generate the page skeleton and routing;
- Then add interaction and state management;
- Finally handle styling and design tokens.
Each step’s output serves as a strong constraint input for the next, equivalent to building a constraint-propagation chain by hand.
5.3 Building a post-generation automated validation matrix
This is the highest-value step. Turning the acceptance checklist into executable checks turns the model from “writing blind” into “iterating with feedback”:
- Type checking: catches field misspellings and structural inconsistencies;
- Static analysis: catches unused variables and dependency-order issues;
- Unit tests: cover pure logic such as filter parameter assembly and pagination resets;
- End-to-end tests: walk through the acceptance checklist item by item, directly exposing omissions like “the filter was not implemented”;
- Accessibility checks: semantic tags, contrast, keyboard reachability.
The accompanying iteration loop can take a form like this:
spec = load_structured_requirements()artifact = generate(stage="scaffold", spec=spec)for round in range(MAX_ROUNDS):report = run_checks(artifact, spec) # type / static / unit / end-to-endif report.passed:breakartifact = repair(artifact, report) # feed failing items back with contextassert report.passed, "there are unclosed acceptance items"
The key design principle: feed only the failing acceptance items back to the model, rather than throwing the entire project back. This both avoids context dilution and gives the repair a clear convergence direction.
5.4 Constraining the visual layer with design tokens
Rather than letting the model freely improvise styles, define a set of design tokens first (semantic variables for colors, spacing, font sizes, corner radius, and shadows) and require the generated code to reference only these variables. This solves the inconsistency in appearance and turns later re-skinning or redesigns into a configuration change rather than a full rewrite.
6. What it means for the open-source model ecosystem: capability gaps are dynamic, controllability is structural
Back to the original question: when an open-source model approaches closed-source flagship level on some tasks, where does the real significance lie?
Capability gaps are dynamic and converge over time. On public leaderboards, the gap between open and closed source often changes on a scale of “months.” Basing model selection on a momentary ranking means betting the architecture on a constantly drifting variable.
Controllability is structural and will not disappear over time. This is the more tangible value for enterprises:
- Clear data boundaries: code and requirements documents are core assets; private deployment means data stays within your perimeter and the compliance assessment path is shorter;
- Sustainable fine-tuning: you can continue training or do parameter-efficient fine-tuning on your own codebase, internal component conventions, and domain terminology, turning “general capability” into “capability that matches your team’s conventions”;
- Predictable cost: you own the inference resources, so marginal cost stays controllable as call volume grows, without depending on pay-as-you-go quota policies;
- Auditable supply chain: weights, inference frameworks, and deployment paths are all under your control, making security review and fault localization easier.
The other side must also be acknowledged: on long-context understanding precision, complex business-rule restoration, and complex reasoning chains, top closed-source models still hold a clear lead, and catching up is an ongoing process rather than a one-time event. The rational approach is not to pick sides, but to replace “the strongest model” with “the most suitable combination of models”—using open-source models for high-frequency, verifiable, data-sensitive batch tasks, using stronger models for a small number of high-difficulty challenges, with a unified orchestration layer doing the routing in between.
7. Four priorities for enterprise adoption
Turning the above practices into organizational capability takes four things.
First, build your own evaluation set. Extract 30 to 100 reproducible scenarios from real requirements, each paired with machine-decidable acceptance scripts. Without an evaluation set, model selection can only rely on gut feeling, and you cannot tell whether capability has regressed after a version upgrade.
Second, build an orchestration and agent layer. Requirement parsing, phased generation, tool invocation, failure retries, and context trimming—these capabilities should not be hard-coded into business logic but abstracted into reusable orchestration components.
Third, bring the generation process into an observability system. Collect prompt versions, generation latency, token consumption, validation pass rates, and manual rework rates, storing them centrally through a logging service and tracking abnormal fluctuations with monitoring and alerting. Debugging generative applications is far harder than debugging traditional systems; without observability, you can barely locate problems.
Fourth, run on general cloud infrastructure. Store engineering artifacts in object storage, run inference services on a container platform, use message queues to smooth out asynchronous validation tasks, and consolidate model services and gateways behind a unified entry point. These capabilities have mature offerings on mainstream cloud platforms; vendors such as Baidu AI Cloud also provide a full capability matrix spanning model services, compute scheduling, and observability, which enterprises can assemble as needed rather than building the entire infrastructure stack themselves.
8. Conclusion
The most valuable finding from this test is not that some model is strong or weak, but that the pattern of capability decay is predictable: display pages already pass, while rule omissions on list pages and constraint restoration on form pages are the current real bottlenecks, with decay unfolding along two axes—“number of state combinations” and “business rule density.”
Two practical conclusions follow. For individual developers, the right response is to turn “one-shot generation” into a pipeline of “structured requirements + phased generation + automated validation + targeted repair,” using engineering means to compensate for the model’s comprehension gaps—a payoff far greater than waiting for the next generation of models. For enterprises, the significance of open-source models lies not in leaderboard position but in four structural advantages—data boundaries, fine-tuning ability, cost control, and an auditable supply chain—which form a foundation you can rely on at the architecture level over the long term, while the specific strength of any given model is left to the evaluation set and routing layer to handle dynamically.
The ability of large models to generate code is still evolving rapidly. Rather than arguing about who leads right now, it is better to build up evaluation, orchestration, and validation capabilities as soon as possible—these three will not become obsolete when models change.
