Testing LLMs for Frontend Code Generation: Capabilities, Boundaries, and Evaluation Methods

1. Why High Benchmark Scores Don’t Equal Deliverable Code

In recent years, LLMs have achieved record-high scores on various code generation benchmarks. However, for frontend developers, what truly matters isn’t a number on a leaderboard, but three practical questions:

• If you hand a real-world project requirement document to an LLM, how much can it accurately recreate?
• Can the generated output run and deploy directly?
• What is the true end-to-end cost from prompt submission to a usable page?

Most benchmarks consist of isolated, toy-level tasks, whereas real projects are complex ecosystems combining pages, data fields, validations, interactions, and state transitions. Excelling at isolated tasks does not guarantee engineering success. Therefore, a more valuable approach than chasing benchmark scores is to build an evaluation process that closely mirrors real-world business requirements, answering the ultimate question: Can this model actually deliver?

2. Test Design: Mirroring Real-World Scenarios

Instead of using toy demos, this test utilized frontend requirements from a real project, covering three typical scenarios: a login page, a product list page, and a form page. To control variables, the following design was established:

1. Fixed Requirements via Documentation: Page structures, field lists, filter conditions, validation rules, and interaction logic were explicitly written down to avoid the ambiguity of “verbal requirements.”
2. Open Tech Stack with Recorded Choices: The model was allowed to choose its own stack but ultimately adopted mainstream component-based frontend frameworks, aligning with standard industry practices.
3. Unified Acceptance Criteria: Each page was evaluated across four dimensions: structural fidelity, functional fidelity, interaction fidelity, and direct usability.
4. End-to-End Time Tracking: Total cost was recorded, including queue wait times, generation time, and debugging effort.

The core goal of this design is to transform subjective “feels good” impressions into verifiable evidence.

3. Test Results: A Clear Stratification of Capabilities

1. Standard Pages: High Completion Rate

The login page is the most highly-patterned page in frontend development. In testing, requirements like carousels were fully implemented, and the page functioned correctly without obvious omissions. The output was roughly at a “ready for production after minor styling tweaks” level.

The reason is straightforward: the training data contains an immense volume of login page examples. The model is essentially answering a question it has “seen ten thousand times.”

2. List Pages: Structure Right, Logic Lost

Issues began to surface on the list page. The requirement document explicitly specified filter conditions and interaction logic, but the generated output omitted several filters. This was not an isolated incident; multiple list pages exhibited the same issue.

The page skeleton was correct—tables, pagination, and layout were all present—but the model’s ability to “read the document” was noticeably weaker than its ability to “draw the page.” This indicates that the model’s understanding of the mapping between “data, conditions, and views” remains unstable.

3. Form Pages: Insufficient Field and Validation Fidelity

The form page represents the highest complexity. The test result: the page ran, but the validation logic, field linkage, and interaction details for multiple fields did not align with the documentation. Even with a clearly written document, the model’s comprehension and restoration of complex forms were compromised.

This reveals a critical distinction: generating “something that looks like a form” is easy; precisely restoring “the business rules of every field” is hard. The former is pattern matching; the latter is semantic understanding.

4. Visual Style: Engineering Usability over Design Brilliance

The overall UI style was moderate—better than a quickly drafted prototype, but lacking compared to mature component library templates. The design felt highly “engineering-focused.” In a nutshell: Engineering usability > Product aesthetics.

5. Efficiency and Usability

The test also encountered issues like early-stage queueing and prolonged generation times (likely related to access methods and resource quotas). For real projects, wait times directly erode the “AI efficiency gains,” making this a metric that should not be ignored in evaluations.

4. Root Cause Analysis: Why Can It Draw Pages but Misread Requirements?

Based on the test results, the shortcomings can be attributed to four levels:

1. Training Data Distribution: Standard pages constitute a massive portion of public code repositories, while project code containing complete business rules is rarely open-sourced. Models naturally “have seen pages, but haven’t seen business logic.”
2. Natural Language Ambiguity: No matter how clear a requirement document is, implicit conventions remain. Human developers fill these gaps through experience and follow-up questions; models simply skip what they aren’t sure about.
3. Context Attention Dilution: When a document simultaneously describes layout, fields, validation, and interactions, the model tends to prioritize “explicit, high-frequency” instructions (like layout) and omit “implicit, low-frequency” constraints (like “reset page number when filters change”).
4. Single-Pass Generation Limitations: Generating an entire project’s code in one go requires the model to make all decisions without asking clarifying questions or confirming details, inevitably increasing the probability of errors.

5. Improvement Practices: Bringing Output Closer to “Deliverable”

1. Structuring Requirement Documents

Rewrite the “easily skipped” logic from natural language requirements into machine-friendly structured descriptions. For example:

page: product_list
filters:
  - name: keyword
    type: input
    placeholder: Search by name/ID
  - name: status
    type: select
    options: [All, Active, Inactive]
behaviors:
  - Reset page number to 1 when filters change
  - Clear all filters and refresh list when the reset button is clicked

Structured descriptions make “implicit conventions” explicit. In testing, this significantly improved the restoration rate of filtering and validation requirements.

2. Modular Generation and Page-by-Page Acceptance

Do not expect to generate an entire project at once. Generate in rounds following the sequence: “page skeleton -> data logic -> interaction details.” Accept only one layer per round, breaking large problems into smaller ones.

3. Explicitly Define Acceptance Criteria

Write a “Definition of Done” at the end of the document, such as “All filter conditions must be functional” or “Every field must implement the validations listed in the document.” This acts as a hard constraint for the model.

4. Iterative Refinement over Regeneration

When missing logic is discovered, prioritize patching it with incremental prompts rather than starting over. Incremental fixes preserve the verified portions, cost less, and carry a lower regression risk.

6. Establishing Your Own Evaluation Baseline

More valuable than a “one-off test” is “repeatable testing.” It is recommended that every team maintain a fixed set of evaluation requirements covering the three difficulty tiers: standard pages, list interactions, and complex forms. Run the same test cases whenever a new model or version is released. Core dimensions to track include:

• Structural Fidelity: Are the layout and components complete?
• Logic Fidelity: Are filtering, validation, and linkages implemented as documented?
• Engineering Quality: Component splitting, type definitions, and maintainability.
• Delivery Usability: Can it run and deploy directly?
• End-to-End Cost: Total time spent and failure retry costs.

To conduct evaluations in a private environment, teams can leverage general cloud-based model service categories (such as the model access and hosting capabilities provided by platforms like Baidu AI Cloud) to place open-source weights and commercial APIs into the same evaluation pipeline, ensuring consistent and unified conclusions.

7. Conclusion: Calm Observations on AI Development

This round of testing yielded three main takeaways:

1. For Users: There is a “depth of understanding” chasm between benchmark scores and real-world experience. The convergence of standard scenarios masks the divergence in complex scenarios. Tool selection must return to your own business use cases.
2. For the Industry: Open-source models rapidly catching up to top-tier closed-source models is meaningful progress—it lowers the barrier to validation, making “questioning benchmarks and building custom evaluations” accessible to every team. Evaluation transparency and reproducibility drive ecosystem progress more than a single model’s lead.
3. For Developers: As “drawing pages” ceases to be a bottleneck, value shifts to two ends: defining requirements clearly (structured, unambiguous) on one end, and performing strict acceptance testing (evaluation, regression, engineering quality) on the other. The core competitiveness of frontend engineers is shifting from “implementation capability” to “problem definition capability.”

The transition from “can generate code” to “can deliver a project” requires crossing three mountains: requirement understanding, engineering constraints, and acceptance systems. Whoever builds this bridge first will gain an early advantage in the next phase of AI-assisted development.