Evaluating LLM Frontend Code Generation in Real Projects: Why Comprehension Depth Determines Practical Quality

Why Real Project Evaluation Matters Beyond Benchmark Scores

Current evaluations of LLM code generation capabilities rely primarily on two approaches: public benchmarks covering algorithm problems, code completion, and multi-language programming tasks; and end-to-end validation in real business scenarios. The former measures how well a model scores on standardized problems. The latter answers the question that actually matters: can the model do real work for me?

The gap between these two approaches shows up in three ways:

1. Task distribution differences. Benchmarks focus on single-point, short-chain tasks. Real frontend requirements are long-chain tasks where one requirements document drives multiple pages, multiple components, and multiple interactions. They test global understanding and sustained instruction adherence.
2. Evaluation criteria differences. Benchmarks have a single correct answer. Real requirements are open-ended. “The page runs” and “the page strictly follows the spec” are completely different standards.
3. Engineering constraint differences. Real projects require runnability, deployability, component reuse, and visual consistency. Benchmarks do not cover these.

Using a real project requirements document and asking a model to produce an accessible, runnable frontend application end to end is therefore a far more practical evaluation than chasing leaderboard scores.

Evaluation Design

1. Scenario Selection

The evaluation used three typical page types, creating a capability gradient from simple to complex:

• Login page: Includes a carousel, form validation, and page navigation. A page with fixed structure and mature patterns.
• List page: Includes filters, field display, pagination, and sorting interactions. Tests understanding of business rules.
• Form page: Many fields, many validation rules, and inter-field dependencies. The most error-prone part of frontend requirements.

2. Requirements Document Principles

• Explicit field lists: Name, type, and default value for each field written out clearly.
• Explicit interaction rules: Filter trigger timing, validation timing, and error message behavior listed item by item.
• Structured layering: Page structure first, then component list, then interaction details. Avoid burying information in long paragraphs.

3. Evaluation Dimensions

Dimension	Focus
Functional completeness	How many documented items were implemented
Comprehension fidelity	Deviation between implementation and the document description
Interaction quality	Completeness of state transitions and exception handling
UI and engineering quality	Visual appearance, code structure, maintainability
Generation efficiency	End-to-end time (affected by service load and queueing; for reference only)

Scenario-by-Scenario Results

1. Login Page: Basic Capabilities Are Adequate

The login page achieved good overall completion. The documented carousel, form, and navigation features were all implemented. The page runs directly and can be put into trial use with minor adjustments. For pages with mature patterns, current models can already generate according to convention — login pages appear countless times in training data, so the model is essentially performing high-confidence pattern reproduction.

2. List Page: Gaps in Interaction Understanding Emerge

Problems began to appear with the list page. Filter conditions explicitly stated in the document were missing from the generated page. This was not an isolated incident — multiple list pages showed the same issue. The page skeleton was correct: layout, list structure, and basic display were all present. But interactions like filtering, which require translating business rules into state management and data flow, were systematically omitted.

This reveals a key fact: the model understood the page but not the rules. Rendering a list is a task with strong statistical regularity. Determining how filter conditions change query parameters, when to trigger a refresh, and how filter state coexists with pagination is a task requiring precise instruction following. These impose completely different demands on the model.

3. Form Page: Complex Logic Restoration Falls Short

The form page showed the largest gap. Field types, validation rules, and field dependencies deviated noticeably from the document description — even though the document was written clearly.

The reason is that a complex form is essentially a constraint network: fields depend on each other, validations have ordering, and error messages have priorities. When generating code token by token, the model tends to focus on one aspect and lose another: it gets the field count right but drops dependency relationships; it includes validation but misses the trigger timing.

4. UI Style: Engineering Usability Over Visual Brilliance

The overall visual quality sits at “better than a random prototype, but still short of a mature design system.” Layouts are orderly and components are conventional. The design leans engineering-focused rather than visually striking. The positioning is closer to “make it work first, then talk about making it pretty.” This aligns with the optimization goals of current generative approaches: structural correctness and functional completeness take priority, with aesthetics as a secondary objective.

5. Generation Efficiency: Significant Variance, but Variables Matter

End-to-end generation time varied greatly across models, ranging from a few minutes to over half an hour. It is important to note that timing is affected by multiple factors: inference service concurrency and queueing, output length, whether generation and execution happen in parallel, and resource quotas tied to account tiers. A slower result does not necessarily mean the model itself has poor inference efficiency. However, a long generation cycle does significantly increase iteration cost and directly affects the engineering experience.

Root Causes of the Weaknesses

Looking at all three scenarios together reveals a clear dividing line in model capability:

• Static structural tasks (layout, common components) → strong performance
• Instruction-following tasks (filter rules, validation logic, field dependencies) → noticeable decline

The underlying causes can be understood from three angles:

1. Semantic loss: Going from natural language requirements to code involves multiple conversion stages — intent understanding, task decomposition, state modeling, and code implementation. Each stage loses information, and long documents amplify this.
2. Data distribution: Training corpora contain far more standard pages than pages with complex business rules. Models naturally fit the former much better.
3. Global consistency: Long-chain tasks require the model to maintain memory of and adherence to document details across thousands of generated lines. This is a severe test of context utilization and attention allocation.

Open Source vs. Closed Source: Where the Gap Lies, and Why It Matters

1. The Shape of the Gap

Comparing top closed-source models on the same scenarios, the difference is not about capability existence but about precision. On simple pages, both are close. On complex interactions and form logic, closed-source models demonstrate higher comprehension accuracy and more complete fidelity to the document, and end-to-end speed is typically more reliable. In other words, open-source models have raised the passing bar significantly, but the final stretch of fine-grained comprehension is not yet complete.

2. The Value of Open Source

Even with this gap, open-source models remain highly significant:

• Lower barriers to entry: Teams can deploy privately, keep data within their own domain, and control cost structure.
• Faster ecosystem iteration: Open weights mean the entire industry can fine-tune, distill, and evaluate. Improvement speed far exceeds single-point closed iteration.
• Raising the overall baseline: When open-source models approach the closed-source first tier, competitive focus necessarily shifts from basic capabilities to tooling, services, and deep integration. The whole industry benefits.

3. Implications for Practitioners

• Do not trust any single evaluation: Leaderboard scores can diverge sharply from real-world performance.
• Choose models by scenario tier: Simple page generation can use cost-effective open-source options. Complex business logic still requires stronger models or more human intervention.
• Treat the model as a junior engineer: It needs clear task specifications, staged acceptance, and iterative feedback mechanisms.

Practical Recommendations for Improving Generation Quality

1. Write Requirements Documents in a Machine-Friendly Structure

Natural language paragraphs are unfriendly to models. Structured lists are the most efficient instruction format. Example:

{
  "page": "Product list page",
  "filters": [
    {
      "field": "category",
      "widget": "select",
      "options": "Dynamically generated from dictionary data",
      "trigger": "Reset pagination and refresh the list immediately after change"
    },
    {
      "field": "priceRange",
      "widget": "range",
      "trigger": "Applied as a server-side filter parameter after clicking the query button"
    }
  ],
  "list": {
    "columns": ["Name", "Category", "Price", "Updated At"],
    "defaultSort": "Updated At descending",
    "pagination": { "pageSize": 20, "mode": "server" }
  }
}

2. Generate Module by Module, and Verify Each Stage

Do not expect to generate the entire application in one pass. Break the work into multiple rounds: page skeleton, then list and filters, then form and validation, then integration. Accept only one module per round. When deviations occur, point out the specific discrepancy and request a fix rather than regenerating the entire page.

3. Build an Acceptance Checklist

Convert the requirements document into a checklist and verify each item against “implemented or not” and “matches the description or not.” This serves as both an acceptance tool and a way to discover ambiguities and omissions in the document itself.

4. Distinguish What Can Be Delegated from What Must Be Reviewed

Visual styling and page skeletons can be generated automatically. Validation logic involving funds, permissions, or data consistency must be manually confirmed line by line. Do not assume the model wrote it correctly.

Conclusion

The real-project evaluation delivers a sobering conclusion: current model code generation can reliably cover structurally mature pages, but on complex interactions and form logic requiring precise instruction adherence, comprehension depth remains the decisive weakness. The rapid progress of open-source models is highly significant — it prevents advanced capabilities from being monopolized by a few players and brings capability evaluation back from spectator-style benchmarking to the essential question of whether the model can actually do the work. For developers, building their own evaluation methodology and collaboration workflow is more valuable than debating who is stronger: let the model’s strengths be fully utilized, and let the process cover its weaknesses.