Evaluating LLM Frontend Code Generation: Methods and Practices

1. Why “Benchmark Leadership” Does Not Equal “Engineering Readiness”

Whenever a new Large Language Model (LLM) is released, the community quickly circulates claims about it rivaling top-tier international models, often supported by a few screenshots of generated code. However, applying these models in real engineering projects often reveals a gap between hype and reality: the page structure might be built, but specific filtering criteria clearly stated in the requirements document are missing; the overall application runs, but form field validation and interaction details are significantly compromised.

This gap is not accidental. It stems from three structural reasons:

1. Distribution mismatch between evaluation benchmarks and real tasks. Public evaluation benchmarks mostly consist of isolated algorithmic problems and short code completions, testing knowledge coverage. Engineering tasks, however, are typical “multi-constraint restoration” problems. A requirements document simultaneously includes field definitions, validation rules, interaction sequences, and state linkages. The model must satisfy all constraints within a long context. This relies more on instruction following and task planning capabilities rather than pure knowledge retrieval. A high leaderboard score indicates broad knowledge, not accurate constraint restoration.

2. Combinatorial explosion from single-page to project-level capabilities. Generating a login form versus generating a complete project with login, list, form, and detail pages that interact with each other represents a combinatorial, not linear, increase in difficulty. Pages have navigation relationships, state passing, and style consistency requirements. Any deviation in understanding is amplified during the integration phase.

3. The gap between “runnable” and “usable” engineering quality. Whether the code can run directly, if component splitting is logical, if type definitions are complete, and if styles are systematic determines whether the output is a one-time demo or an iterable work-in-progress. These dimensions are often missing in viral evaluations.

For development teams, rather than chasing leaderboard scores, it is more valuable to establish a reproducible internal evaluation method. This method applies equally whether the model is deployed privately via open-source weights or accessed through cloud-based LLM services like Baidu AI Cloud.

2. Three Principles for Evaluation Task Design

The credibility of evaluation results depends primarily on task design. Three principles are recommended:

1. Authenticity: Use real project requirements, not toy demos. Toy tasks (like “write a counter”) are easily solved by all models, offering low differentiation. Real requirements naturally contain ambiguous boundaries, implicit conventions, and cross-page dependencies, effectively exposing a model’s understanding shortcomings. It is recommended to extract a medium-sized module (e.g., “filtered product list + create/edit form”) directly from historical team projects as an evaluation task.

2. Verifiability: Every requirement must be objectively pass/fail. If a requirement simply states “make a beautiful list page,” evaluation degrades into subjective aesthetics. It should be rewritten into verifiable clauses, such as “provide a category dropdown filter; upon selection, immediately refresh the list and reset pagination to the first page.” Verifiability is the prerequisite for objective scoring.

3. Granularity: Test in layers to locate capability boundaries. The same set of requirements should be split into three layers for sequential testing:

• Component-level: Single form controls or cards, testing basic coding capabilities.
• Page-level: Complete single pages, including data fetching, validation, and interactions, testing requirement restoration.
• Project-level: Multiple pages + routing + shared state + unified build configuration, testing engineering and consistency.

Layering facilitates attribution: when project-level results are poor, you can pinpoint whether the model “cannot write the code” (fails at component-level), “misunderstands the requirements” (good at component-level but poor at page-level), or “lacks engineering organization skills” (single pages are correct but fail to integrate into a project).

3. Five-Dimensional Evaluation System

With tasks defined, a scoring dimension system is needed. It is recommended to score independently across five dimensions to avoid “one total score masking all issues.”

1. Requirement Understanding and Restoration (Recommended Highest Weight). The core question: Did the model implement what was documented, and did it hallucinate features that were not specified? Typical checkpoints include whether filtering conditions are complete, whether fields are generated according to definitions, whether validation rules are implemented item-by-item, and whether interaction sequences match the description. In practice, this dimension shows the greatest variance between models. Everyone can build a page skeleton, but omissions where “the document explicitly states it, but the model fails to implement it” occur most frequently in filtering logic and complex forms. This is a systematic shortcoming, not an isolated issue.

2. Functional Completeness. Beyond restoration, check if the end-to-end loop is closed: Are data fetching, loading states, empty states, error states, and submission feedback all present? Many models prioritize implementing the “happy path” while omitting exception paths—precisely where engineering accidents occur most easily.

3. Code Quality and Engineering Level. Check if component splitting is logical (whether all logic is crammed into a single giant file), if state management is clear, if type definitions are complete, and if there are magic strings and duplicated code. Static analysis tools can assist with statistics, supplemented by manual spot checks of key modules.

4. UI and Style Quality. It is recommended to distinguish three levels: structurally correct (layout does not collapse), systematic (spacing, colors, and font sizes form a system, preferably based on design tokens or mature component libraries), and design-oriented (detail polish). Most models can stably achieve the first level. The second depends on training data quality, and the third is generally lacking currently. Evaluation should clarify that “engineering usability takes precedence over visual astonishment” to avoid being misled by pretty screenshots.

5. Delivery Format and Efficiency. Is the deliverable a runnable project (including build configurations and startup instructions) or scattered code snippets? Can it be previewed with one click? How long does generation take? Does the service frequently rate-limit or queue? At the same quality level, “runnable project + results in a few minutes” and “code snippets + half an hour with frequent queuing” offer completely different engineering value. The efficiency dimension also looks at stability: does the result variance differ much across multiple runs of the same task?

4. Structured Writing for Requirement Documents

Half of the reproducibility of an evaluation comes from the quality of the requirement document. It is recommended to write requirements using a structured format to make implicit conventions in natural language explicit. Below is an example of a filtering requirement for a list page:

page: product-list
filters:
  - id: category
    type: select
    options_source: GET /api/categories
    behavior: On selection, immediately refresh the list and reset pagination to page 1
  - id: price_range
    type: range-slider
    behavior: Trigger request after drag ends, debounce 300ms
  - id: keyword
    type: input
    behavior: Trigger on Enter key or clicking the search button
list:
  page_size: 20
  default_sort: created_at desc
  empty_state: Display "No data available" message and provide a "Reset filters" button
  error_state: Display a retry entry on request failure

This approach has three benefits:

1. Enumerable constraints. Each behavior acts as an acceptance criterion. During evaluation, check them one by one; omissions are clear at a glance.
2. Directly reusable as generation constraints. Feeding the same structured document to different candidate models is equivalent to giving them the same test paper, making horizontal comparisons meaningful.
3. Facilitates regression testing. After a model version upgrade, simply re-run the same document to compare capability changes.

Form requirements follow the same logic. Write validation rules as item-by-item verifiable clauses:

form: product-create
fields:
  - name: title
    required: true
    error_message: "Please enter the product name"
  - name: price
    type: number
    rules:
      - min: 0.01
      - message: "Price must be greater than 0"
  - name: stock
    type: integer
    rules:
      - min: 0
submit:
  loading: Disable the submit button and show loading state
  success: Show success prompt and redirect to the list page
  failure: Preserve form data and display backend error messages

5. Acceptance Checklist and Scoring Method

Each evaluation task should be accompanied by an acceptance checklist, verified item by item during evaluation:

• All three filter items are implemented, and interaction behaviors match the document
• List refreshes and pagination resets after filter changes
• Form required field validation and numeric range validation work item by item
• Submission process has a loading state; failed submissions preserve entered data
• Empty and error states are handled
• Project can directly install dependencies and start
• Code is split by page / component / request layer, without single-file堆积 (stacking)
• Type definitions are complete, without widespread generic types (if using a typed language)

Scoring should adopt a tiered system rather than a 100-point scale: record 0 (not implemented), 1 (implemented but does not match description), or 2 (fully matches) for each item, and aggregate by dimension. The benefit of a tiered system is that it exposes the intermediate state of “implemented but done incorrectly”—a type of problem missed by binary “yes/no” statistics, yet the most direct evidence of requirement understanding capability.

To control randomness, it is recommended to run the same task on the same model at least two to three times: take the best score to measure the capability ceiling, and the average score to measure stability. Recording both answers the question: “Is this model occasionally amazing, or consistently reliable?”

6. Common Shortcomings and Attribution

Using the above method for evaluation, most models will exhibit a similar distribution of shortcomings. Understanding their causes helps targeted improvement:

1. Complex Forms and Filtering Logic are Prone to Omission. The characteristics of these requirements are: numerous constraints, cross-element dependencies (e.g., “filter change resets pagination” involves two modules), and implicit conventions (e.g., “preserve data on submission failure”). During generation, models tend to prioritize visible visual structures, and long-tail constraints are easily truncated or ignored. The attribution is mainly insufficient long-context constraint retention and instruction-following precision, rather than lacking coding capability.

2. Generally “Plainly Engineered”, Lacking Design Sense. Generated interfaces are often structurally correct but visually bland, resembling generic admin template levels. This is because models learn the “greatest common divisor” of massive real-world code, while design sense requires additional design knowledge injection and subjective trade-offs, currently an industry-wide challenge.

3. Significant Variance in Generation Speed and Stability. Different services have vastly different queuing and rate-limiting strategies. The experience under free quotas cannot represent the paid capability ceiling. Evaluation reports should note the invocation method and resource conditions; otherwise, conclusions are incomparable.

4. Differences in Delivery Format. Some services output complete runnable projects, while others only provide code snippets. The former significantly reduces integration costs. This should be recorded as an independent dimension during evaluation, rather than glossed over within “capability.”

7. Practical Recommendations for Improving Generation Quality

The ultimate goal of evaluation is to improve usage outcomes. The following practices are effective in most scenarios:

1. Requirement Side: Structured, Explicit, with Acceptance Criteria. Use the structured document from Section 4 as the standard format for communicating with the model. Explicitly write out all interaction sequences and edge cases, and append an acceptance checklist at the end of the document. The model’s adherence to “verifiable clauses” is much higher than its adherence to adjectives (“beautiful,” “user-friendly”).

2. Process Side: Step-by-step Generation, Skeleton First, Details Later. Generating an entire project at once easily loses constraints. A more robust approach is to have the model output the page structure and technical plan first, confirm it, then generate page by page, and finally add validation and interaction details. With more focused context at each step, the constraint restoration rate improves significantly.

3. Iterative Correction: Feed Errors and Checklists Back. After the initial generation, compile runtime errors and failed acceptance items into correction instructions and submit them again. Usually, it takes two to three rounds to converge to a usable state. Replacing the expectation of “the model gets it right in one shot” with a “human-machine collaboration for rapid convergence” workflow better aligns with current technological realities.

4. Engineering Side: Use Tools as a Safety Net. Integrate type checking, static scanning, and critical path testing into the generated code. Machine-detectable issues should not rely on the human eye. Combine this with manual code reviews focusing on business logic correctness.

8. Conclusion

The narrative of “the new model rivaling model X” will continue to emerge. However, for engineering teams, the truly valuable question is never “what is its benchmark score,” but “on my requirement documents, how many constraints can it restore, how many defects remain, and how many rounds does it take to converge.” Establishing an evaluation system based on real requirements, verifiability, and layered granularity—shifting model selection from emotion-driven to data-driven—is the reliable path to transforming LLM capabilities into engineering productivity.