Testing LLMs on Full-Project Frontend Generation: Capability Boundaries and Evaluation Methods

    A structured, reproducible framework for evaluating large language models that generate complete frontend projects from requirement documents. It covers how to write machine-checkable specs, score outputs across six dimensions, pinpoint where capability degrades across login, list, and form pages, the technical causes of that degradation, and the engineering practices—layered generation, configuration-driven contracts, automated check loops, and cloud hosting—that move generated output toward a deliverable state. Includes a ready-to-use evaluation checklist.

    1. From “Code Completion” to “Project Generation”: Why the Evaluation Standard Has to Change

    Over the past two years, the capability boundary of AI coding tools has shifted noticeably. Early on, they answered “how should this line be written”; now they answer “how should this requirement be implemented.” When the input changes from a function signature to a requirements document, and the output changes from a code snippet to a fully runnable project, the old evaluation methods—checking whether a single function is elegant, or whether completions feel smooth—no longer apply.

    The reason is that project-level generation turns the problem from “local correctness” into “long-range consistency.” A requirements document scatters dozens of implicit constraints across its pages, and the model has to keep data structures, page structure, interaction states, validation rules, and visual conventions all consistent with one another at the same time. Any single omission shows up as “the page runs, but the behavior is wrong.”

    For that reason, evaluating these tools needs a structured method, not reliance on an overall impression of “wow, that looks impressive.”

    2. A Reproducible Evaluation Framework

    2.1 Input Side: Writing Requirements as Machine-Checkable Specs

    The first step of evaluation is not choosing a model, but writing the requirements document into a form where each item can be checked off. Vague adjectives (“smooth experience,” “beautiful layout”) cannot be verified and must be converted into explicit rules. It helps to split requirements into three categories:

    • Structural: what blocks the page contains, their hierarchy, and the field list with field types;
    • Behavioral: trigger conditions, interdependencies, and boundary states (empty, loading, error, no-permission);
    • Validation: the rule for each field, when the message appears, and the message text.

    An example of a filter-bar requirement written as a spec:

    1. page: product_list
    2. blocks:
    3. - filter_bar:
    4. fields:
    5. - name: keyword
    6. type: text
    7. rule: [trim, max_length=50]
    8. - name: category
    9. type: multi_select
    10. source: dictionary_api
    11. - name: price_range
    12. type: range
    13. unit: CNY
    14. rule: [min<=max]
    15. behavior:
    16. - on_change: debounce(300ms) -> refresh_list(page=1)
    17. - on_submit: merge_active_filters -> query
    18. - on_reset: clear_all -> query(default_state)
    19. - data_table:
    20. columns: [name, category, price, status, created_at]
    21. states: [loading, empty, error, success]
    22. pagination: {mode: server_side, default_page_size: 20}

    The key point is that every item can be checked automatically. Whether debounce exists, whether reset actually clears all conditions, whether pagination is server-side—all of these can be decided by test code, not by eyeballing.

    2.2 Output Side: Score by Dimension, Not as a Whole

    I recommend a six-dimension score, 1–5 points each, with mandatory evidence (which page, which requirement ID):

    Dimension What to judge Evidence
    Structural completeness Whether blocks, fields, and routes are all present Requirement item coverage
    Functional fidelity Whether interactions and linkages are implemented Test case pass rate
    Validation rigor Rules, timing, message text Boundary case pass rate
    Engineering conventions Directory organization, type definitions, maintainability Static analysis results
    Visual completeness Hierarchy, spacing, state styling Visual regression comparison
    Generation efficiency End-to-end latency and usability Timestamp and usability records

    The overall score is only supplementary. What really matters are the two quantifiable metrics: “requirement item coverage × test case pass rate.”

    3. Layered Testing: Where Capability Degrades

    Using a real requirements document containing three kinds of pages—login, list, and form—as a sample, a fairly consistent pattern emerges.

    3.1 Login Pages: Completeness Near Usable

    This kind of page has low requirement density and mature patterns: carousel, account/password input, captcha, agreement checkbox, error prompts. The model can generally produce all of it in one pass, and the output is usually a fully accessible, complete project rather than an isolated code block—an important point for later verification, because only something that runs closes the loop. The conclusion at this layer is usually “tweak it and ship.”

    3.2 List Pages: Comprehension Starts to Slow Down

    Once you introduce “filtering + linkage + server-side pagination,” problems start to appear: the page skeleton is correct, the table fields are correct, but filter conditions are missing or only one or two are implemented, and the linkage relationships (reset to page 1 after a condition changes) are almost always omitted.

    It is worth noting that this kind of omission is usually systematic rather than occasional. The same type of omission shows up across multiple list pages in the same document, which indicates it is not sampling noise but the model’s tendency to compress “repeated patterns + implicit rules”: it tends to generate a “reasonable-looking generic list” rather than implementing every filter item listed in the document one by one.

    3.3 Form Pages: Fidelity Degrades Most Noticeably

    Forms are the most constraint-dense of the three page types: field types, required flags, conditional visibility, cross-field validation, whole-form validation before submission, and locating the specific field after a failure. In testing, common problems are that field relationships get simplified (a one-to-many relation collapsed into a single field), conditional visibility is missing, and validation only checks for non-empty values.

    This points to a practical conclusion: a clearly written requirements document does not mean the model will faithfully reproduce it. Document clarity determines the checkability of the evaluation, not the fidelity of generation.

    3.4 Visual Quality and Efficiency: Usability and “Wow Factor” Are Two Different Things

    The output is usually “a bit better than a wireframe prototype, but still short of the look of a mature component library and design system,” and it leans toward being engineering-oriented overall. Meanwhile, generation latency is significantly longer than in code-completion scenarios; anywhere from a few minutes to over half an hour is common. Some services also throttle, queue, or cap free quotas at peak times, further amplifying the difference in experience.

    The two things must be looked at separately: latency is affected by service load and quota policies—it is an environmental variable; comprehension accuracy is the model’s capability. Treating queue time as evidence of model capability is the most common mistake in evaluation.

    4. Why the Degradation Happens: Three Technical Causes

    1. A level difference between effective context and instruction following: the longer the document, the more easily the specific items in the middle get “summarized” away. The model holds the global structure (which pages exist) well but holds detailed enumeration (each filter item) poorly.
    2. No closed loop of execution feedback: a single generation is open-loop. Without compilation, without a successful run, without a failed-assertion signal, omissions do not surface themselves.
    3. High cost of cross-file consistency: type definitions, state management, routing, and the API layer all need to change in sync. The more scattered the changes, the higher the probability of forgetting.

    The corresponding mitigations are also straightforward: reduce the granularity of a single task, provide executable acceptance criteria, and replace manual review with automated checks.

    5. Moving Generated Output to a Deliverable State: Engineering Practice

    5.1 Layered Generation: Contract First, Then View

    Do not ask for “the entire project” in one shot. Split it by dependency order:

    1. Step 1: produce the data model and API contract (fields, types, error codes)
    2. Step 2: produce the page skeleton and component tree (no business logic)
    3. Step 3: fill in interaction and validation page by page (one page at a time)
    4. Step 4: apply global style tokens and design-system constraints

    Each step produces a reviewable intermediate artifact, so errors are cut off early rather than accumulating in the final generation.

    5.2 Replace Natural-Language Descriptions with Configuration Constraints

    Extract repeated interaction rules into configuration, so the generation target changes from “understanding prose” to “filling in a structure”:

    1. // Declarative interaction contract; the generation result must align with it item by item
    2. interface FilterSpec<T> {
    3. fields: Array<{
    4. key: keyof T;
    5. control: "text" | "select" | "multiSelect" | "range";
    6. rules?: string[];
    7. }>;
    8. onMutate: "debounce" | "submit";
    9. resetPageOnChange: boolean;
    10. }

    Structured input noticeably improves fidelity, because it eliminates the space for ambiguity: the model does not have to guess “should filtering trigger linkage or not.”

    5.3 Build a Closed Loop of Automated Checks

    Right after generation, run three layers of checks, turning “I feel like a filter is missing” into “the test is red”:

    1. # Pseudocode, illustrating flow order; choose concrete tools per your team's existing engineering stack
    2. typecheck # Type consistency: does the implementation match the contract
    3. lint # Convention checks: structure, naming, forbidden patterns
    4. unit-test # Rule validation: required, length, range, cross-field linkage
    5. e2e-test # Behavior validation: full filter -> query -> reset -> pagination chain
    6. visual-diff # Visual regression: screenshot comparison of key states

    The unit tests should be generated directly from the spec, not written by hand afterward:

    1. it("resets to the first page when a filter condition changes", async () => {
    2. await page.goto("/products?page=3");
    3. await selectOption("[data-testid=category]", "A");
    4. await expect(page).toHaveURL(/page=1/); // Contract: change resets to first page
    5. });
    6. it("the reset button clears all filter items", async () => {
    7. await fillKeyword("phone");
    8. await selectCategory("A");
    9. await click("[data-testid=reset]");
    10. await expect(inputKeyword).toHaveValue("");
    11. await expect(selectCategory).toHaveValue("");
    12. });

    The value of this loop is not just filtering out defects; it also forms the input for the next generation. Pasting the failing cases back into the context lets the model do targeted fixes rather than regenerate the whole thing.

    5.4 Cloud Hosting: Making the Output Runnable

    Whether a project can be accessed directly and shared for verification within a team often depends on whether the infrastructure is in place. The general approach is to put the generated static artifacts into object storage and distribute them via a CDN, deploy dynamic services on a container platform, decouple time-consuming tasks with a message queue, and then hook into logging and monitoring/alerting to form a complete “generate—deploy—observe” pipeline. Only with logs and metrics can you answer “what exactly about this generation is hard to use,” and only then does evaluation have an objective basis.

    6. Open-Source Models Becoming Comparable: Significance and Preconditions

    When an open-source model that can be self-deployed is rated “close to the top closed-source models” on a specific task, the significance is not the leaderboard ranking but three verifiable engineering values:

    • Self-hostable: code and business data stay inside the private network, which is a hard prerequisite for compliance-sensitive scenarios;
    • Fine-tunable: you can adapt it to your own codebase, component library, and historical requirement documents—this usually improves actual fidelity more than a general leaderboard score does;
    • Cost and control: inference resources can be self-held and elastically scheduled by peak, so the long-term cost structure is predictable.

    But this optimism needs boundaries:

    1. Leaderboard score and task completion are not the same thing. The same model performs very differently across different task distributions, and a lead on general benchmarks cannot be used to infer “my form page will be generated correctly in one pass.”
    2. The gap mainly shows up in depth of comprehension, not surface completeness. Anyone can do the login page well; it is the “constraint-dense zones” like filter linkage and cross-field validation that are the dividing line. Evaluation must sample these zones heavily.
    3. The open-source advantage has to be realized through engineering capability. Self-deployment means taking on inference optimization, concurrency scheduling, version upgrades, and effect regression yourself—this is a cost, not a free lunch.

    So the correct posture for tool selection is: build your own task set and acceptance criteria, and make decisions with coverage and pass rate, not with the impression from a single demo or a public score.

    7. A Ready-to-Use Evaluation Checklist

    • Requirements are split into structural / behavioral / validation items, each checkable
    • At least three page types are covered: low-constraint (login), medium-constraint (list), high-constraint (form)
    • High-constraint areas have no fewer than 50% of the total test cases
    • Coverage and test case pass rate are recorded, not just an overall impression
    • Generation latency and usability are recorded separately, not counted in the capability score
    • The automated check pipeline (type, lint, unit, e2e, visual) is in place
    • Failing cases are fed back into generation to verify targeted-fix ability
    • If evaluating an open-source option, add three items: private deployment, fine-tuning, and concurrency load testing

    8. Closing

    The capability of full-project generation is already enough to change the shape of the development process: the cost of prototype validation, admin/back-office systems, and internal tools drops sharply. But treating it as an “equivalent compiler for requirement documents” is still premature—in constraint-dense areas, comprehension degradation is a stable phenomenon, not something a prompt trick can work around.

    What really determines output quality is the structuring on the input side and the executable acceptance criteria on the output side. Whoever can write “is the filter actually filtering, does reset clear everything, does validation trigger in stages” into a test that fails—that is who can push these tools from “looks good” to “actually usable.” This is also the right posture for open-source models to compete: what is being compared is not the wow factor of a single demo, but the certainty of being self-hostable, fine-tunable, and repeatedly verifiable by an automated acceptance system.