Evaluating LLMs for Frontend Project Generation: A Scenario-Based Approach to Testing and Deployment

    Public leaderboard scores say little about whether a large language model can turn a real requirements document into a frontend project that actually runs, works, and is maintainable. This article walks through a scenario-based evaluation of an LLM building a frontend project with login, list, form, and detail pages. It covers how to set up a consistent testing environment, define weighted dimensions, and write numbered, verifiable acceptance items; reports where the model succeeded and where it fell short relative to leading closed-source models; and turns those findings into practical improvements — structured specs, contract-first design, acceptance-driven prompts, a generate–verify–fix loop, and automated quality gates. It is written for engineers and teams who want a repeatable way to evaluate and adopt code-generation models.

    Introduction: From Benchmark Scores to Real Scenarios

    Every time a new model version ships, the community is quick to declare that its capabilities are “approaching those of a leading international model.” Such verdicts usually rest on single-metric results from public leaderboards. They tell you how a model performs on standardized questions, but they don’t answer a more practical one: handed a real requirements document, can it deliver a project that runs, works, and is maintainable?

    Take “generate a complete frontend project from a requirements document” as an example. The task looks simple, but it is actually a long chain: read and understand the requirements, break them into tasks, scaffold the structure, implement interactions, handle edge cases, and keep everything runnable. If any link is understood poorly, the whole result suffers. That is why evaluation has to happen in real scenarios and be verified through a reproducible process.

    This article uses a frontend project spanning a login page, a product list page, a form page, and a detail page as a worked example, and discusses how to design this kind of evaluation and how to improve the way you use these models based on the results.

    1. Setup Before You Evaluate: Environment, Criteria, and Test Set

    1.1 A Consistent Environment and Execution Standard

    • Fixed inputs: the same requirements document, the same technology stack, and the same starting state (a blank project or an empty workspace).
    • Fixed generation method: if you are comparing one-shot generation against multi-round refinement, every round must start from the same input, so you can analyze which round closed which capability gap.
    • A single timing baseline: start the clock when generation begins and stop it when the project opens in a browser and the homepage renders correctly. Queue and rate-limiting fluctuations are recorded separately and never folded into the timing metric.
    • Agree on acceptance criteria first: define the pass bar before testing starts to avoid confirmation bias.
    • Repeat the run: run the same task at least three times and take the mode, to filter out one-off factors.

    1.2 Evaluation Dimensions and Weights

    Dimension Focus Suggested Weight
    Requirement comprehension Whether the explicit requirements and implicit constraints in the document are accurately identified 25%
    Feature completeness Whether the main features on each page are present, and whether any whole blocks are missing 20%
    Interaction and details Validation rules, linkage, and empty/loading/error state handling 20%
    Engineering quality Structure, state management, API-layer design, and type constraints 15%
    Runtime stability Whether it builds and runs, and whether there are console errors 10%
    Generation efficiency End-to-end time, retry count, and queue/rate-limit impact 10%

    The weights can shift with the team’s goals. If the priority is “get a demo-ready version fast,” raise the weight on generation efficiency; if the target is a production deliverable, raise the weight on engineering quality and runtime stability. One clarification: these weights represent the share each dimension’s score contributes to the total. How each dimension is scored from 0 to 100 is decided by the acceptance cases below.

    1.3 Case Design: Page Layers and Numbered Acceptance Items

    The evaluation kit contains a requirements document and a code-free interactive prototype. The prototype exists to align on acceptance criteria — not to let the model “look at the picture and copy the answer.” Acceptance items are organized as “page — number.” The table below uses the list page as an example:

    ID Acceptance Item How to Check
    L-01 First load shows all data with the correct total count Compare the returned count
    L-02 Keyword fuzzy-searches the name field Enter a fragment and check the hits
    L-03 Multiple conditions combine with AND Set several conditions and verify the result set
    L-04 Changing any condition automatically returns to page 1 Change a condition after paging
    L-05 “Reset” restores defaults and refreshes the list Click reset and verify inputs and results
    L-06 Filters sync to the address bar and persist after refresh Refresh the page and verify conditions and page number
    L-07 Sorting still applies after paging again Page through and verify the order
    L-08 Pagination is server-side, with correct page size and total Check request parameters and response structure
    L-09 An empty list shows an empty state and a clear-filters entry Construct a no-result condition
    L-10 A failed request shows an error message and a retry entry Simulate an API error

    L-01 through L-08 can run automatically against the API documentation and end-to-end scripts. L-09 and L-10 can be triggered with API stubs (mocks) that force error states, after which scripts assert the page’s behavior. When state-based cases cannot be fully automated, a human can spot-check them quickly.

    The login and form pages are numbered the same way, covering every explicit requirement in the document. During scoring, each item is judged “pass / partial / fail,” worth 1 / 0.5 / 0 points respectively, then averaged per page. If the document is structured enough, this whole acceptance set can be generated programmatically from the spec, guaranteeing that every model and every round is judged by exactly the same standard.

    2. Test Results and a Layer-by-Layer Breakdown

    2.1 Overall Performance

    • Run result: a complete project generated in one shot started directly in the unified environment, with the first screen rendering normally.
    • Time: roughly half an hour from kicking off generation to a project that opens. Generation involved queuing, and concurrency-quota notices appeared multiple times; this server-side queuing and rate-limiting clearly weighed on generation time.
    • Output form: an accessible project URL, not just code fragments. Being able to run and preview it significantly lowered the cost of verification.

    2.2 Login Page: High Completion

    All the documented requirements — carousel display, account/password input, “remember login state,” and redirects — were implemented, along with the required-field validation and error messages on submit. It is essentially ready to use and only needs polishing.

    Pages like this have a fixed structure and a mature pattern, and there is plenty of public material on them. The model has seen many similar forms during training, so high completion is expected. It proves the model has the basic ability to produce a usable page from a document, but it does not discriminate much between models.

    2.3 List Page: Where Comprehension Starts to Slow Down

    The page skeleton, table, and pagination controls were all generated correctly. But a group of related requirements — “combined filters + page reset + syncing conditions to the URL” — was only partly implemented:

    • some filter fields were missing entirely;
    • the page number did not reset when a condition changed;
    • the URL did not reflect the current conditions, so filter state was lost on refresh.

    What is notable is that these were not hidden constraints — they were explicit items listed one by one in the document. So the problem is not whether the model can understand a single sentence, but whether it can hold several interrelated requirements together as one consistent whole. When the page framework is right but local features are missing, this “right shape, missing substance” result is exactly what a single overall score tends to hide.

    2.4 Form Page: Insufficient Fidelity to Detail

    The form as a whole could be submitted, but field types, required-field rules, cross-field validation, and conditional visibility diverged from the document in several places. For example:

    • the quantity field was not restricted to non-negative integers;
    • a submit was not blocked when the end time was earlier than the start time;
    • some options that should appear only under certain conditions were always shown.

    A form is essentially a set of business rules, not a set of controls. When the rules are “close enough,” the result can be wrong — and these errors often do not surface until the business is actually running.

    2.5 Head-to-Head with Leading Models: The Gap Is in Depth of Understanding

    As a baseline, we ran the same task in the same environment with the same document using a set of currently leading closed-source models, and combined that with common observations from similar community evaluations:

    • Login page: roughly on par — all reach a usable level;
    • List page: the baseline models restored filter conditions and field structures more completely, with more coherent related logic;
    • Form page: the baseline models held a clearer advantage in field relationships, validation rules, and conditional linkage;
    • Generation time: the baseline models typically finished in minutes; our test was slower by an order of magnitude.

    This result comes down to one point: the gap is mainly in the depth and consistency of requirement understanding, not in whether the code runs. The speed gap also has an objective cause: this test used a free tier with concurrency limits, whereas the baseline models were usually called through paid APIs. Billing model, concurrency quota, and inference infrastructure all affect generation speed, so raw generation time alone does not equal a capability gap. A more rigorous approach is to record “end-to-end usable time” and analyze it alongside task complexity and retry count.

    The task structure also shows a pattern: the more structured the requirements, the clearer the constraints, and the more templated the pages, the smaller the gap; the more the requirements depend on implicit business rules, cross-field relationships, and error scenarios, the wider it gets. The contrast among the login, list, and form pages illustrates exactly this.

    3. From Results to Practice: Improving Generation Quality

    The point of evaluation is improvement. Around the failure points above, generation quality can be raised in five ways.

    3.1 Make Requirements “Machine-Executable”: A Structured Spec

    Natural-language documents suffer from ambiguous references and scattered constraints. Turning the key constraints into a structured spec before handing it to the model noticeably reduces comprehension drift. Take the list page:

    1. page: ProductList
    2. route: /products
    3. data:
    4. source: GET /products
    5. fields: [id, name, price, stock, status, updatedAt]
    6. filters:
    7. - name: keyword
    8. type: text
    9. match: name_contains
    10. default: ""
    11. - name: status
    12. type: select
    13. options: [on_sale, off_shelf]
    14. default: all
    15. - name: priceRange
    16. type: range
    17. unit: CNY
    18. default: null # null means no restriction
    19. interactions:
    20. - on_filter_change: reset_page # condition change -> back to page 1
    21. - apply: manual # request only after clicking "Search"
    22. - reset: restore_defaults_and_reload
    23. - sync_url: true # write conditions and page number to the URL
    24. - sort_persists_across_pages: true
    25. pagination:
    26. mode: server_side
    27. pageSize: 20
    28. states:
    29. empty: { text: "No data", action: clear_filters }
    30. error: { text: "Failed to load", action: retry }

    The value of a structured spec is that “everything is verifiable”: every rule written in the spec maps directly to an acceptance case.

    3.2 Contract First, Implementation Second

    Before generating, produce an “API contract + data model” that pins down fields, types, request parameters, and response structures, and confirm it is correct before generating page code. The API contract describes the request method, path, query parameters, response fields, and error codes, and can be written in an industry-standard API description format. This gives page behavior a reference to check against, so the model stops “coding while guessing.”

    3.3 Bake the Acceptance Checklist into the Generation Constraints

    Feed the numbered acceptance items in as a “definition of done” and require the model to self-check against them one by one. An example prompt (swap in any code-generation tool’s system prompt):

    1. You are a frontend engineer. Implement the "Product List Page" strictly
    2. according to the requirement spec below.
    3. Implementation requirements:
    4. 1. Implement filters, interactions, and pagination exactly as specified below;
    5. 2. For anything the spec does not cover, follow the conventions already used
    6. by existing pages in the project — do not add features beyond the requirements;
    7. 3. After finishing each feature, self-check against the acceptance checklist
    8. item by item and output a self-check table;
    9. 4. When unsure, list "open questions" at the end of your reply — do not silently
    10. simplify.
    11. Acceptance checklist:
    12. L-01 First screen shows all data with the correct total count
    13. L-02 Keyword matches the name field fuzzily
    14. ...
    15. L-10 Show the error and a retry entry when a request fails

    3.4 Phased Generation and a “Generate–Verify–Fix” Loop

    Generating an entire project in one shot tends to fix one thing and break another. A safer approach is to split it into two steps: scaffold first, then refine page by page.

    • Step 1: scaffold the project (routing, layout, shared components, request wrappers).
    • Step 2: generate pages one at a time, running automated checks immediately after each.
    • Step 3: feed the failing items and error messages back to the model for targeted fixes on the existing code, rather than regenerating the whole page.
    1. // Example: generated output automatically enters the "verify-fix" loop
    2. async function generateWithFeedback(generate, verify, { maxRounds = 3 } = {}) {
    3. let attempt = await generate();
    4. for (let round = 1; round <= maxRounds; round++) {
    5. const report = await verify(attempt); // build, unit tests, end-to-end, lint
    6. if (report.passed) {
    7. return { artifact: attempt, rounds: round };
    8. }
    9. // only feed back failed cases and key errors to avoid unbounded context growth
    10. attempt = await generate({ feedback: report.failures });
    11. }
    12. // if still failing after the max rounds, hand off to manual review
    13. return { artifact: attempt, needsHumanReview: true };
    14. }

    The most critical part of the loop is that verification must be automated. Many of the issues in this evaluation are behavioral defects, such as “the page number does not reset after a condition changes.” Only by actually running the page and simulating user actions can you catch these reliably.

    3.5 Engineering Acceptance: Quality Gates

    Before generated code lands in the repository, it should pass a common set of quality gates:

    • Static and type checks: consistent code style, fewer type errors;
    • Unit tests: covering pure functions, data transforms, and form-validation rules;
    • End-to-end tests: mapped directly to the numbered acceptance items, e.g., L-04’s “change a condition and return to page 1”;
    • Build checks: ensuring the artifacts package and deploy cleanly;
    • Human review: focused on state management, API-layer encapsulation, and error handling.
    1. test('L-04 page number resets to 1 after changing a filter', async ({ page }) => {
    2. await page.goto('/products?page=3&status=on_sale');
    3. await page.getByPlaceholder('Search product name').fill('headphones');
    4. await page.getByRole('button', { name: 'Search' }).click();
    5. const url = new URL(page.url());
    6. expect(url.searchParams.get('page')).toBe('1');
    7. expect(url.searchParams.get('keyword')).toBe('headphones');
    8. });

    3.6 Iterative Testing: Before vs. After

    After optimizing the process with the methods above, we re-tested with the same requirements document and compared results item by item against the numbered cases:

    • List page: 9 of 12 acceptance items passed; after improvement, 11 passed, with the remaining one being an edge case in URL syncing;
    • Form page: 8 of 15 acceptance items passed; after improvement, 13 passed;
    • Generation time: with structured requirements and a contract defined up front, rework rounds dropped and end-to-end time fell noticeably.

    These numbers are representative results after multiple repetitions in the same environment — the emphasis is on relative change, not absolute values. Another interesting observation: after the improvements, first-shot generation quality was closer to the output of the first stage in the “scaffold–page” two-stage process, which suggests that phased generation effectively eases the forgetting and simplification that plague long-chain tasks.

    4. The Bigger Takeaway: Beyond Open vs. Closed, It’s the Engineering Loop

    The difference between open- and closed-source paths has always been a focus of LLM discussion. Open source has advantages in deployment flexibility, data control, and secondary customization; closed source is usually more mature in response speed, supporting services, and tooling ecosystems. But for engineering teams, what matters is not “who is benchmarking whom” — it is these points:

    1. Make evaluation scenario-based: public leaderboards reflect general capability; business scenarios decide real value. Translating your own requirements into test cases via numbered acceptance items is the only way to reach conclusions that mean something to you.
    2. Judge speed within the full pipeline: generation time is only the start; the time spent debugging, editing, and reviewing is the real cost. A model that lacks “depth of understanding” may generate fast in a single pass but pay a higher price in rework.
    3. Rule-heavy tasks are the dividing line: form-heavy, filter-heavy, permission, and state-transition scenarios tax requirement understanding the most — and are the most worth prioritizing for process changes.
    4. The model is just one stage in the pipeline: structured requirements, API contracts, automated verification, and quality gates — this engineering foundation sets the floor for generation results and stays valid across model versions.

    Meanwhile, operational factors such as call quotas, queuing, and rate-limiting have a real impact on experience. Organizations should factor them into selection decisions rather than comparing only how models score on their own answers.

    Conclusion

    This test shows that, measured by “generating a complete frontend project,” the model is already near usable on templated, clearly structured pages, but still trails leading models by a clear margin on rule-heavy, richly interrelated pages. That gap is about “depth of understanding” — which is exactly the part evaluation should quantify.

    For developers, rather than arguing over “who is benchmarking whom,” the better move is to do three things: write requirements clearly, establish acceptance criteria, and automate verification. When evaluation becomes a repeatable, traceable engineering process, any model upgrade can be validated quickly and adopted smoothly. That is the real prerequisite for LLMs to enter the development workflow.