LLM Code Generation Evaluation Methods: Using Understanding Depth to Guide Model Selection
This article presents a reproducible framework for evaluating how well large language models (LLMs) generate code from requirements. Rather than judging models by line count or a single polished demo, it focuses on whether a model truly understands the implicit constraints in a requirements document. Using login, list, and form pages as test scenarios, it provides a verifiable test configuration, weighted scoring checklists, a scoring formula, layered results, common misjudgments, and selection guidance so teams can decide whether a model fits real projects.
1. From “Can Generate” to “Can Understand”: The Evaluation Problem Needs to Evolve
Evaluating LLM code generation cannot stop at whether a demo produces a page. It must also determine whether the model understands the implicit constraints inside a requirements document. Focusing on login pages, list pages, and form pages, this article offers a reproducible evaluation method, checklists, scoring formulas, and selection recommendations to help teams decide whether a model is suitable for real projects. The evaluation is not about line count or how impressive a single demo looks, but about whether fields are complete, whether interactions are wired together, whether validation is thorough, whether the output is buildable, and whether the pages are maintainable.
In many demo scenarios, a single natural-language instruction can quickly generate a runnable interface. But once you move into a real project, the evaluation target must upgrade to understanding depth. Does a list page have only filters, or do those filters actually narrow the results? Does a form page have only fields, or do the fields interact conditionally? Does a login page have only input boxes, or does it give clear feedback for empty values, format errors, and failed submissions? Judging LLM code generation capability therefore means looking past “did it generate anything” and asking “did it understand the implicit constraints in the requirements document.”
2. Reproducible Test Configuration: Making the Evidence Verifiable
All of the items below are required. If key fields are missing — especially model version, runtime environment, mock data, or build and preview commands — you must not output quantifiable measured conclusions, and may only offer qualitative observations.
2.1 Evaluation Target and Version
- Candidate model X: model name — required; version number — required; retrieval date — required.
- If the exact version cannot be confirmed, label it as “the model tested in this run” and explicitly note the limitation that the version cannot be verified.
- If candidate models Y, Z, and others are included, fill them in using the same format.
2.2 Test Date and Prompt
- Test date — required.
- Keep the original prompt verbatim. For example: “Generate a directly runnable frontend project based on the following requirements document. Requirements: complete pages, all fields present, working interactions, and build instructions included; do not invent requirements that were not specified.”
2.3 Key Excerpts from the Requirements Document
The requirements document serves as the denominator for completeness and semantic-understanding checks, and should include at least:
- Project type: frontend for a content site.
- Login page: account/password login, carousel module, form validation.
- List page: multi-condition filtering, pagination, and filter–list linkage.
- Form page: complex fields, field interactions, validation logic, submission feedback.
- Supplementary list: filter names and count, list fields, form fields and count, interaction relationships, conditional required fields, and a list of validation rules.
2.4 Runtime Environment
- OS — required: Linux distribution and version, e.g., Ubuntu 22.04.
- Node.js: recommended pinned to 22.
- npm: recommended pinned to 10.
- Browser: Chromium 126 or the actual version number.
- Automation tooling — required: specify the Chromium/Playwright install commands and driver method, e.g., npm init, npm install, npx playwright install chromium.
- Mock data or API contract — required: include at least the filters, list fields, form fields, and interaction/required rules.
- Build and preview commands — required: e.g., npm install, npm run build, npm run preview, or npm run dev.
2.5 Model-Side Constraints
The following parameters must be kept consistent; otherwise generation time and stability are not comparable:
- Temperature: recommended set to 0.
- max_tokens: must be recorded.
- Timeout setting: must be recorded.
- Web access enabled: must be recorded.
- Tool calling allowed: must be recorded.
- Failure retry count: must be recorded.
2.6 Generated Output Notes
The following must be recorded:
- Whether a complete project was generated.
- Whether it includes package.json.
- Whether dependencies can be installed.
- Whether the preview can be started.
- Whether routes were generated.
- Whether build commands are included.
- Whether page screenshots or an accessible URL are provided.
2.7 Control Groups
It is recommended to set up a baseline group and an experimental group:
- Baseline group: use a spoken description such as “build login, list, and form pages for a content site.”
- Experimental group: use a structured requirements document.
- The baseline and experimental groups must use the same candidate model and version, differing only in input method.
- The goal of the comparison is not vendor ranking, but to test whether structured input improves semantic understanding.
2.8 Example Reviewable Record Table
| Field | Example entry |
|---|---|
| Evaluation target and version | Candidate model X, model name / version / retrieval date |
| Test date | 2025-XX-XX |
| Original prompt | Generate a directly runnable frontend project based on the following requirements document |
| Requirements document | Field and rule list for login, list, and form pages |
| Runtime environment | Linux Ubuntu 22.04, Node.js 22, npm 10, Chromium 126 |
| Automation install command | npx playwright install chromium |
| Mock data | Filters, list fields, form fields, interaction rules |
| Build command | npm install, npm run build |
| Preview command | npm run preview |
| Model-side constraints | temperature=0, max_tokens, timeout, retry count |
| Output notes | Whether installable, startable, and accessible |
3. Scoring Template: Replace Subjective Descriptions with Checklists
An evaluation should not just say “performed well” or “close to production-ready,” but should be broken down into verifiable fields. The following weights are recommended:
| Evaluation dimension | Suggested weight | Verifiable fields | Scoring method |
|---|---|---|---|
| Completeness | 30% | Page structure, buttons, forms, fields, and modules present | Check off item by item against requirements |
| Semantic understanding | 35% | Filter linkage, field interactions, conditional required fields, validation rules, submission feedback | Record hits and misses |
| Engineering usability | 20% | Build, dependency install, startup, preview, routes | Whether it runs and can be deployed |
| Experience metrics | 15% | Generation time, debugging cost, number of rework rounds | Record time and modification cost |
3.1 Scoring Formula
Total score = 100 × (0.30 × T_A/A + 0.35 × T_B/B + 0.20 × T_C/C) + 15 × E/10
Where:
- T_A: completeness checklist score; A: total possible completeness items.
- T_B: semantic understanding checklist score; B: total possible semantic understanding items.
- T_C: engineering usability checklist score; C: total possible engineering usability items.
- E: experience score, ranging from 0 to 10.
- Each checklist item is scored on three levels — 0, 0.5, or 1: 1 means fully satisfied, 0.5 means partially satisfied, 0 means not satisfied.
- Before evaluation, A, B, and C must all be greater than 0; if a dimension has no configured checklist items, the total score formula must not be used.
3.2 Checklist Attribution Mapping
To avoid inconsistent classification, all items must be attributed in advance:
- Completeness (A): three common items count toward completeness — whether a preview URL is provided, whether the page opens, and whether the main page structure is visible. Structural items such as account input, password input, submit button, carousel module, filter count and names, filter button, pagination control, and field count and definitions also count as completeness.
- Semantic understanding (B): filter–list linkage, pagination reset after filtering, field interactions, conditional required fields, validation rules, submit/save feedback, error display, and empty-value or format-error prompts count as semantic understanding.
- Engineering usability (C): build command, dependency install, startup and preview, and usable routes count as engineering usability.
- Generation time counts only toward the experience score (E) and must not be double-counted under completeness, semantic understanding, or engineering usability.
3.3 Common-Item Checklist
Common items must be recorded in every scenario. The preview URL, page accessibility, and visible main structure count toward completeness; build, dependencies, and startup/preview count toward engineering usability; generation time counts toward the experience score.
| Check item | Attributed dimension | Requirement | Recording method |
|---|---|---|---|
| Preview URL | Completeness | Whether an accessible URL is provided | Accessible / not accessible |
| Page accessibility | Completeness | Whether the page opens | Yes / No |
| Main structure | Completeness | Whether the main page structure is visible | Yes / No |
| Build command | Engineering usability | Whether provided and executable | Success / Fail |
| Dependency install | Engineering usability | Whether installation completes | Success / Fail |
| Startup, preview, and routing | Engineering usability | Whether it starts and can be reached via routes | Yes / No |
| Generation time | Experience metric | From submission to a previewable artifact | Minutes and seconds, with start/end noted |
3.4 Login Page Checklist
| Check item | Attributed dimension | Requirement | Recording method |
|---|---|---|---|
| Account input | Completeness | Whether it exists | Present / Absent |
| Password input | Completeness | Whether it exists | Present / Absent |
| Submit button | Completeness | Whether it exists | Present / Absent |
| Carousel module | Completeness | Whether the module is generated | Present / Absent |
| Error prompt | Semantic understanding | Whether it prompts on empty or format errors | Present / Absent |
| Form validation | Semantic understanding | Whether non-empty validation is covered | Passed / Missing items |
3.5 List Page Checklist
| Check item | Attributed dimension | Requirement | Recording method |
|---|---|---|---|
| Filter count and names | Completeness | Whether consistent with requirements | Hit / Missing |
| Filter button | Completeness | Whether it exists | Present / Absent |
| Pagination control | Completeness | Whether it exists | Present / Absent |
| Filter–list linkage | Semantic understanding | Whether the list changes after clicking a filter | Yes / No |
| Pagination reset after filtering | Semantic understanding | Whether it returns to the first page | Yes / No |
| Empty state | Semantic understanding | Whether a prompt appears when there is no data | Present / Absent |
3.6 Form Page Checklist
| Check item | Attributed dimension | Requirement | Recording method |
|---|---|---|---|
| Field count and definitions | Completeness | Whether it satisfies requirements | Hit / Missing |
| Field interactions | Semantic understanding | Whether one field controls another’s display or hiding | Hit / Missing items |
| Conditional required fields | Semantic understanding | Whether the required state toggles by condition | Hit / Missing items |
| Validation rules | Semantic understanding | Whether format, length, and non-empty checks are covered | Passed / Missing items |
| Submit/save feedback | Semantic understanding | Whether a prompt appears on success or failure | Present / Absent |
| Error display | Semantic understanding | Whether it points to the specific problematic field | Present / Absent |
3.7 Notation Legend and Experience-Score Basis
Notation legend: each item is scored on three levels — 0/0.5/1. 1 means fully satisfied, 0.5 means partially satisfied, 0 means not satisfied. Example: if 1 linkage is required and 0 are actually configured, that semantic item scores 0; if a scenario has 9 items totaling 6.5, that is 72%.
Generation-time basis for experience score E: timing starts when the requirements document is submitted and stops when the artifact is started and its preview URL is accessible. Using the median time of the baseline group’s output under the same configuration as the reference:
- Time no more than 1.5× the baseline: E gets 10 points.
- Time more than 1.5× but no more than 2× the baseline: E gets 6 points.
- Time more than 2× the baseline: E gets 2 points.
- Artifact does not run: E gets 0 points.
- Each additional rework round deducts 3 points, down to a floor of 0.
- If there is no baseline group, E is recorded as 0; if a qualitative review is needed, it can be recorded manually, but it must not replace the formula-based score.
4. Layered Results: Structural Correctness Is Not Semantic Correctness
The following are placeholder examples that do not represent any model and do not represent real measured results. Before the required fields are filled in, such records should not be used as selection conclusions.
| Scenario | Completeness | Missing items | Semantic example | Generation time | Accessibility | Conclusion |
|---|---|---|---|---|---|---|
| Login page | Not-yet-tested placeholder | To be added | To be measured: whether non-empty account/password validation hits | To be added | To be added | Placeholder example, not representative of any model |
| List page | Not-yet-tested placeholder | To be added | To be measured: whether filters and list are linked | To be added | To be added | Placeholder example, not representative of any model |
| Form page | Not-yet-tested placeholder | To be added | To be measured: whether field interactions, conditional required fields, and error display hit | To be added | To be added | Placeholder example, not representative of any model |
From a layered-evaluation perspective, login pages are usually relatively standardized and easier to generate in terms of page structure; but even when the structure exists, you must verify whether semantic items such as empty-value prompts and format-error prompts are actually implemented. The key to a list page is not the skeleton but whether the filters genuinely take effect, whether filter conditions can drive the result set, and whether pagination resets after filtering. Form pages further amplify semantic differences: whether fields are controlled, whether conditions are required, whether validation is complete, and whether there is feedback after submission all reflect understanding depth.
Therefore, the gap between models should not be judged only by generation speed or line count, but by observed ability to reconstruct the implicit constraints in the requirements document. Which model better understands “the business logic that was not explicitly written out” must be proven with real data, not inferred from examples.
5. Common Misjudgments and Q&A: Details Most Often Overlooked
Misjudgment 1: “It runs” equals “it’s ready to ship”
Many tests only ask “does the page open,” but going live also requires the build, dependencies, routing, interactions, and validation to hold up. A page that runs only proves that a basic structure exists; it does not mean the business semantics are complete.
Misjudgment 2: More lines of code equals deeper understanding
More code does not necessarily mean deeper understanding. The truly effective metric is the requirements hit rate: whether the filters are complete, whether fields interact, whether required rules are correct, and whether error prompts are clear.
Misjudgment 3: A good demo equals engineering usability
Demos usually have clear boundaries and simple dependencies. Real projects have hidden constraints and repetitive scenarios, so the evaluation must cover multiple pages of increasing difficulty, not just a single showcase.
How do you evaluate LLM code generation capability?
It is recommended to use a structured requirements document as input, covering typical scenarios such as login, list, and form pages, and to record completeness, semantic hits, generation time, and engineering usability. It is best to set up a baseline group and an experimental group — for example, spoken-description input versus structured requirements-document input — to observe how the prompting method affects the result.
How do you judge whether a model understands requirements?
Focus on whether implicit requirements are implemented: whether list-page filters genuinely link to results, whether form-page fields show or hide based on conditions, whether required rules change with fields, and whether, after a failed submission, the specific error field can be located. These reflect understanding depth more than page aesthetics.
How do you choose an LLM code generation model?
First, list the page types common in real business — for example, login, list, form, and content-configuration pages. Second, prepare a structured requirements document and minimal mock data for each page type, specifying filters, fields, interactions, required fields, and validation rules. Third, run the test with the same model, same environment, and same prompt, then decide whether a model enters the candidate pool based on total score, experience score, and engineering boundaries.
What should you watch for in code generation evaluations?
Follow this checklist: pin the model version and temperature; unify the Linux distribution, Node.js, npm, and Chromium or Playwright environment; provide initial mock data or an API contract; record the start/end points of generation time; check build commands, preview URLs, and dependency installation; distinguish “the page opens” from “the business semantics are complete”; and record the failure retry count and number of rework rounds.
6. Engineering Perspective and Applicability Boundaries
This evaluation method suits frontend page generation, especially common business pages such as forms, lists, login, and content-configuration pages. It does not apply to backend API generation, complex state management, multi-person collaborative projects, high-concurrency services, or deep architectural design. To extend the evaluation, you should add dimensions such as API contracts, database models, access control, and test coverage.
In real engineering, a model’s generation capability usually needs to work alongside general infrastructure — object storage, message queues, container platforms, logging services, monitoring, and alerting — to move from “can generate” to “can ship, can operate, and is observable.” Developers can integrate, extend, and validate within their own environments, but open access does not mean exemption from testing; complex semantic understanding still needs to be validated continuously through real evaluations.
Conclusion
Ultimately, evaluating LLM code generation capability answers one question: can it understand and reconstruct real requirements. Login pages rely more on stable basic structure, list pages rely more on semantic reconstruction such as filtering and linkage, and form pages rely more on the completeness of field relationships, conditional required fields, and validation feedback. For those making selection decisions, a more reliable approach is not to watch a single demo, but to use real requirements documents, reproducible configurations, quantitative checklists, and clear boundaries to determine whether a model genuinely fits the business scenario.
