LLM Code Generation Evaluation: From Requirements Documents to Runnable Projects

    Leaderboard scores rarely predict how well a large language model handles a real, multi-page software project. Drawing on a hands-on evaluation against a complete frontend requirement, this article shows where models succeed (page skeletons) and where they break down (cross-condition filtering, field dependencies, and validation). It lays out a reusable framework: how to design a discriminating requirements document, a layered and weighted scoring rubric that leaves subjective judgment to humans and hands objective checks to an automated gate, and six practical engineering strategies for getting reliable output. It also examines why open model weights matter for the industry. The recurring conclusion is that generation quality depends less on a model's raw coding ability and more on how structured the input specification is and how automated the acceptance pipeline is.

    1. The Starting Point: From Leaderboard Narratives Back to Engineering Usability

    Over the past two years, public discussion of model capability has relied heavily on benchmark narratives—how many percentage points one model beats another on a given benchmark—turning words like “on par with,” “caught up to,” and “drop-in replacement” into constant talking points. But when developers actually hand a requirements document to a model and ask it to produce a runnable project, they run into a clear explanatory gap between leaderboard scores and the engineering experience.

    The reason is not mysterious. Benchmarks are mostly short-range, closed-ended, single-point tasks: one algorithm problem, one function completion, one unit-test generation. Delivering a real project is a long-range, open-ended, multi-constraint task: a requirements document carries implicit priorities, pages share state, form fields carry dependencies and cross-field validation, and the styling system must follow an established design language. The former tests “single-step accuracy”; the latter tests “long-chain consistency.”

    So when evaluating whether a model has project-level code generation ability, the right question is not “where does it rank,” but rather:

    • Can it reconstruct the information architecture and data model from a natural-language spec?
    • Can it maintain a consistent design and state semantics across multiple pages?
    • Can it correctly handle boundary conditions and validation rules?
    • Can its output be accepted by an automated acceptance pipeline, rather than only “looking right”?

    This article walks through the full generation process for a real frontend requirement, and offers a reusable evaluation framework and improvement strategy.

    2. Designing the Evaluation Task: Building a Discriminating Requirements Document

    To make an evaluation discriminating, the requirements document cannot be a single sentence; it must deliberately cover several tiers of difficulty. A usable sample structure looks like this:

    1. project: Internal data management platform (frontend)
    2. stack_constraint: Chosen by the model; selection rationale must be stated
    3. pages:
    4. - name: Login page
    5. requirements:
    6. - Multi-image carousel with autoplay, manual switching, and indicators
    7. - Form with username, password, and CAPTCHA
    8. - Frontend validation for required fields and format, with inline error messages
    9. - Persist login state so it survives a page refresh
    10. - name: Product list page
    11. requirements:
    12. - Three-way filtering by category, price range, and listing status
    13. - List refreshes immediately when filters change; filters can be cleared
    14. - Pagination + sorting, where sort state and filters take effect together
    15. - Cover empty, loading, and error states
    16. - name: Product form page
    17. requirements:
    18. - Field linkage: category changes affect the available spec options
    19. - Validation includes required, length, format, and cross-field consistency
    20. - Validate everything on submit and focus the first erroneous field
    21. - Support draft saving and a confirmation step when discarding edits
    22. design:
    23. tone: Minimal and engineering-focused, prioritizing information density
    24. quality_gate:
    25. - Zero type-check errors
    26. - Critical interactions covered by automated tests

    The design intent is this: the login page tests “completeness,” the list page tests “understanding of cross-condition combinations,” the form page tests “field dependencies and validation semantics,” and the quality gate tests “machine verifiability.” These four tiers rise in difficulty in sequence, matching the failure patterns commonly seen in practice.

    3. Observed Results: Three Layers of Capability, Three Gaps

    3.1 Layer One: The Interface Skeleton—Easiest, and Easiest to Misjudge

    On the login page, results tend to look the best. Carousel, form, CAPTCHA placeholder, state persistence—these requirements are semantically simple with convergent implementation paths, so the model can usually get them right in one pass, producing something close to “tweak it a little and ship it.”

    This is also the most dangerous signal. A developer glances at a good-looking login page, forms overly high expectations for the pages that follow, and only discovers the problems midway through the project. Evaluation conclusions must be based on the hardest page, not the best-looking one.

    3.2 Layer Two: Cross-Condition Combinations—The Real Dividing Line

    The list page’s problem is usually not “the page wasn’t written” but “the page was written without implementing the constraints.” Typical symptoms include:

    • Filter controls are missing, or only exist as UI with no state binding;
    • Filtering, sorting, and pagination are not combined into a single unified query state, so changing any one condition drops the others;
    • Empty and error states are missing, making interface behavior unpredictable on exceptional paths.

    The common thread in these failures is that the conditions in the requirements document are parallel and mutually coupled, while the model tends to organize the implementation around “page sections” rather than a “state machine.” It treats “filtering” as a visual component rather than a set of variables that participate in query-state computation.

    3.3 Layer Three: Field Dependencies and Validation—Where Engineering Judgment Is Most Needed

    The form page is a dividing line above the dividing line. Cross-field linkage, conditional required fields, uniqueness checks, error focus, and summary prompts all require the model to maintain an implicit field-dependency graph. The typical degradation observed in practice: the fields are mostly present and the page runs, but the linkage relationships are reduced to static display, and validation degrades to non-empty checks.

    The engineering response here is not to retry the same prompt over and over, but to make the implicit dependencies explicit: put the field-dependency table and validation matrix directly into the requirements document.

    Field Trigger condition Validation rule Linked effect
    Category Always required Must belong to a given enumeration Resets spec options and clears their selected values
    Spec option Required when category is selected Must belong to that category’s available set Affects price-range boundary validation
    End time Required when start time is filled Must be later than start time Triggers a cross-field consistency warning

    Writing this table into the spec noticeably improves the model’s performance. The general lesson is this: the model does not lack coding ability; it lacks structured input for the requirements.

    3.4 On Generation Time and Product Form

    In practice, generating a complete project took anywhere from several minutes to several tens of minutes, with the magnitude varying by an order of magnitude. This difference is often not about raw compute, but about service-form differences: free tiers face queuing and rate limiting, and long-context reasoning is inherently slower. So when evaluating speed metrics, you must record the “quota tier + concurrency time window,” or the numbers are not comparable. This point is frequently overlooked in head-to-head model selection.

    4. Evaluation Method: Turning Subjective Impressions into Reproducible Scores

    I recommend a “layered weighting + machine-first” scoring sheet:

    Dimension Weight Assessment method
    Feature completeness 30% Check each requirement off one by one; deductions for anything missing
    Business-logic correctness (linkage/validation) 25% Manually constructed boundary cases
    Structural maintainability (component breakdown, state ownership) 15% Human review + a rough cyclomatic-complexity measure
    Styling and design consistency 15% Comparison against the established design language
    Verifiability (type checks, test coverage) 10% Output directly from the automated pipeline
    Generation time and stability 5% Record the variance across multiple runs

    The key design is to hand the last two items to the machine. Subjective dimensions must be left to people, but judgments like “does the type check pass” and “do the critical interactions have automated tests” should not be made by people. The following acceptance-gate pseudocode is feasible:

    1. # Generate-and-verify: block on any failure to avoid "looks like it runs"
    2. step "Type check" run type-check
    3. step "Static check" run lint --max-warnings=0
    4. step "Build artifacts" run build
    5. step "Critical-path tests" run test --filter=critical-path
    6. step "Interaction regression" run e2e --spec=list-filter,form-validation
    7. report summary.md # Emit a structured score for side-by-side comparison

    The point of this pipeline is that it converts “how well the model wrote it” from a subjective argument into a continuously regressable quantitative curve. When you swap models or upgrade a version, the same gate produces comparable conclusions within the same hour.

    5. Engineering Recommendations: Treat the Model as a Junior Engineer with Clear Delivery Standards

    Bringing together the practical findings, here are six strategies that can be applied directly:

    1. Spec first. Write states, write the validation matrix, and write the exceptional paths—not just visual descriptions.
    2. Generate in layers, not all at once. First generate the data model and routing skeleton, confirm it is correct, then fill in pages one by one, and finally converge on styling. Generating a long-chain task in a single pass inevitably degrades consistency.
    3. Make the context explicit. Write the already-decided components, naming conventions, and state-management approach into the prompt as constraints to reduce the room for the model to improvise.
    4. Constraints over trust. Anything the type system and static checks can catch should not rely on manual review.
    5. Keep a human backstop for critical logic. High-stakes areas like permission checks, amount calculations, and cross-field validation must be reviewed by a person and backed by tests.
    6. Leave a trail for regression. Store each generation result along with its score sheet, building your team’s own evaluation set and gradually replacing reliance on external leaderboards.

    6. A Deeper Look: What Open Model Weights Mean for AI Development

    Back to the original question—what does it mean for the industry that open-weight models are approaching frontier closed-source models in this kind of capability? I think the value concentrates in four points, none of which is “performance parity” itself.

    First, evaluation turns from belief into evidence. Open weights mean research institutions and engineering teams can reproduce, stress-test, and build domain-specific evaluation sets on their own. Technical judgment no longer depends on a single source’s promotional narrative; the community can argue using the same data. For a fast-evolving technical field, this is necessary infrastructure.

    Second, data compliance and private deployment become possible. In industries with hard requirements that data must not leave its domain, closed-source, API-style services often fall short. Available weights let companies deploy inference in their own environment, pair it with container platforms for elastic scaling, use object storage to hold models and data, and use logging and monitoring services for call-chain observability—securing both capability and compliance at once. For many projects, this is a precondition for moving from pilot to production.

    Third, control over domain adaptation returns to the user. Vertical-industry data distributions differ enormously from general-purpose corpora. With trainable weights, you can do incremental pre-training, instruction fine-tuning, and preference alignment, turning general capability into capability that understands business terminology and internal standards. This cannot be done in an API-only model.

    Fourth, the cost structure changes. Available weights drive the rapid spread of inference-side optimization techniques—quantization, distillation, operator optimization, and batch scheduling—and the results of this work can be reused directly by any user. In the long run, the cost curve of a unit of intelligence is flattened by the whole community rather than priced by a few service providers.

    It is also worth seeing the other side objectively: open-weight models still lag in long-chain consistency and complex instruction following, and that gap is exactly the root cause of the list-page and form-page failures observed in this evaluation. So the correct judgment is that “the capability boundary has been clearly measured,” not that “alignment is already complete.” Precisely because it is measurable, there is a clear target for catching up.

    7. Conclusion

    Between a requirements document and a runnable project lies not the model’s IQ but how structured the requirements are and how automated the acceptance is. The experiments repeatedly point to the same conclusion: when the input is prose-style description, the model’s implementation is prose-style; when the input is explicit states, constraints, and a validation matrix, the implementation quality rises a notch immediately.

    For enterprises and individual developers alike, what is truly worth investing in is not chasing the next version number but three things that will not depreciate: a specification-writing standard of high quality, an automated pipeline that verifies on generation, and a private evaluation set that belongs to your own team. These three determine that, no matter how the model iterates, the floor of your delivery quality stays under control.