Evaluating LLMs on Complete Project Generation: Benchmark Design and Engineering Practices
This article presents a hands-on evaluation of how large language models perform when asked to generate complete, runnable projects end to end rather than isolated code snippets. It explains how to separate demo quality from engineering usability, proposes a seven-dimension scoring framework for reproducible and comparable results, and documents three recurring failure patterns: missing features under a correct skeleton, reproduction decay on complex forms, and style that is usable but not polished. It also covers how to run credible comparison experiments, four engineering practices for turning model output into stable deliverables (structured requirements, contract-first generation, tabular validation rules, and closed-loop local regeneration), and the trade-offs between open-weight and closed-source models. Readers get a practical checklist for building their own domain evaluation set and running repeatable benchmarks.
1. The Real Starting Point: Separating Demos from Engineering Usability
Generative models are easy to misjudge in coding tasks, because two very different kinds of material tend to get mixed together: a carefully selected single-function snippet, and a complete, runnable, deliverable project. The former tests local syntax and idioms; the latter tests long-range requirement retention—whether the model can, across a handful of pages, dozens of interaction points, and hundreds of field constraints, reliably carry every requirement written in the document into code.
A typical hands-on workflow looks like this: take a real project’s frontend requirements document, impose no language or framework constraints, and have the model generate the whole thing end to end. The deliverable is a project URL you can open and keep iterating on, not a code block pasted into a chat box. The whole process takes on the order of tens of minutes, and along the way you may hit concurrent queuing and quota prompts—operational factors that directly change how “speed” compares, and that must be separated from model capability itself during evaluation.
The conclusion is more nuanced than expected: the page skeleton and visual structure are largely trustworthy, while functional and interaction fidelity shows a systematic decay.
2. Designing Evaluation Dimensions for Reproducible, Comparable Results
To avoid vague judgments like “feels pretty good,” first break subjective impressions into scorable dimensions. I recommend building a scoring sheet around the following seven items, each scored 0–5, while keeping the deliverable links and generation logs so results can be reviewed.
| Dimension | What to check |
|---|---|
| Requirement coverage | Whether every explicit requirement in the document has a corresponding implementation |
| Structural correctness | Whether page division, component hierarchy, routing, and state ownership are reasonable |
| Interaction completeness | Whether filtering, pagination, reset, empty states, loading states, and error states are all present |
| Validation logic | Field required flags, format, boundaries, dependencies, and error message copy |
| Style quality | Whether it follows a mature design system, and whether spacing, hierarchy, and responsiveness are consistent |
| Runnability and maintainability | Dependency locking, directory organization, naming consistency, cost of follow-up edits |
| Generation efficiency | End-to-end time, number of failed retries, quota impact |
At the same time, fix four variables: the same requirements document, the same output target (a project rather than snippets), the same runtime environment, and multiple independent samplings. Multiple samplings are especially important—a large share of whether a single generation looks good or bad comes from randomness, and only the mean and variance across three or more samplings carry statistical meaning.
3. Three Failure Patterns Revealed in Testing
3.1 Correct Skeleton, Missing Features
Login and carousel-type pages usually have the highest completion rate: their requirements are intuitive, they have few states, and they closely resemble common examples, so models easily hit high-frequency patterns in their training distribution. The output tends to be at a “tweak it a little and it works” level.
Problems surface on the product list page: the document explicitly specified multi-condition filtering and linkage rules, yet the generated result simply omits the filters. This is not occasional—it is a pattern that appears across multiple list pages. The page structure and visual framework are both there; what is missing is functional intent.
3.2 Reproduction Decay on Complex Forms
Form pages place higher demands on capability, because they stack three kinds of constraints together: the field set, relationships between fields (dependent show/hide, cascading selection, amount caps that vary by category), and the timing of validation and error presentation. In testing, forms can run, but field and validation details diverge noticeably from the document.
The decay usually traces to three causes:
- Instruction dilution: when the requirements document is long, constraints in the middle and end sections lose weight during generation;
- Implicit assumptions: statements like “validate according to business rules” leave the model to guess the default rules;
- Local-optimum preference: the model tends to first produce a version with correct structure and complete visuals, leaving boundary logic blank.
3.3 Style Positioning: Engineering Usability Over Product Polish
The overall visual style is “a bit better than a rough prototype,” but still falls short of a finished product from a mature component system. This matches a general trait of current models: they are good at generating interfaces with clear structure and clean semantics, but weak at making design-intent judgments—trade-offs around information hierarchy, whitespace rhythm, and responsive breakpoints need explicit design tokens to converge.
4. Making Comparison Experiments Convincing
Handing the same document to a more capable model for comparison is the most effective way to judge which layer the gap is in. The general pattern observed is: the gap is small on simple pages and grows the further you move toward interaction- and business-rule-dense areas—the difference lies not in syntactic correctness but in depth of understanding.
For the comparison to be credible, control a few things:
- Use the same task, same document, same environment, and same acceptance checklist, and avoid comparing a hand-crafted prompt against someone else’s default input;
- Record first-pass rate and manual rework effort, not just whether it runs;
- Annotate speed metrics with the quota tier and concurrency conditions—timing in a free or queued environment cannot serve as evidence of capability;
- Output an accessible deliverable and a code diff so third parties can verify.
5. Turning Model Capability into Stable Output: Four Engineering Practices
Since the failure points cluster around the “requirements → code” information handoff, the fix is not rewording the prompt but writing requirements as a contract that a machine can check.
5.1 Structure the Requirements Document
## Page: Product List- Filter area: keyword, price range (dual input), status dropdown (all/on-sale/off-sale/sold-out)- Interaction: any filter change → debounced query after 300ms; reset clears filters and returns to page 1- List columns: name / price / status / updated-at / actions- Pagination: 20 items per page, show page numbers and total; when the current page is out of range, fall back to the last page- States: no data → empty state; request failed → error state + retry entry- Acceptance: filter conditions must map to request parameters, see the contract file
5.2 Contract First, Then Generate Implementation
First have the model output type contracts for parameters and responses, confirm they are correct, then generate the page code. This turns whether a filter “exists” from a subjective judgment into a verifiable fact.
{"filters": { "keyword": "string", "priceMin": "number?", "priceMax": "number?", "status": "all|on|off|soldout" },"page": { "index": 1, "size": 20 },"response": { "items": "array", "total": "number" }}
5.3 Make Validation Rules Explicit and Tabular
Write each validation as four columns: field / rule / trigger timing / message copy. Models reproduce tables noticeably better than they reproduce prose.
5.4 Closed-Loop Validation with Local Regeneration
After generation, automatically run type checks, static analysis, and key unit tests, feed the failures back together with the original requirements sections, and regenerate only the failing modules. This costs an order of magnitude less than regenerating the whole package, and it avoids fixing one thing while breaking three others.
6. Why Open-Weight Models Matter
Open-weight code models bring four practical benefits worth weighing alongside closed-source options:
- Reproducible evaluation baselines: with weights available, others can reproduce results on the same inputs, and only then can the industry have truly comparable cross-evaluation instead of self-promoted leaderboard numbers;
- Controllable data and compliance: keeping code inside the intranet is a hard requirement for finance, government, healthcare, and similar domains;
- Tunable domain capability: internal frameworks, private component libraries, and historical defect patterns can feed into training or retrieval augmentation, compensating for general models’ inherent weakness in understanding business rules;
- Predictable cost structure: you plan inference resources yourself, avoiding the delivery uncertainty that quotas and queuing bring.
It is fair to point out that between open-weight solutions and the strongest closed-source models there remains a measurable gap in depth of understanding complex business semantics; self-hosting also adds the extra costs of inference resources, quantization accuracy loss, operations, and security hardening. The sensible path is to layer by task: put high-tolerance steps like retrieval, completion, boilerplate code, and unit-test generation on self-hosted models, leave long-chain tasks like cross-module refactoring and complex rule reproduction to more capable solutions, and use contracts and automated checks to hold the quality floor.
7. Implementation Checklist
- Build an evaluation set of requirements documents and acceptance checklists for your own business domain, with at least 10 real tasks, including 2 complex forms and 2 multi-condition lists;
- Sample at least 3 times per run, recording coverage, first-pass rate, and manual rework hours;
- Force every generation through three gates: type checks, static analysis, and unit tests;
- Turn reusable requirements sections into templates to reduce the reproduction decay caused by document ambiguity;
- Choose models by layer, and quantify cost, latency, and data compliance;
- Re-run the same evaluation set with new versions every two quarters, tracking the capability curve rather than single-point impressions.
The conclusion is clear: these models can already handle “from zero to a runnable skeleton” work, compressing delivery cycles to the level of hours; but on functional completeness, complex form rules, and interaction details, manual completion is still needed. Whether a model fits your project is decided not by marketing claims but by a fixed, repeatable evaluation workflow—and as models move toward open ecosystems, that is the foundational engineering most worth investing in.
