Evaluating Whole-Project Generation in Large Language Models and the Value of Open Source

    This article examines how to evaluate large language models (LLMs) on whole-project generation rather than single-function completion. Its core argument is that as generation tasks move from code snippets to full, runnable projects, the real bottleneck shifts from coding ability to spec-driven understanding. The article presents a reproducible evaluation method built on fixed inputs, unified runtime conditions, multi-dimensional scoring, and repeated sampling; reports field observations across login, list, and form pages; analyzes why comprehension degrades as requirements grow more complex; and proposes engineering countermeasures such as structured specs, a contracts-first workflow, and automated validation loops. It closes with what the rise of open-source models means for teams and a six-step checklist for bringing these capabilities into production.

    1. Background: Evaluation Is Shifting from “Writing One Line Right” to “Delivering a Working System”

    Over the past two years, the mainstream way to assess code-oriented LLMs has still been function-level completion, single-file generation, and pass rates on algorithm problems. These evaluations share a common premise: clear task boundaries, short context, and correctness that unit tests can judge directly.

    But when a model is asked to “deliver a runnable front-end project from a requirements document,” the nature of the task changes fundamentally:

    • The input shifts from code snippets to natural-language specifications. The model must first parse the requirements, extracting constraints scattered throughout the document into executable engineering decisions.
    • The output shifts from a single file to a multi-file project. This involves routing, state management, component layering, API mocks, and build configuration—any inconsistency in one place can break the whole.
    • Correctness shifts from “it runs” to “it matches the spec.” A page that renders does not mean the filter options are implemented, the validation rules are in place, or the interaction states are covered.

    So the core question is no longer “can the model write code?” but rather: Is the model’s depth of understanding sufficient to support a complete engineering delivery? This is why this article calls this class of evaluation a “spec-driven generation capability evaluation.”

    2. Method: Turning “Feels Good” into Reproducible Conclusions

    Many hands-on write-ups stop at impressionistic description, and their conclusions cannot be reproduced. To give an evaluation engineering value, four things must be fixed.

    1. Fix the input and the task boundary.

    Give the model a realistic business requirements document covering three typical page types—login, list, and form—and explicitly state the fields, filter conditions, validation rules, and interaction logic. The key principle is not to pre-specify a tech stack, letting the model choose its own implementation path—because technology selection is itself part of the capability.

    2. Unify the runtime environment.

    Use the same network environment, the same time window, and the same account type to avoid letting infrastructure differences contaminate the results. Latency for generation tasks is highly sensitive to queuing policy and concurrency quotas, and this must be stated explicitly in the conclusions.

    3. Score across multiple dimensions.

    A single “does it look right” judgment has no diagnostic value. It is better to break the assessment into at least the following dimensions:

    Dimension Focus How to Judge
    Requirement coverage Whether each constraint in the doc has a corresponding implementation Check requirements item by item
    Structural soundness Whether layering, component splitting, and state ownership are clear Human review
    Runnability Whether it builds and starts successfully on the first try Automated build
    Interaction fidelity Whether linkages, validation, and edge states are implemented Manual walkthrough + test cases
    Visual consistency Whether it follows a unified design language Human review
    End-to-end latency Time from submission to a usable artifact Timing
    First-pass success rate Proportion of first attempts that meet the usable standard Sampling statistics across runs
    Manual correction cost Amount of change needed to reach a production standard Effort estimation

    4. Sample multiple times rather than conclude from a single run.

    The generation process is stochastic, so a single result can serve only as an observation, not a conclusion. It is advisable to sample the same task three to five times and tally the first-pass success rate and the distribution of failure modes.

    3. Field Observations: Clear Capability Tiers, “Strong on Single Pages, Weak Across Pages”

    Testing a new generation of open-source large models with the method above revealed clearly layered behavior.

    Usability and latency: During peak hours, requests queue up, and it may take several retries or a higher quota to get stable service. Generating a complete project took anywhere from several minutes to tens of minutes, with the magnitude driven mainly by three factors—queuing and quota policy, the batching and scheduling configuration of the inference service, and the output length of a single generation. When comparing horizontally, then, it is essential to distinguish “latency from the model’s capability” from “latency from service policy,” or the conclusion will be distorted.

    Login page: high completion. Carousels, form inputs, basic validation, and responsive layout were all implemented, with no obvious functional gaps—at the “tweak it slightly and it works” level. For single-page, single-block tasks, the model’s performance is quite stable.

    List page: comprehension begins to decay. The page skeleton, list rendering, and empty state were all correct, but the filter conditions and filter-linkage logic explicitly stated in the document were omitted entirely. This is not an occasional miss but a systematic defect on this class of pages—the model tends to implement a structure that “looks complete” while skipping the parts that require cross-component state coordination.

    Form page: field and validation fidelity is insufficient. The page runs, but field types, required rules, conditional display, error-message copy, and post-submit state transitions all deviate noticeably from the document. The more the document contains business rules rather than visual rules, the larger the deviation.

    Visual style: engineering-usable, but lacking a product feel. Compared with the bare prototypes produced by earlier models, this is clear progress, but there is still a gap relative to directly adopting a mature component library or an atomic styling scheme. A fair summary: engineering usability > product completeness, with a default style that leans toward “engineered plainness.”

    An easily underestimated strength: the results provide a directly accessible preview URL, not just code snippets. This substantially lowers the cost of verification—a reviewer can click through the pages like a real user rather than inferring behavior by reading code. This has high practical value.

    4. Root-Cause Analysis: Why Comprehension “Slows Down” as Documents Get More Complex

    Several explainable technical reasons lie behind the observations above, and understanding them is the key to designing effective countermeasures.

    1. Attention dilution in long context. A requirements document easily runs to thousands of tokens, with constraints scattered across different sections. When a key constraint (for example, “the list must support combined filtering by status and price range”) sits in the middle of a long document, its weight is easily diluted by large amounts of descriptive text. This is consistent with the industry-known phenomenon that “information utilization in the middle of a long context drops”: increasing context length does not equal improved constraint adherence.

    2. Implicit constraints are not made explicit. “Implement a filter bar” is a sentence of natural language in the document, but in engineering terms it implies a chain of decisions: where the filter state is stored, whether switching filters resets pagination, whether the query parameters sync to the URL, and how an empty result is presented. A human fills in these default decisions automatically; a model tends to do only the most surface-level thing.

    3. No runtime feedback loop. After the model finishes generating, it is usually not automatically compiled, tested, and walked through before being corrected. That is, by the time a defect is discovered, generation is already over. A human engineer’s efficiency comes precisely from a short “write—run—observe—correct” cycle, and this cycle is entirely absent in one-shot generation.

    4. The local self-consistency trap. Each page looks reasonable on its own, but data structures, field naming, and state definitions are inconsistent across pages. The generation process lacks a “contract” that spans the whole, so consistency is left to chance.

    5. The expressive quality of the requirements document itself. The more a document resembles a “visual description,” the better the model performs; the more it resembles “business rules,” the worse it performs. This shows that the current capability gap is concentrated in business-semantic understanding, not in code-syntax generation.

    Conclusion: the bottleneck has shifted from “syntactic correctness” to “requirement consistency and engineering completeness.” This shift imposes new demands on both evaluation methods and engineering practice.

    5. Engineering Countermeasures: Turning a Model’s One-Shot Capability into a System’s Stable Capability

    Since the gaps lie in requirement understanding and missing feedback, the solution should not be “switch to a stronger model” but rather to close these two loops.

    1. Structure the requirements document: turn implicit constraints into explicit contracts

    Beyond the natural-language document, additionally maintain a declarative spec that a machine can validate:

    1. page: product_list
    2. components:
    3. - filter_bar
    4. fields:
    5. - name: status
    6. type: enum
    7. options: [draft, active, archived]
    8. multi: true
    9. - name: price_range
    10. type: range
    11. unit: CNY
    12. behavior:
    13. on_change: [reset_pagination, sync_url_query, refetch]
    14. - table
    15. columns: [id, name, status, price, updated_at]
    16. states:
    17. empty: {trigger: result_count == 0}
    18. error: {trigger: request_failed}
    19. loading: {trigger: pending}

    The value of this spec is that it can be read by a program and therefore serve as an acceptance criterion, not just something for humans to read.

    2. Contracts first: define data and interfaces before generating the implementation

    Before generating the UI, require the model to first produce type definitions and interface signatures, and confirm them manually. Once the type definitions are fixed, the field conventions of every subsequent page are forcibly constrained, and cross-page inconsistencies drop significantly.

    3. Generation–validation loop: catch defects at the generation stage

    After the model outputs, automatically run four gates: static and type checks, build, unit tests, and end-to-end smoke tests. If any fails, feed the error information back to the model and request a fix, forming a bounded self-repair loop.

    1. while attempt < MAX_ATTEMPTS:
    2. result = model.generate(spec, contract, context)
    3. report = run_checks(result) # lint + typecheck + build + smoke
    4. if report.passed and coverage(result, spec) >= THRESHOLD:
    5. return result
    6. context += report.failures # feed the errors back into the context
    7. return escalate_to_human(context)

    Three design points are critical: the number of attempts must have a cap (to avoid unbounded spending); the feedback must include actionable error information (not just “build failed”); and requirement coverage must be tallied separately (a passing build does not mean the requirements are implemented).

    4. Divide and conquer + context assembly

    Do not make the model swallow the entire document at once. Generate in slices by page or by feature, and in each round assemble only the spec, contracts, and existing-code summaries relevant to that slice. The more focused the context, the more reliable the constraint adherence.

    5. Design-system constraints

    Uniformly inject the component-library choice, design variables (colors, spacing, font sizes), naming conventions, and directory-structure conventions. Visual consistency is not a matter of the model’s aesthetic taste but a matter of constraint injection—write it as rules, and the style will naturally converge.

    6. Observability and regression evaluation

    Record and archive the inputs, outputs, failure points, and manual-correction effort of every generation. Artifacts and traces can be stored in object storage, and pipeline metrics such as latency, failure rate, and correction ratio can be observed through logging services and a monitoring-and-alerting system. When building this pipeline in-house, enterprises can rely on general-purpose cloud capabilities from platforms such as Baidu AI Cloud rather than provisioning their own infrastructure.

    Over the long term, these records will accumulate into an evaluation set that belongs to your own business—and this reflects real-world performance better than any public leaderboard.

    6. What This Means for Open-Source Models and AI

    Back to the original question: what does it mean that open-source models are approaching leading closed-source models on certain capabilities?

    1. Accessibility and data boundaries. Open weights support local or private deployment, so code and business data need not leave your environment. For heavily regulated industries such as finance, government, and healthcare, this is not a bonus but an entry requirement.

    2. Modifiability. With closed models, the capability ceiling is set by the vendor; open models allow continued pre-training, instruction fine-tuning, and preference alignment for vertical domains, as well as distilling smaller models suited to your own hardware. Enterprises can turn “general capability” into “capability for my scenario.”

    3. Cost and scheduling autonomy. Inference scale, batching strategy, caching strategy, and model routing are all yours to define. Tiered routing—simple tasks to small models, complex tasks to large models—often delivers better cost-performance than a single large model.

    4. The real gap is not in single-point benchmark scores. From testing, the gap concentrates in depth of requirement understanding and ecosystem maturity: data quality and cleaning pipelines, post-training and alignment methods, the completeness of tool-calling and agent frameworks, the rigor of evaluation systems, and the level of inference-engineering optimization. None of these convert directly from parameter scale; they are the result of long-term engineering accumulation.

    5. How to read benchmark comparisons correctly. “Approaching model X on some task” is a valuable but limited signal. More worthwhile metrics are the first-pass success rate, the manual-correction hours, and the improvement slope across versions on your own business requirements set. The first two determine adoption cost; the third determines whether the approach is worth a long-term investment.

    6. The significance for the AI landscape. The open-source route turns capability supply from a closed product held by a few vendors into a public foundation that can be broadly audited, modified, and redistributed. The focus of competition therefore shifts from “whose score is higher” to “whose evaluation is more rigorous, whose toolchain is more complete, and whose unit cost is lower.” For developers, this is a net gain—even if they ultimately choose a closed service, the existence of open-source models keeps driving prices down and transparency up.

    7. Implementation Checklist

    For readers preparing to bring this kind of capability into production teams, here are six steps you can execute immediately:

    1. Build an evaluation set: extract 20–50 tasks from real requirements documents of the past six months to form an internal baseline; do not rely on public samples alone.
    2. Define the metrics: start with four—requirement coverage, first-pass success rate, end-to-end latency, and manual-correction hours.
    3. Add a structured spec layer: pull business rules like filtering, validation, and state machines out of prose and turn them into verifiable declarations.
    4. Build an automated validation pipeline: chain static checks, type checks, build, and smoke tests into a loop that feeds failures back automatically.
    5. Do a security review: license compatibility of generated code, third-party dependency vulnerabilities, and hard-coded secrets must all be scanned before check-in.
    6. Clarify the human–machine division of labor: the model handles skeletons, boilerplate, and repetitive implementation; humans handle business-rule adjudication, acceptance, and architectural decisions.

    8. Conclusion

    The most valuable output of this kind of testing is not a verdict on whether “model X is good,” but a structural fact it reveals: when generation tasks move from snippets to whole projects, the bottleneck shifts from coding ability to requirement-understanding ability—and the gap in requirement understanding can only be closed by engineering means. Structured specs, contracts-first, automated validation loops, and focused context assembly—these four things matter more than blindly waiting for the next generation of models.

    For open-source models, the gap with leading closed models is real, but its shape is “depth of understanding and ecosystem maturity,” not an insurmountable generational divide. Whoever makes evaluation more rigorous, the toolchain more complete, and the unit cost lower is closer to being engineering-usable—and this path is precisely where the open-source route excels.