From Requirements Document to Runnable Project: How to Quantify AI Coding Capabilities
This article examines how to quantitatively evaluate AI coding capabilities by measuring how faithfully a model reconstructs a business requirements document into a runnable project. Drawing on tests of login, list, and form pages generated from the same requirements without a specified framework or component library, it shows that restoration fidelity declines as constraint density rises, and traces the failures to attention dilution in long documents, a preference for local completeness, missing self-checks, and unstated assumptions. It offers a reproducible acceptance checklist with a scoring function, a six-dimension selection scorecard, an analysis of open-weight models approaching top-tier capability, and practical rules for phased delivery, contract-first design, feedback loops, and supply-chain governance — arguing that reproducible acceptance, not demos, should drive model selection.
1. The Starting Point: Generating Code Is Not the Same as Delivering a Project
Over the past two years, code generation has evolved from “completing a single line” to “generating a whole file,” and now to “producing an accessible project URL from a short requirements description.” It is tempting to read this progression as a pure jump in model capability, but from an engineering standpoint the real watershed is something else: the proportion of the requirements that gets faithfully reconstructed.
A login page that runs, or a page that renders a list, only proves the model has mastered a common framework pattern. It does not prove it understood the constraints in the requirements document that actually determine success or failure — the interdependencies among filter conditions, the dependencies between fields, when validation is triggered, and how exception branches are handled. These details are nearly invisible in a demo, yet they account for the bulk of real delivery work.
So this article tries to answer three questions:
- Given only a business requirements document — no framework, no component library — what does the model actually produce?
- Do these failure modes follow a pattern, and can they be attributed to specific causes?
- If this kind of capability keeps approaching the top tier, what does it mean for technology selection and AI engineering practices?
2. Evaluation Design: Turning “Something Feels Off” into Decidable Checks
For an evaluation to be reproducible, the input must be structured and the acceptance criteria must be itemized.
2.1 Input: Writing a “Verifiable” Requirements Document
Vague requirements are the main source of failed evaluations. Effective input should contain at least five categories of information:
- Feature list: A page-level breakdown, with each page listing the modules that must exist.
- Field definitions: Field name, type, whether required, default value, valid range.
- Validation rules: When they trigger (on blur / during input / on submit), where error messages appear, whether they block submission.
- Interaction states: Loading, empty, error, success, disabled.
- Visual constraints: Layout grid, spacing rhythm, restraint on color, responsive breakpoints.
The task is deliberately left open: no framework, no component library, no build tool specified. This lets us observe the model’s default technology preferences when unconstrained — preferences that often directly affect a project’s maintainability.
2.2 Acceptance: A Checklist Scanned Item by Item
Acceptance cannot rely on “overall impression”; it must break down into items. A minimal checklist structure looks like this:
{"page": "product-list","checks": [{ "id": "L1", "desc": "Filter area is present", "type": "hard" },{ "id": "L2", "desc": "Switching category refreshes the list in real time", "type": "hard" },{ "id": "L3", "desc": "Filter conditions support reset", "type": "hard" },{ "id": "L4", "desc": "Empty-state text shown when no data", "type": "normal" },{ "id": "L5", "desc": "Loading skeleton placeholder", "type": "normal" },{ "id": "L6", "desc": "Pagination keeps current filter conditions", "type": "normal" }]}
Paired with a minimal scoring function that converts subjective judgment into numbers:
def score(checks, results):total = len(checks)passed = sum(1 for c in checks if results.get(c["id"]))hard = [c for c in checks if c["type"] == "hard"]hard_passed = sum(1 for c in hard if results.get(c["id"]))return {"coverage": round(passed / total, 2),"hard_rate": round(hard_passed / len(hard), 2) if hard else 1.0,# Hard constraints are the release gate; failing any means an automatic fail"shippable": hard_passed == len(hard)}
The key design is distinguishing hard from normal: a missing feature is a hard-constraint failure, while a style detail is an optimization opportunity. Confusing the two is why so many evaluation reports are misleading.
3. Observations: Restoration Fidelity Shows a Clear Page-Level Gradient
Under the same document and the same task setup, three page types showed a clear gradient of performance.
3.1 Login / Auth Pages: High Restoration Fidelity
The carousel, username and password inputs, “remember me,” error messages, and post-submit redirect specified in the requirements were all implemented. The page structure was complete, with no obvious functional gaps. This page was at “modify a bit and it ships” quality.
The reason is that these pages are highly convergent in pattern: almost every project implements them with a similar structure, training data density is high, and there are few implicit assumptions. The model does not need to reason — it just needs to invoke the most typical pattern.
3.2 List Pages: Correct Skeleton, Discounted Functional Understanding
The page structure was right — the filter area, list area, and pagination were all there. But the filter conditions explicitly written in the document were not implemented. This was not an occasional omission; it appeared across multiple list pages, indicating a systematic shortcoming in the model’s capability rather than random variance.
A notable failure signature: the model preserved the visual skeleton but dropped the behavioral semantics. It knew what a list page “looks like,” but did not treat “filter conditions” as a state-management mechanism that must be implemented.
3.3 Form Pages: Field Relationships and Validation Logic Simplified
Form pages ran overall, but the number of fields, validation rules, and field-linkage details deviated from the document. Common patterns: required-field validation was implemented but format validation was missing; field dependencies were replaced with static display; post-submission failure handling was missing.
Forms are the page type with the highest constraint density — each field may carry a set of validation and linkage rules, and these rules interact with one another. This kind of high combinatorial complexity places the highest demands on understanding long documents.
4. Failure Attribution: Why It “Slows Down”
Abstracting the differences among the three page types, we can identify four main causes.
First, attention dilution in long documents. The longer the requirements document, the lower the weight assigned to any single constraint. A field table written clearly at the beginning gets carried through, while a “supports filtering” mentioned in a single sentence in the middle tends to get summarized away.
Second, a preference for local completeness. The model tends to produce a component that “looks complete” rather than one that is “behaviorally complete.” Filling in a filter UI is natural, but wiring up the linkage between filtering and the list data flow requires implicit cross-file conventions, costs more, and is easily skipped.
Third, the absence of an executable self-check step. One-shot generation means delivery happens the moment generation ends, with no opportunity to turn the document into assertions and self-verify. This is a process flaw, not entirely a model flaw.
Fourth, implicit assumptions left unstated. If the requirements document only says “filter conditions” without providing the data structure, the model must fill in the field definitions on its own — and the result will almost certainly differ from what you had in mind. Half the responsibility for this divergence lies in input quality.
5. Visual Style: The Gap Between Engineering-Usable and Product-Attractive
The default visual quality is usually slightly better than an ordinary wireframe, but clearly behind the look provided by mature component libraries and design systems. The gaps concentrate in a few areas:
- Lack of design tokens (hierarchical organization of colors, spacing, border radii, and font sizes); styles are scattered;
- Crude handling of responsive breakpoints, with layout squeezing on narrow screens;
- Missing engineering details such as dark mode, focus states, and accessibility annotations;
- Colors and hierarchy that are too “safe,” lacking the design expression of information priority.
The positioning judgment is accurate: engineering-usable over product-attractive. The improvement path is also clear — translate “looks good” into constrained input. Providing design token definitions, reference screenshots, and spacing grid specs is far more effective than a single line saying “make the interface beautiful.”
6. Performance and Cost: Don’t Count Engineering Pipeline Time as Model Time
In practice, the waiting feels pronounced. But two segments of time need to be distinguished:
- Model side: requirements understanding, planning, and per-file generation and decoding.
- Engineering side: dependency installation, build, type checking, and preview environment startup.
End-to-end generation of a multi-page project typically takes on the order of tens of minutes, and the engineering side often accounts for a substantial share. So evaluation should separately record “time to the first usable artifact” and “time to final accessibility,” otherwise the slowness of the build pipeline gets misjudged as model slowness.
In addition, peak-hour queuing and concurrency quota limits further amplify the sense of waiting. For production selection, rate-limiting policies and quota recovery cycles should be included as evaluation items, not just capability scores.
7. A Practical Scorecard
Bringing the above together, a selection scorecard for models or AI coding tools can be built from six dimensions:
| Dimension | Metric | Notes |
|---|---|---|
| Requirement coverage | Checklist pass rate | Itemized, split into hard and soft |
| Constraint hit rate | Hard-constraint pass rate | Release gate; a fail means elimination |
| Delivery efficiency | First-pass rate / rework rounds | Rework rounds reflect real labor cost |
| Maintainability | Directory structure, type completeness, code duplication | Determines the cost of future evolution |
| Engineering quality | Tests, static analysis, accessibility | Whether a foundation for sustainable maintenance exists |
| Cost and stability | Time, quota consumption, rate-limit behavior, retry rate | Key constraint for production |
The core principle: record the cost of fixes, not just the generation result. A solution that scores 90 but needs eight rounds of rework to reach 95 costs far more in reality than one that scores 80 but is only missing three details.
8. When Open-Weight Models Approach the Top Tier, What Does It Mean?
As open-weight models keep approaching closed-source flagship capability, the impact goes beyond “local models have caught up.” It brings four practical changes on the engineering side.
The choice of deployment model returns. Open weights mean private deployment is possible, with data staying within the domain — decisive for strongly regulated scenarios like finance, healthcare, and government and enterprise. This is not a capability question but a feasibility question; capability that cannot be deployed has zero value in these scenarios.
Cost shifts from unpredictable to calculable. Pay-as-you-go spending fluctuates with call volume, while self-hosted inference cost is determined mainly by compute and utilization, making capacity planning feasible.
Fine-tuning and distillation become possible. Open weights allow adaptation training for an enterprise’s private code style and internal framework conventions — a path that API-style calls cannot offer. Generated code that fits team conventions significantly reduces the rework rate.
Supplier replaceability improves. The model abstraction layer goes from “nice to have” to “architecturally required.” Decoupling prompt orchestration, tool calling, and result validation from any specific model is the only way to avoid a forced migration when capabilities are upgraded or prices change. Whether through localized deployment or via general-purpose models and AI coding services on platforms such as Baidu AI Cloud, the selection standard should be the same reproducible acceptance process, not the effect of any single demo.
But one common misreading must be clarified: parameters or benchmark rankings approaching each other does not mean the engineering experience approaches each other. Context length, tool-calling stability, long-document adherence, ecosystem maturity, and failure-recovery mechanisms — the divergence among these engineering properties is often greater than the gap in capability scores. And the latter is what determines production usability.
9. Practical Recommendations: Six Rules for Bringing AI into Real Delivery
Deliver in phases, not in one shot. First produce the information architecture and data model, confirm them, then generate page skeletons, and finally fill in interaction details. Verify at each phase, breaking a large task into a correctable sequence.
Write implicit requirements as verifiable items. Field tables, state-transition tables, validation-rule matrices — anything that can be written as a table should not be written as a sentence.
Contracts first. Define interface contracts and types first, and let the model generate the implementation within existing constraints, rather than letting it decide structure and logic at the same time.
Build a feedback loop. Turn the acceptance checklist into automated tests, and auto-retry when self-checks fail. This is more sustainable than manual item-by-item review.
Clarify the human-machine division of labor. Scaffold code, boilerplate pages, unit tests, and type definitions are well suited to the model; domain rules, edge cases, security policies, and performance-sensitive paths should stay with humans.
Practice supply-chain governance. Generated code also needs review: are dependency sources trustworthy, are licenses compliant, and are there hardcoded credentials or common injection risks? The gains from faster generation are easily offset by a single security incident.
10. Conclusion
The conclusion from this practice is not dramatic: on pages with clear patterns, the model is already close to directly deliverable; on constraint-dense pages, requirements fidelity shows systematic decay; and in visual expression, it remains more engineering than design.
The value of this conclusion lies in its actionability — the failure points are clear, and the improvement directions are clear too: more structured input, more phased processes, and more itemized acceptance. Once the capability boundary is clearly described, model selection is no longer a matter of reputation and hype, but a number you can produce by running the same scorecard in your own scenarios.
This, too, is the most tangible significance of open weights: it turns “evaluable, deployable, adaptable” from a slogan into engineering options.
