Testing Chinese AI Coding Models End-to-End: From Requirements Document to Deliverable Project
An end-to-end test of Chinese AI coding models using a real product requirements document (PRD) covering three typical page types: login, list, and form. The findings show that structure-driven tasks are reliable enough to ship with only light adjustments, while logic-dense tasks such as filter linkage, conditional required fields, and cross-field validation remain systemic weak spots rather than one-off misses. The article distills a reusable evaluation methodology, seven practical engineering techniques for closing the gap, and an enterprise perspective on turning a single trial into sustainable AI coding infrastructure. It also explains what the rise of open-weight models actually changes. The core takeaway: model capability sets the ceiling, but engineering discipline determines how high you can actually reach.
Start With the Conclusion: Leading the Benchmark Is Not the Same as Being Deliverable
There has been a lot of discussion lately about how Chinese large models have caught up with the world’s leading models on code generation. The buzz is loud, but what actually decides whether a tool can enter production is never a side-by-side score comparison. It comes down to three very concrete questions:
- Given a real requirements document, can it generate a runnable project in a single pass?
- Does the generated code meet your own acceptance criteria for functional completeness?
- From requirements to delivery, how much extra human effort do you need to spend cleaning up after it?
With those three questions in mind, I designed an end-to-end test: I fed a real product requirements document that I had actually used before — covering three typical page types (login, list, and form) — completely into an AI coding tool. I imposed no tech-stack constraint, made no manual corrections, and asked no follow-up questions. I only looked at the final output.
This article neither recommends nor reviews any specific product. Instead, it turns the results of this test into a reusable evaluation methodology and an engineering playbook for putting AI coding tools to work. The real question at the center is: where exactly are the capability boundaries of AI coding tools, and how should enterprises actually adopt them?
Evaluation Design: Making the Comparison Reproducible
Most debates about AI coding ability never reach a consensus because the inputs aren’t comparable, the environments aren’t consistent, and the scoring criteria are vague. Before the conclusions can hold up, the evaluation design itself has to be disciplined.
2.1 Four Preconditions
- Fixed input: The same requirements document, the same version, with no additional prompts. Adding follow-up prompts turns the test into “human-AI collaboration,” which is a different thing from “one-shot delivery” and must be tested separately.
- Consistent environment: The same runtime environment, the same dependency version constraints, and the same directory-structure requirements (or none at all).
- Zero human intervention: No corrections, no steering during generation. The moment you start asking follow-up questions, you’re no longer measuring the model’s comprehension — you’re measuring your prompt-engineering skills.
- Standards set upfront: Write the acceptance criteria down before generation, so you avoid subjective judgments like “looks fine to me.”
2.2 Evaluation Dimensions and Criteria
| Dimension | What to Observe | Criteria |
|---|---|---|
| Usability | Rate limiting, queuing, quota limits | Can it reliably enter the task state? |
| Runnability | Can it start up and be accessed directly? | Does it require manual fixes to dependencies or config? |
| Structural completeness | Are all pages/modules present? | Page coverage against the requirements document |
| Semantic accuracy | Interaction logic, field relationships, validation rules | Number of deviations from the document |
| Generation efficiency | End-to-end time | Time from submission to a runnable output |
| Visual completeness | Layout, spacing, hierarchy, consistency | Does it have a baseline quality good enough to ship? |
| Maintainability | Component breakdown, type constraints, naming | Can the team take it over later? |
2.3 Layering the Requirements Document
I deliberately wrote the requirements document across three difficulty tiers so I could pinpoint where “capability degradation” begins:
- L1 (structural): Login page — fixed layout, few fields, a single interaction path.
- L2 (logic): Product list page — pagination, multi-condition filtering, empty and loading states, and filter state synced to URL parameters.
- L3 (constraint): Form page — a dozen-plus fields, cross-field linkage, conditional required fields, format validation, and submit and error states.
These three tiers mirror the typical distribution of real-world work, and they also correspond to three different kinds of test for an AI’s capabilities.
Test Observations: Where Capability Degrades
3.1 Availability Comes Before Ability: Don’t Forget the Reality of Rate Limiting
On the first attempt, the tool kept showing “too many users right now, you need to queue or upgrade your quota,” and only after several retries did it finally enter the task state. This detail seems unrelated to model capability, but in a real production environment it is the most easily overlooked and the most disruptive to your delivery cadence.
The engineering takeaway is straightforward: any R&D workflow that depends on external inference services must treat rate limiting, queuing, and quota exhaustion as the norm, not as an exception. Requests need backoff-and-retry, tasks need checkpoint-resume, and critical paths need a fallback channel. Taking “it can just generate” for granted is the first pit many teams step into after adopting AI coding capabilities.
3.2 L1 Login Page: Both Structure and Visuals Pass
I had explicitly required a carousel, a form layout, and basic interactions in the document. The result was fairly satisfying: the carousel was implemented, the form elements were all present, the layout had no obvious misalignment, and the interaction path was complete and usable. Overall completeness was at the level of “adjust the styling and it’s ready to ship.”
This tells us that for tasks with clear structure and single-purpose semantics, mainstream models are now quite reliable. This kind of requirement — admin back-office pages, simple display pages, standard form-collection pages — can be handed to AI generation with confidence.
3.3 L2 List Page: Depth of Understanding Starts to Fall Behind
Problems appeared here. I had clearly written the filter conditions, the linkage between filters and the list, and the requirement that filter state sync to the URL. In the actual output:
- The page skeleton was correct — the table, pagination, and headers were all there;
- The filter controls were entirely missing;
- The linkage logic between filters and the list naturally couldn’t exist either.
It’s worth stressing that this was not an occasional oversight. Across multiple list-page generations, the same kind of problem recurred — which shows this is a systemic shortcoming, not random variance.
Correct page structure but missing functional logic is a diagnostically valuable combination: the model understood “this is a list page,” but it did not understand “this is a list page with filter linkage.” It completed the recognition of the page type, but not the modeling of the business semantics.
3.4 L3 Form Page: Constraint Restoration Is the Biggest Weakness
The form page ran, but a closer look revealed the gaps: some fields weren’t implemented, cross-field linkage was missing, conditional required logic wasn’t applied, format validation covered only the most basic case, and the position and timing of error messages didn’t match the document.
The key point is: the requirements document itself was not vague. I had listed the field inventory, required rules, linkage relationships, and validation regexes clearly. The model’s shortcoming isn’t “it can’t read the text” — it’s “it didn’t translate the constraints in the text into code, one by one.”
The defining trait of an L3 requirement is high constraint density, with dependencies between constraints (field B is only required when field A is set to X). This is exactly the scenario generative models handle worst — they’re good at continuing with plausible-looking content, but bad at maintaining a table of constraints that hold each other in check.
3.5 Visual Completeness: Engineering-Usable, Not Product-Stunning
The UI style was unremarkable: better than a typical prototype mockup, but still short of what you get by directly applying a mature component library with atomic styling. The design felt simple and engineering-oriented — the layout was tidy but lacked hierarchy and breathing room, and the details (spacing rhythm, state feedback, empty-state design) weren’t polished.
The positioning is clear: engineering-usable > product-pretty. For internal systems this is a plus. For customer-facing products, the visual layer still needs a designer or a template library as a backstop.
3.6 The Efficiency Dimension: Don’t Only Look at Results
For generation time, a single complete run took more than half an hour. I had built the same requirement once before using another approach (a mature template library plus manual assembly), and the time cost was clearly lower.
The efficiency gap has multiple causes: the task was large, resources were constrained under a free quota, generation was serial, and there was no parallel splitting. One common misjudgment is worth flagging here: many people misread slowness caused by rate limiting and quota as slow model inference. When running an evaluation, always record “platform-constraint time” and “model generation time” separately, or your conclusions will be distorted.
Attributing the Gap: From “Generating Code” to “Understanding Constraints”
Abstracting away the phenomena above, the capability gap of AI coding tools concentrates on four things.
1) Limited capacity to carry structured constraints. The more constraints a requirement has, and the more deeply they’re coupled, the more likely the model is to drop them. It’s not that it fails to understand a single rule — it’s that it can’t simultaneously maintain dozens of cross-referencing rules during generation.
2) Implicit needs must be made explicit. When humans write documents, they automatically omit the “obvious” parts — “filter conditions should sync into the URL,” “switching pages should preserve filter state.” These are things developers don’t need to say to each other, but for the model they’re completely missing information. What’s not written in the document is treated as not required.
3) A self-verification loop is missing. After generating the code, there’s no mechanism to check “out of the 20 requirements in the document, how many did I implement?” The model’s output is one-directional; it doesn’t go back and reconcile.
4) Attention is diluted on complex tasks. Generating an entire project in one pass means the attention budget is spread across dozens of files. By the end, each file gets only a shallow understanding — which is exactly why modules like the list page and form page, with “high logic density but not much code,” are more prone to problems.
All four attributions point to the same conclusion: the essence of the capability gap is an engineering gap. And since it’s an engineering problem, it can be closed with engineering means.
Putting Uncertainty in a Cage: Seven Practical Techniques
5.1 Contract First: Write Requirements as Machine-Readable Constraints
To a model, a natural-language document is a “suggestion”; a schema is a “constraint.” A large part of the form-page problem can be solved with an up-front field-definition file:
{"entity": "device_form","fields": [{ "name": "deviceType", "type": "enum", "required": true,"options": ["gateway", "sensor", "camera"] },{ "name": "serialNo", "type": "string", "required": true,"pattern": "^[A-Z]{3}-\\d{8}$" },{ "name": "firmwareVersion", "type": "string", "required": false,"visibleWhen": { "deviceType": ["gateway"] } },{ "name": "firmwareVersion", "type": "string", "required": true,"requiredWhen": { "deviceType": ["gateway"] } }],"submit": { "mode": "single", "idempotent": true }}
Pair it with a constraining instruction:
The implementation must treat the schema above as the single source of truth.The field list, validation rules, visibility and required conditions must all begenerated from the schema. Hardcoding any field name or regex in the businesscode is not allowed.
The benefits are twofold: the model gets an explicit basis for reconciliation, and you get an intermediate artifact that can be automatically validated.
5.2 Write the Acceptance Checklist Before Generating
Turn “I assumed it understood” into “I can tick off each item”:
- All N fields listed in the document exist, with matching names, types, and order
- Required-field empty-value validation triggers, and the error message appears in the correct position
- Visibility-linked fields appear/disappear correctly after toggling, without clearing already-entered values
- Repeated clicks during submit are blocked
- No request is fired when validation fails
This checklist is both your acceptance standard and a direct input to the prompt. Putting acceptance criteria upfront is the lowest-cost way to raise the first-pass success rate.
5.3 Generate in Layers, Iterate Module by Module
Don’t generate the entire project at once. The recommended cadence is:
- First generate the project skeleton and confirm the tech stack (runnable, startable);
- Then generate page by page or module by module, focusing on one page at a time;
- Immediately after each module, run a manual or automated acceptance check, and move on only once it passes.
When the attention budget is concentrated on a single module, the fidelity for logic-dense requirements like L2 and L3 improves noticeably.
5.4 Use Static Checks as an Automatic Judge
Type checking and static analysis are the most cost-effective gatekeepers. With strict mode enabled, misspelled field names, type mismatches, and unused variables get caught directly:
# Run static checks and type checks in strict mode; use zero warnings as the merge gatetypecheck --strictlint --max-warnings 0
These are especially worth setting as hard gates for AI-generated code. The typical problem with such code isn’t that it “looks ugly” — it’s that it “looks right but fails when it runs,” and only strict static checks can fish out this kind of problem at low cost.
5.5 Add Automated Acceptance on Critical Paths
Static checks handle structural issues; behavioral correctness goes to hand-written test cases. Even the most minimal set of assertions can cover the vast majority of gaps like “the filter controls weren’t implemented”:
def test_filter_panel_exists():page = render_list_page()assert page.count("filter_item") == 3, "Filter count does not match the document"def test_filter_drives_list():page = render_list_page()page.select_filter("status", "offline")assert page.query_params["status"] == "offline", "Filter condition not synced to URL"assert page.first_row["status"] == "offline"def test_conditional_required_field():form = render_form()form.fill({"deviceType": "gateway", "serialNo": "ABC-12345678"})form.submit()assert form.has_error("firmwareVersion"), "Conditional-required rule not enforced"
These three tests land precisely on the three failure modes exposed in the test. A test case is the “acceptance contract” you hand to the model.
5.6 Reuse Mature Visual Infrastructure
Don’t make the model design a visual system from scratch. Pick a mature component library and styling approach as the foundation, and let the model do only the business assembly — visual-consistency problems then largely disappear on their own. If you’re delivering to external customers and care about design quality, the visual layer should be defined by a designer (color, spacing, type scale, component states), and the model fills in within that spec.
5.7 Clarify the Boundary of Human Intervention
Set review intensity by page type:
| Page Type | Recommended Intervention |
|---|---|
| Display / list | Light review, static checks + smoke tests |
| Form | Medium review, item-by-item against the acceptance checklist |
| Permissions, payments, data writes | Heavy review, line-by-line human review + security scan |
Put human effort where it’s needed most, instead of spreading it evenly across all the code — this is the dividing line for whether AI coding actually improves efficiency.
Build Your Own Evaluation Baseline: Turn Experience Into Data
A single test only tells you “how it did this time.” To keep making decisions, you need an evaluation baseline that can be run repeatedly.
6.1 Metric Set
- First-pass success rate: The share that meets acceptance criteria without manual modification
- Fix rounds: The number of iterations needed from first output to acceptance
- Defect density: Defects per hundred lines of code, graded by severity
- End-to-end time: Split into “platform queue wait” and “model generation”
- Manual effort hours: This is the real bulk of the cost
- Constraint coverage: The share of constraints in the document that were correctly implemented (the most diagnostically valuable metric)
6.2 Build Failure-Mode Labels
Classification is more useful than a score. Maintain at least these labels: structural omission, semantic misunderstanding, constraint omission, constraint conflict, visual inconsistency, unavailable dependencies, and security issues. Only when you know which category the failures cluster in do you know whether to optimize the prompt, add tests, or change the generation strategy.
6.3 Draw the Evaluation Set From Real Requirements
Build the evaluation set from real historical requirements (already shipped, with acceptance criteria); 20–30 samples are enough to create meaningful discrimination. Mix them by L1/L2/L3 tier, and re-run after every model or toolchain change — use trend lines instead of single-point conclusions.
The Enterprise View: From a Single Trial to Sustainable Capability Infrastructure
There’s an engineering chasm between personal experimentation and enterprise adoption. The latter needs four pieces of infrastructure.
Capacity and scheduling. Inference services must scale elastically with team size, and the request side needs queueing, rate limiting, quotas, and priority mechanisms to keep critical paths from being crowded out by routine calls. The queuing problem mentioned earlier is, at its core, a capacity-governance problem.
Cost observability. The cost of AI coding isn’t in the number of calls but in token consumption and manual rework. You need to break down usage by team, by project, and by task type, and match cost against actual delivery value — only then do you know where to invest more and where to rein it in.
Hybrid routing strategy. Not every task deserves the most expensive model. Tasks like generating single-file utility functions, writing unit tests, and filling in comments can be routed to lightweight models; cross-module architecture changes and complex business-rule implementations should use flagship capability. Routing in tiers through a unified model-service gateway on a cloud platform usually improves overall output per unit of cost.
Data and compliance boundaries. Before feeding private code to an external inference service, you must first clarify data flows and retention policies. For sensitive codebases, private deployment or inference in a dedicated environment is the safer choice — on a cloud platform like Baidu AI Cloud, general capabilities such as dedicated network isolation, access-control permissions, operation-log auditing, and key management can keep the data boundary within a controlled range. These are all generic capability categories; refer to your platform’s official documentation for specific configuration.
In addition, logging and monitoring/alerting are equally indispensable: record the input summary, time taken, failure reason, and manual rework for each generation. Only then can you build the failure-mode label library described above and let evaluation data truly drive improvement.
What the Rise of Open-Weight Models Really Means
Back to the bigger question: with Chinese models rapidly closing in on the world’s leading level, what does this mean for the open-source ecosystem and AI development? Here we’re talking about open-weight models specifically, not all “open-source AI.”
First, it returns the choice to the user. Available weights mean you can deploy privately, run on your own hardware, and keep sensitive data entirely within your network. For enterprises, this is the key leap from “can I use it?” to “dare I use it?”
Second, it turns evaluation from hearsay into experiment. Once weights are open, anyone can reproduce conclusions on their own data. The reason the evaluation method above can be laid out clearly is precisely that the model can be deployed locally, run repeatedly, and compared across configurations. Reproducibility is the most basic fuel of technical progress.
Third, fine-tuning lets models fit the domain. General models don’t understand your business terminology, field-naming conventions, or internal component library. With weights, you can continue training or do parameter-efficient fine-tuning on your internal codebase, so the model “speaks your dialect.” This is a depth of customization that’s hard to get when calling a closed-source API.
Fourth, competition pushes down costs along the whole chain. The more capability providers there are, the lower the price per token goes, and the more likely small and medium teams can afford it. AI coding becoming an inclusive tool rather than a big-company privilege rests on exactly this.
Fifth, stay clear-headed. Open weights don’t mean unrestricted use — the commercial scope, distribution obligations, and derivative-naming requirements in the license terms must be confirmed item by item before going to production. At the same time, the availability of weights does not automatically solve the problems exposed in this test: understanding constraints and restoring complex business logic depend on engineering methods, not parameter count. Open weights solve the “can I control it” question, not the “can it get it right in one pass” question.
Summary: What This Test Really Tells Us
- Structural requirements can be handed to AI with confidence. Display pages, standard list pages, and simple forms can be delivered with one generation plus light adjustments.
- Logic-dense requirements still need human-AI collaboration. Filter linkage, conditional required fields, and cross-field validation are clear current weaknesses — and systemic ones, not occasional.
- The essence of the gap is an engineering gap. Getting contract-first, explicit constraints, layered generation, static checks, automated acceptance, and visual-infrastructure reuse right will produce a real improvement in first-pass success rate.
- Rate limiting and quotas must be designed for as the norm. Backoff-and-retry, checkpoint-resume, and fallback channels are essential when integrating an inference service.
- Evaluation must be accumulated as a data asset. A metric set + failure-mode labels + a real-requirements evaluation set is what turns a one-off experience into a sustainable basis for decisions.
- Open weights are a long-term dividend, not a short-term answer. They give you control and customization, but “getting it right in one pass” still depends on engineering discipline.
To close it all out: model capability sets the ceiling for AI coding, but engineering method determines how high you can actually stand. Rather than fretting over one output that isn’t stunning, first build the three pieces of infrastructure — contracts, validation, and acceptance. All three carry over no matter which model you switch to.
