Evaluating Open-Source Chinese Code Generation Models: A Real-World Engineering Assessment

1. Background: Open-Source Code Models Are Reaching “Usable” Status

Over the past year, code generation models have advanced dramatically. What started as function-level autocomplete has evolved into generating complete, accessible frontend projects from a single requirements document. The models have crossed a critical threshold in engineering usability.

Domestic open-source models have been particularly active in this wave of progress. Some now score close to leading international closed-source models on public benchmarks, fueling ongoing debates about performance parity.

But benchmark scores and real-world development experience are not the same thing. Whether a model can actually deliver in a real project depends on how deeply it understands the requirements document, how accurately it reproduces interaction details, and how consistently it handles multi-page coordination.

This article skips the benchmarks. Instead, I ran an end-to-end evaluation using a complete frontend requirements document, with three goals in mind:

• Identify the current capability boundaries of open-source code models.
• Design a reusable evaluation methodology.
• Provide practical guidance on how developers and teams should use these models in real engineering workflows.

2. Evaluation Design: Document-Driven Generation to Simulate a Real Scenario

2.1 Task Setup

The evaluation used a requirements document for three typical pages: a login page with a carousel, a product list page with multi-condition filtering and interaction logic, and a form page with field dependencies and validation rules. No technology stack was mandated—the model was free to choose. In practice, it generated a single-page application based on a mainstream component-based framework.

The key point here is that the requirements document was written clearly. Filters, field validation rules, and interaction feedback were all explicitly specified. This eliminates ambiguity as a confounding factor, so any errors can be attributed to the model’s understanding and execution rather than to an unclear brief.

2.2 Evaluation Dimensions

I assessed the output across four dimensions, each scored independently for cross-model comparison:

1. Functional completeness: Whether all explicitly required features were implemented.
2. Semantic precision: Whether filtering conditions, field relationships, and validation logic matched the document’s intent.
3. Deliverable format: Whether the output was a code snippet or a fully runnable, accessible project.
4. Generation efficiency: Time from submitting the requirements to receiving a usable result.

The third dimension deserves emphasis. The gap between a code snippet and a runnable project is really the gap between syntactic correctness and engineering competence. The former only proves the code compiles; the latter proves the model can handle directory structure, dependency management, routing, and project organization.

3. Results: Clear Differences Across Capability Layers

3.1 Usability and Deliverable Format

The generation process was not entirely smooth. During peak hours, the system showed queue notifications and required multiple retries before starting. This did not materially affect the evaluation conclusions, but it highlights a practical concern: free-tier quotas and concurrency limits directly determine whether a model is viable for team collaboration. Production use requires proper quota planning.

The deliverable format was a highlight of this evaluation. The model produced an accessible project link rather than scattered code snippets. This means the path from requirements to a runnable artifact was fully connected, which significantly lowers verification costs.

3.2 Login Page: Nearly Ready with Minor Tweaks

The login page included a dynamic carousel requirement. The model correctly identified and implemented the interactive behavior. All expected page elements were present, with no obvious missing functionality. The overall quality was “structurally correct with minor details to polish”—essentially a few style adjustments and brand-specific customizations away from production readiness.

This suggests that for well-structured pages with standardized interaction patterns, the model’s fidelity is already quite reliable.

3.3 List Page: Structured Understanding Starts to Slip

The problems concentrated on the list page. The requirements document clearly defined multiple filter groups and their interaction logic, but the generated output omitted the filter controls entirely. The page structure was correct but functionally incomplete. More notably, this pattern repeated across multiple list pages, indicating a systematic weakness rather than an occasional mistake.

From a technical perspective, the challenge of a list page is not rendering—it is state management and conditional composition. Multiple filter dimensions depend on each other, requiring the model to maintain a consistent, derivable state model. The current model is better at “static structure reproduction” and still shows clear limitations in “state space modeling.”

3.4 Form Page: Incomplete Field Relationships and Validation Logic

The form page ran without errors, but field definitions, validation rules, and interaction feedback deviated from the document. Even when the document explicitly stated the constraints between fields, the model did not fully implement them. This exposes the flip side of the same issue: the model tends to generate forms that “look right” rather than forms that are “semantically correct.”

A practical workaround is to split complex forms into smaller sub-requirements, generate them separately, and then manually merge the validation logic. This offsets the model’s limited comprehension depth with an engineering process.

3.5 Overall Style: Engineering Usability First

The UI style was serviceable: better than most simple prototypes, but not at the polish level of a mature design system. The visual design felt understated and engineer-oriented. This positioning is actually reasonable—the goal of a code model is to produce a maintainable engineering skeleton, while visual refinement should be handled by design systems and manual adjustments.

4. Side-by-Side Comparison: The Core Gap Is Comprehension Depth

I ran the same requirements document through a more capable baseline model for comparison:

Dimension	Open-Source Model	Baseline Model
Login page	Near parity	Near parity
List page	Correct structure, missing filters	More complete filter and field structure
Form page	Some fields and validation rules missing	Accurate field relationships and validation
Generation speed	Slower (partly due to free-tier quota)	Noticeably faster

The conclusion can be summed up in one sentence: the gap is not in “can it write code” but in “how deeply does it understand the requirements.” Page skeleton generation has become largely commoditized. The real differentiator is semantic parsing precision—especially for complex logic involving multi-condition combinations, field dependencies, and state transitions.

This also explains why the two models perform similarly on simple pages but diverge sharply as page complexity increases.

5. Implications for Open Source and AI Development

5.1 Open Source Makes Capability Comparison Verifiable

With closed-source models, capabilities can only be observed indirectly through outputs. Open-source models allow developers to inspect weights, fine-tune for specific domains, and reproduce benchmark results. This means performance comparisons are no longer just vendor claims—they become independently verifiable propositions. Verifiability is one of the most important values that open source provides.

5.2 Lowering the Barrier to Engineering Integration

Open source brings deployment freedom: you can self-host to meet data compliance requirements, fine-tune for specific technology stacks, or customize output style to match team conventions. For organizations with code asset confidentiality needs, this often matters more than raw benchmark scores.

5.3 Pushing Evaluation from Benchmarks Toward Task-Based Assessment

This evaluation also demonstrates that general-purpose benchmark scores poorly reflect real engineering performance. The open-source ecosystem is better suited to task-based evaluation methods: fixed requirements documents, fixed acceptance criteria, and multiple models evaluated against the same task set. This approach is low-cost, reproducible, and over time will push model development toward “engineering usefulness” rather than “leaderboard performance.”

5.4 A Sober View of Parity Narratives

It is important to recognize that parity on a single task does not mean equivalent overall capability. Measurable gaps remain in comprehension depth, complex logic reproduction, and generation stability. For developers, the sensible posture is: use open-source models for the tasks they handle well, and let human engineers or more capable models handle complex logic as a fallback. Do not expect a single generation pass to solve every problem.

6. Practical Recommendations

Based on the observations above, here are several actionable practices:

1. Layer model usage by page complexity. Standardized pages like login and display pages can be generated as first drafts. For pages with complex state and validation logic, split the work into smaller sub-requirements and generate them incrementally.

2. Treat requirements documents as prompt assets. The more structured the document—field tables, state tables, interaction checklists—the higher the model’s fidelity. Teams can build reusable requirement templates as generation assets.

3. Establish an acceptance checklist for generated output. At minimum, cover functional completeness, field and validation consistency, and routing and dependency integrity. Standardized acceptance criteria are necessary to integrate model output into a regular development workflow.

4. Plan quotas and concurrency. Queue delays under free-tier quotas will directly impact the experience. Production use requires advance planning for API quotas and fallback strategies.

5. Keep human oversight in the loop. Visual refinement, complex interactions, and edge-case handling should remain human responsibilities. The model’s job is to minimize the cost of getting from zero to a working foundation.

7. Conclusion

Domestic open-source code generation models have crossed the threshold of producing runnable projects and reached engineering-useful quality on standard page scenarios. The gap with leading models lies in requirement comprehension depth, not in basic generation capability. The open-source nature of these models makes this gap measurable, addressable, and customizable—which is itself a meaningful step forward in AI engineering practice.

For developers, the highest-value action is not waiting for models to catch up. It is building a working method now: evaluate, use models selectively by task layer, and keep human expertise as the final safeguard. Models will keep evolving; a sound methodology will remain useful throughout.