Evaluating Domestic LLMs in Real-World Engineering: From “Runnable” to “Production-Ready”

1. Background: The Real Challenge Beyond Benchmark Scores

Recently, domestic large language models (LLMs) have frequently announced that their performance rivals top-tier international models on public leaderboards. Metrics such as parameter scale, reasoning capabilities, and multimodal support are hitting new highs. From a technical communication perspective, these are undoubtedly exciting signals. However, for engineers actually embedding these models into development workflows, benchmark scores are just the starting point. The true value of a model lies in its performance in real-world engineering tasks: Can it understand a structured requirement document? Can it fully implement interaction logic? Is the generated code merely a “demo snippet” or a “deliverable project”?

This article does not focus on parameters or benchmark scores. Instead, it returns to the engineering context, systematically analyzing the capabilities, shortcomings, and improvement paths of domestic LLMs in practical use, centered around a complete frontend project generation task. It is important to note that this discussion addresses general engineering capabilities of LLMs. The tools and platforms involved are described neutrally, focusing on methodology and capability evaluation frameworks rather than promoting specific products.

2. Evaluation Design: Testing True Capabilities with Real Requirements

2.1 Task Setup

The evaluation task selected a typical mid-sized frontend project containing three core pages: a login page with a carousel, a product list page with multi-condition filtering, and a form page with field validation and linked logic. Requirements were provided as a natural language document without specifying a tech stack, and the model independently chose a mainstream component-based framework for implementation.

The rationale for this design is that in real projects, requirement documents often contain numerous implicit constraints and interaction details. Whether a model can extract a complete “behavioral contract” from the text is a critical watershed for engineering usability. While the number of pages is small, they cover three typical scenarios: static display, data filtering, and form interaction, which is sufficient to observe the model’s depth of understanding across different levels of abstraction.

2.2 Observation Dimensions

The evaluation was conducted across four dimensions:

• Functional Completeness: Whether all interactions explicitly described in the document are implemented.
• Structural Rationality: Whether page component division and state management conform to engineering conventions.
• Restoration Precision: Whether field relationships, validation rules, and linked logic match the document.
• Generation Efficiency: The time elapsed from submitting requirements to producing an accessible result.

These four dimensions form a continuous spectrum from “runnable” to “usable”, better reflecting engineering value than a single accuracy metric.

3. Scenario-Based Results Analysis

3.1 Login Page: High Completeness, Approaching Direct Deliverability

The login page performed the best among the three. The carousel, form layout, and basic validation required in the document were fully implemented. The page structure was clear, component division was reasonable, and although the visual style was plain, there were no obvious functional omissions.

From an engineering perspective, this result is expected: the interaction pattern of a login page is highly standardized, and similar implementations are abundant in the training data. The model can rely on pattern matching to produce high-quality output. This indicates that for highly patterned tasks, current model outputs have reached a “ready-to-use with minor tweaks” level.

A notable detail is that the model provided a directly accessible project URL rather than scattered code snippets. This “end-to-end delivery” format is more developer-friendly and reduces verification costs, serving as a positive signal of improved engineering capabilities.

3.2 List Page: Correct Structure, but Functional Understanding Slows Down

Upon moving to the list page, issues began to emerge. The document explicitly described multi-condition filtering and the linked logic between filter states and list data. However, in the actual output, the filtering was not implemented, and the page only displayed a static list structure.

This phenomenon is not an isolated incident but recurred across multiple list pages, indicating a systematic capability shortfall rather than a random generation error. The root cause can be understood from two perspectives:

First, filtering logic is a “state-driven” interaction. It requires the model to maintain an implicit state model during generation, linking user actions, filter conditions, and data refresh together. This type of cross-component logic orchestration demands strong intermediate reasoning capabilities.

Second, filtering descriptions in requirement documents often list conditions in natural language. The model tends to interpret this as “display these fields” rather than “implement filtering behavior for these fields”. The gap between semantic understanding and behavior modeling is the primary bottleneck for current models in complex interaction scenarios.

3.3 Form Page: Insufficient Restoration Precision for Fields and Validation

The overall framework of the form page was runnable, but the detail restoration was noticeably inadequate. Field types, mandatory constraints, inter-field linkage rules, and pre-submit validation flows defined in the document were not fully implemented. Some fields were simplified, and some validation logic was missing.

Forms represent the highest information density in a requirement document. Every field comes with multiple attributes like type, constraints, default values, and error messages, and dependencies may exist between fields. To accurately restore these, the model must continuously maintain a structured “requirement mapping table” during generation and reflect it item by item in the code. Current models are better at generating code that “looks like a form” but still show a significant gap in aligning with the requirement contract item by item.

3.4 Overall Style: Engineering Usability First, Design Aesthetics Second

Visually and structurally, the output was “standard”: better than most hand-drawn prototypes, but falling short of the polished look of applying mature component libraries and design systems directly. The style leaned towards simple and engineering-focused, lacking fine design polish.

This positioning is not inherently wrong. In engineering scenarios, the priority of “correct function and clear structure” is much higher than “visually stunning”. However, if the goal is end-user product delivery, the model’s output requires secondary processing in areas like style specifications, component reuse, and responsive adaptation using mature design systems. The model builds the skeleton; the design system provides the flesh. This division of labor remains the most realistic collaboration model at this stage.

4. Comparative Analysis: The Core Gap Lies in “Understanding Depth”

A valuable comparison can be made by assigning the same requirements to a model with stronger reasoning capabilities. On the login page, both models performed similarly. However, on the list and form pages, the latter showed significantly more complete restoration of filtering conditions, field structures, and validation logic, with higher accuracy in interaction details. Regarding generation efficiency, the latter typically completed the task in a few minutes, while the former took noticeably longer.

This comparison reveals a key conclusion: the primary gap between current domestic models and top-tier international models is not whether they can generate code, but whether they can accurately understand requirements. Code generation itself has become highly patterned. The real difficulty lies in translating natural language requirements into a complete, unambiguous behavioral contract and implementing it item by item.

This gap is masked by pattern-matching capabilities on simple pages but fully exposed on pages with complex interactions. It also explains why many developers feel that “the demo is stunning, but the actual implementation falls short”—demo scenarios are usually patterned, while the complexity of real project requirements far exceeds that of demos.

5. Practical Recommendations for Engineering Integration

Based on the observations above, several actionable practices can be derived to help teams use LLMs more efficiently for frontend engineering tasks.

5.1 Layered Requirement Decomposition: Avoid “One-Shot Massive Documents”

Break down requirement documents into finer-grained sub-tasks by page and interaction module, submitting them for generation and verification iteratively. Submitting a massive document at once easily dilutes the model’s attention across a long context. Decomposing it item by item allows each generation cycle to focus on a clear behavioral contract, significantly improving restoration precision.

5.2 Establish a “Behavioral Contract” Checklist for Item-by-Item Acceptance

Before submitting requirements, convert the interaction descriptions in the document into verifiable checklist items, such as “When filter A changes, list data refreshes” or “When field B is empty, the submit button is disabled.” After generation, check them against the list one by one, transforming the vague “looks right” into an explicit “passes item by item.” This practice effectively exposes the model’s omissions in state linkage and validation logic.

5.3 Replace Pure Natural Language with Structured Descriptions

For form scenarios with complex field relationships and multiple validation rules, introduce structured expressions in the requirement description. For example, list field names, types, constraints, default values, and linkage rules in a table. Structured input has lower ambiguity and reduces the cost for the model to align with requirements.

Field: Phone Number
Type: string
Constraint: Required, 11 digits
Linkage: Disable submit button if empty

This type of description is not complex but can significantly reduce the model’s understanding deviation.

5.4 Human-Machine Division of Labor: Model for Skeleton, Engineers for Details

Position the model as a “skeleton generator” rather than a “fully automated deliverer.” Let the model complete the page structure, basic components, and main process code, and then have developers fill in state management, boundary handling, style specifications, and design system integration. This division of labor leverages the model’s efficiency advantages while mitigating its shortcomings in complex logic.

5.5 Establish a Regression Verification Mechanism

For key pages generated by the model, build reusable verification cases covering core interactions like filtering, validation, and linkage. Automatically run regressions after each model iteration or requirement change to ensure that capability improvements are real and measurable, rather than subjective fluctuations.

6. Implications for Open Source and Industry Development

Returning to the core question: What does the performance parity of domestic models with international top tiers mean for open source attributes and industry development?

First, the engineering value of the open-source ecosystem is shifting from “usable” to “user-friendly”. As model capabilities approach top-tier levels, the open-source community gains not just runnable weights, but a productivity tool that can be embedded into real R&D workflows. This will accelerate the accumulation of engineering assets built around the models, such as scaffolding tools, evaluation sets, and best practices.

Second, the focus of evaluation needs to shift from leaderboards to scenarios. Parameters and benchmark scores cannot reflect engineering-critical metrics like requirement understanding depth and interaction restoration precision. Establishing public evaluation sets oriented toward real-world tasks is more conducive to healthy industry development than chasing a single score.

Third, exposing gaps is a prerequisite for progress. The “correct structure, discounted logic” phenomenon observed in this article clearly points out the direction for the next stage of capability building: strengthen requirement modeling capabilities in long contexts, improve the generation quality of state-driven interactions, and enhance field-level restoration precision under complex constraints. These directions are clear, verifiable, and conducive to forming an iterative closed loop.

7. Conclusion

The performance of domestic LLMs in engineering tasks has crossed the threshold of “being able to generate code” and now stands before the threshold of “being able to accurately understand requirements.” The stable performance on the login page proves the maturity of patterned tasks, while the shortcomings on the list and form pages reveal the gap in understanding depth. For developers, the rational attitude is neither blind hype nor blanket dismissal, but rather establishing a practical methodology of layered verification and human-machine collaboration. Place the model where it excels currently, and use engineering means to bridge its deficiencies. As models evolve, methodologies must evolve in tandem—this is perhaps more worthy of attention than any single evaluation result.