Evaluating LLM Code Generation in Real Projects: A Reproducible Method and the Open-Source Value

Introduction: After the Leaderboard Gap Narrows, What Really Matters

Open-source large models have been iterating at a striking pace over the past two years. Whenever a new model is released, the most common phrasing in community discussion is that its “performance matches some closed-source flagship model.” Rapidly narrowing benchmark scores are encouraging, but for frontline developers and enterprise technical decision-makers, the real question is: how capable are these models on actual engineering tasks? Does the “matching” claim hold up under project-level scrutiny?

This article does not focus on any specific product. Instead, using page-level front-end code generation—a high-frequency scenario—as an example, it distills a reproducible evaluation method, analyzes common failure modes, and further discusses what the development of open-source models means for the AI ecosystem and for enterprise model selection.

1. Why Benchmark Scores Cannot Replace Real-World Evaluation

1.1 Differences in Task Distribution

Public evaluation sets are mostly standardized problems: short prompts, few constraints, and answers that are either unique or automatically scorable. Real project requirements are long-tailed: a single requirements document may simultaneously contain layout descriptions, interaction logic, field rules, exception handling, and implicit team conventions. A model’s high score on standardized problems does not automatically transfer to this kind of “multi-constraint combination” task.

1.2 The Unreliability of Single-Sample Experience

Much of the “hands-on experience” content in the community relies on only one sample: test once, take a few screenshots, and draw a conclusion. Single-sample conclusions have extremely high variance and may be affected by account rate limiting, network fluctuations, prompt wording, and other factors. To reach credible conclusions, you must introduce a reproducible process rather than relying on a one-off “feel.”

2. A Reproducible Method for Evaluating Code Generation

2.1 Task Design: Use Real Project Slices Instead of Toy Demos

It is recommended to slice three pages of increasing difficulty from a real business system as evaluation tasks:

• Login page: static layout plus a small number of composed components (forms, carousel, validation messages), testing basic reproduction capability;
• List page: data display plus interaction logic such as filtering, sorting, and pagination, testing instruction following and logic implementation;
• Form page: multiple fields, field linkage, validation rules, and error messages, testing the ability to reproduce complex constraint combinations.

These three tasks cover a capability gradient from “static reproduction” to “logic implementation” to “multi-constraint combination,” and any weak link is clearly visible.

2.2 Requirements Document First: Define the “Ground Truth” Before Evaluating

Evaluation presupposes an explicit, structured requirements document as the baseline. Vague requirements only produce vague conclusions. It is recommended to describe page specifications in a structured format, for example:

page: product_list
filters:
  - name: category
    type: single_select
    behavior: after selection, refetch the list and reset pagination
  - name: price_range
    type: dual_slider
    behavior: trigger filtering 300ms after dragging stops
pagination:
  page_size: 20
acceptance:
  - after any filter item changes, the list data must refresh
  - filter state must be synced to URL query parameters and restored after page refresh

2.3 Build a Multi-Dimensional Scorecard

A simple “works / doesn’t work” judgment is too coarse. It is recommended to score by dimension:

Dimension	What to Examine	Reference Weight
Feature coverage	Proportion of feature points in the requirements document that are implemented	30%
Depth of understanding	Correct reproduction of filtering logic, field relationships, and validation rules	25%
Engineering quality	Directory structure, component decomposition, naming, and maintainability	15%
UI quality	Layout reasonableness, visual completeness, and design sense	15%
Delivery format	Whether a runnable project is provided rather than scattered snippets	10%
Efficiency and stability	Impact of generation time and queueing/rate limiting on the workflow	5%

2.4 Control Variables and Multiple Sampling

• Use the same requirements document and the same tech stack constraints, changing only the object under test;
• Sample each task at least three times to observe success rate and fluctuation range;
• Record experience metrics such as queueing time, rate-limit prompts, and end-to-end generation time—they are also part of “usability.”

3. Typical Failure Modes and Cause Analysis

Under the method above, many models—open-source or closed-source—often expose three categories of common problems.

3.1 Conditional Requirements Are Silently Ignored

The most typical phenomenon: the page skeleton is complete and visually flawless, but filter items explicitly written in the requirements document are not implemented. Such omissions do not raise errors or warnings; they are “silent failures.” There are two main causes:

• Conditional and secondary descriptions in long requirements documents experience attention-weight decay during generation;
• Models tend to prioritize “high-frequency common structures” (lists, cards, pagination) and treat custom logic as optional.

This is also why “the page structure is right, but functional understanding is discounted” becomes a common observation.

3.2 Insufficient Fidelity on Complex Forms

Multi-field forms involve a set of intertwined constraints: field linkage, validation order, and timing of error messages. Models easily generate the “statistically most common” form skeleton while missing personalized validation chains. During evaluation, focus on checking: whether the number of fields is consistent, whether linkage relationships hold, and whether validation timing matches the document.

3.3 Differences in Delivery Format and Efficiency

Some models output only code snippets that require manual assembly; others can provide a directly runnable project directory, which significantly lowers implementation cost. In addition, generation time and queueing/rate-limiting mechanisms directly affect development rhythm—during evaluation, you should look not only at the “final artifact” but also at “the process of obtaining it.”

4. Engineering Practices for Improving Fidelity

Model capabilities are evolving rapidly, but engineering-side coordination matters just as much. The following practices can significantly improve the consistency between generated results and the requirements document.

4.1 Rewrite Natural-Language Requirements as Structured Specifications

As in the example in 2.2, describing a page with the four elements “field, type, behavior, acceptance criteria” yields much higher fidelity than a prose description. A rule of thumb: every “should,” “needs to,” and “after” in the document should become a judgeable item.

4.2 Staged Generation and Incremental Acceptance

The failure rate of generating an entire site’s requirements in one shot is far higher than that of staged generation. Recommended flow:

1. First generate the page skeleton and routing structure, and confirm manually;
2. Then generate page by page, each with its own acceptance checklist;
3. Finally perform regression acceptance at the integration level.

4.3 Make Acceptance Criteria Part of the Prompt

Writing acceptance conditions directly into the context, or even converting them into test cases, is one of the most effective ways to constrain model output:

// Example acceptance tests (pseudocode)
describe("list page filtering", () => {
  it("should refetch data and reset pagination after selecting a category", async () => {
    await selectFilter("category", "digital");
    expect(requestCount).toBe(2);
    expect(listItems.every(i => i.category === "digital")).toBe(true);
  });
  it("should sync filter state to the URL and restore it after refresh", async () => {
    await selectFilter("category", "digital");
    await reload();
    expect(getFilterFromUrl()).toEqual({ category: "digital" });
  });
});

After generating code, run the cases directly and feed failure information back to the model for correction, forming a “generate—verify—fix” loop.

4.4 A Checklist of Key Points for Manual Review

Even with a sound process, manual spot checks should still be performed before delivery: click through each filter item, validate each form field, and go through boundary values (empty states, extreme values, duplicate submissions). AI handles efficiency; humans handle the safety net.

5. The Significance of Open-Source Models and Rational Model Selection

Back to the initial question: as open-source models’ public results gradually approach closed-source flagships, what does this mean for AI development?

5.1 Ecosystem-Level Significance

Open source means researchable, fine-tunable, and privately deployable. Open weights allow academia and engineering teams to reproduce, improve, and combine model capabilities, spreading innovation from a few organizations to the entire community; much of the recent vertical-domain fine-tuning practice is built on this foundation. The catch-up pressure from open source also pushes the closed-source camp to iterate faster, forming a virtuous capability spiral.

5.2 Enterprise-Level Value

• Data security and compliance: private deployment keeps sensitive data inside the internal network;
• Controllable cost structure: pay for compute investment rather than long-term per-call fees, yielding a better cost curve in large-scale scenarios;
• Supply diversity: avoid single-vendor lock-in and gain more leverage in architecture design and commercial negotiation.

5.3 Implementation Path: Build on Cloud Infrastructure

It is worth emphasizing that “open source” does not equal “free and easy to use well.” Building your own inference cluster involves a series of engineering issues such as GPU memory planning, throughput optimization, monitoring and alerting, and canary releases. For most enterprises, a more pragmatic path is to leverage mature cloud infrastructure: platforms such as Baidu AI Cloud typically provide general capabilities including model services, elastic compute, and logging and monitoring alerts, and together with basic components such as object storage and message queues, they can quickly engineer the “evaluate—pilot—launch” chain so that teams can focus on business value itself.

5.4 A Rational View of “Matching”

“Matching closed-source flagships” is a simplified expression in a communications context. For enterprises, the right approach is: don’t look at the ads, look at the scorecard—test with tasks sliced from your own business, requirements documents you wrote yourself, and acceptance criteria you defined yourself. Whichever model understands your scenario more deeply, reproduces requirements more accurately, and delivers more stably is the better choice right now. This is precisely the biggest benefit of open-source prosperity: you always have options, and you have the ability to verify them yourself.

Conclusion

The process of open-source models catching up with closed-source flagships is essentially a process in which AI capability supply moves from scarcity to diversity. For developers, rather than watching benchmark scores, it is better to turn evaluation methods into team assets: a set of structured requirements templates, a multi-dimensional scorecard, and a set of regression-ready acceptance cases. These three things do not depend on any specific model, yet they allow you to reach your own reproducible conclusions within half a day each time a new model is released.