Evaluating AI Coding Assistants: Building a Complete E-commerce Site from Scratch

1. Why a Complete E-commerce Site Is the Ideal Benchmark

When developers try out AI coding tools, they tend to ask the model to build a to-do list or a Snake game. Tasks like these only verify whether the tool “can write code”—not whether it “can deliver software.” Between a real project and a demo lies an entire engineering gap: navigation logic between pages, shared data models, a unified visual language, and a runnable build artifact.

An e-commerce site with 10 pages covers all of these elements:

• Transaction flow: home (product list), product detail, purchase, and purchase success pages form a complete browse–select–buy–complete journey.
• Authentication flow: login and registration pages test form interaction and state management.
• User center: dashboard, orders, wallet (coupon pack), and settings pages test shared multi-page layouts and cross-page data flow.

The difficulty is not the complexity of any single page, but the site’s overall integrity and consistency—precisely the weakest link in today’s AI coding assistants. Single-page code generation is already a mature capability; what is genuinely hard is keeping routing, state, and styling globally consistent across a long context. In other words, this benchmark measures not “code completion ability” but “software planning ability.”

2. Designing an Evaluation That Produces Comparable Results

For evaluation results to be trustworthy and reproducible, you must control variables—otherwise every comparison is an apples-to-oranges exercise.

1. A unified prompt. Every tool receives exactly the same task description, with no ad-hoc additions mid-run:

Using a mainstream component-based frontend framework, build a frontend
shopping site from scratch. It must contain the following 10 pages:
home (product list), product detail, purchase, purchase success,
user login, user registration, user center dashboard, orders,
wallet (coupon pack), and settings. Requirements: runnable, complete
end-to-end flow, basic form validation, mobile responsive.

2. Explicit graded acceptance criteria. Break the vague notion of “it works” into four levels:

• L0 Runnable: the build passes with no blocking errors;
• L1 Complete flow: all 10 pages are reachable and the main transaction path works end to end;
• L2 Usable experience: form validation, responsive adaptation, and a unified visual style;
• L3 Maintainable code: clear structure, decoupled data and views, no empty placeholder directories.

3. Lock and record the interaction mode. Mainstream AI coding assistants generally offer multiple modes, such as conversational and agent-style autonomous execution. Mode differences can affect results as much as differences between the underlying models. The mode must be fixed during evaluation, or the conclusions cannot be attributed to any specific cause.

3. Five Core Evaluation Dimensions

3.1 First-Pass Success Rate and Repair Cost

Record the total number of rounds from submitting the task to obtaining a runnable project. In practice, the most common outcome is a project that “runs but comes with errors,” requiring multiple rounds of error–fix–error cycles. Repair rounds are a hard metric of a tool’s engineering maturity—they map directly to human cost in real development. Some tools converge in two or three rounds; others are still failing after five or six and have to be abandoned. That is where the real gap lies.

3.2 Site Integrity and Consistency

• Are all 10 pages generated, and do the routes interconnect?
• Is the visual style unified? Tests frequently produce a site with fragmented color schemes, sometimes with individual pages that “don’t look like they belong to the same project.”
• Is the shared layout of the user center (such as sidebar navigation) actually reused, rather than copy-pasted into every page?

3.3 Interaction Quality

• Form validation: do login and registration check required fields and input formats, and are the error messages user-friendly?
• Responsiveness: is the site usable on mobile? Watch for contrast accidents such as a button the same color as its background—“white on white” text that simply cannot be seen.
• Information density: many generated product detail pages are too thin, with only a title and a price—clearly short of “usable.”

3.4 Code Quality

• Are styles and logic separated, or are hundreds of lines of inline styles mashed together?
• Is the component breakdown reasonable, or are there thousand-line “god components”?
• Is mock data coupled with business code? Data hardcoded into pages means site-wide rework when real APIs are connected later.
• Do “formally complete but hollow” skeletons exist—directories reserved for hooks or global state that turn out to contain empty files? These bare-shell deliverables are the easiest way to mislead whoever inherits the project next.

3.5 Engineering Efficiency

• Generation speed and stability: does the tool hang for long stretches without responding?
• Context management: after multiple rounds of edits, does the model still remember the conventions established earlier?
• Does the structure stay disciplined as scale grows? Tests have produced folders nested inside identically named folders, and path errors severe enough to prevent the project from starting at all.

4. Typical Failure Patterns Observed in Practice—and Countermeasures

Distilling the problems seen across multiple runs yields several high-frequency failure patterns, and these are exactly the entry points for prompt optimization.

Pattern 1: Placeholder resources. Every image on the site is a placeholder, so the deliverable “looks like a website but works like a wireframe.” Countermeasure: specify the resource strategy in the prompt. Require a unified image placeholder component, so that swapping in real assets later is a change in one place only.

Pattern 2: Missing design system. Each page goes its own way, with colors left to the model’s “random improvisation” and no guaranteed floor on visual quality. Countermeasure: have the model produce a set of design tokens (color palette, type scale, spacing) first, then require every page to reference the same variables file.

Pattern 3: Broken state flow. After a successful login, the user center cannot retrieve user information; the purchase success page has no idea what was bought. Countermeasure: explicitly provide the data model and the fields that must be passed between pages in the prompt.

Pattern 4: Structure drift. As the number of pages grows, directory conventions break down, producing duplicate nesting and naming chaos. Countermeasure: lock the directory structure at the start of the task, and require the model to declare the target path before creating any new file.

5. A Reusable Structured Prompt Template

[Role] You are a senior frontend engineer.
[Task] Using a mainstream component-based frontend framework, implement
a frontend shopping site from scratch, 10 pages in total.
[Directory convention] (locked; declare the path before adding any file)
  src/
    pages/       # the 10 page components
    components/  # shared components
    layouts/     # shared layouts (e.g., the user center sidebar)
    hooks/       # reusable custom logic, no empty files
    store/       # global state such as user and shopping cart
    styles/      # design tokens and global styles
    mock/        # centralized mock data, never inlined into pages
[Data model]
  Product(id, title, price, images, desc, stock)
  User(id, name, orders[], coupons[])
[Hard requirements]
  1. Mobile responsive across the whole site, with adequate
     foreground/background contrast;
  2. Login, registration, and settings pages include form validation
     and error messages;
  3. All images are rendered through the placeholder component so they
     can be replaced in one place;
  4. After completing each page, self-check against the route list.

The key to this template is not “how many requirements you write,” but front-loading the acceptance criteria as constraints the model can self-check—so problems are intercepted during generation instead of erupting during manual acceptance.

6. How to Run the Repair Loop Correctly

When a deliverable arrives with errors, the low-efficiency approach is to paste the error back and say “fix it.” More effective strategies:

1. First ask the tool to output a “project health report”: a list of all errors, suspicious files, and dependency relationships.
2. Sort issues by how severely they block progress, fix one problem at a time, and verify immediately after each fix to avoid cross-interference between multiple issues.
3. For modules that resist repeated fixes, decisively isolate and rewrite them rather than keep patching.
4. Preserve a runnable minimal version (for example, cut the wallet page first), then add features back step by step—avoiding an all-or-nothing deadlock.

7. Rational Tool Selection: Build Your Own Evaluation Set

A single demo or a single video is not enough to support a tooling decision. Teams should establish their own benchmark: pick two or three representative real tasks (full-site generation, legacy code refactoring, defect localization), score them against unified criteria, and retest once per quarter. Meanwhile, data security and compliance are often the real prerequisites for team adoption—for example, whether a privacy mode is offered, and whether code data is used for model training—and these belong on the scorecard as well.

At the industry level, the AI coding assistant ecosystem in China has advanced significantly in recent years. Services such as Baidu AI Cloud continue to evolve in code generation, project-level understanding, and multi-mode collaboration, and generally carry natural advantages in local access and compliance within that market. Developers can absolutely build their own evaluation yardsticks using the methods above, rather than relying on vendor marketing or one-off live demos.

Conclusion

The real gap between AI coding assistants is not “can it write a page” but “can it deliver a site.” Use a complete business flow as the touchstone and quantifiable dimensions as the ruler, and tools that “all seem usable” will separate into genuinely different tiers. More importantly, the evaluation process itself is how a team distills its engineering conventions into prompt assets—and no matter how the underlying models iterate, that asset will keep compounding in value.