Evaluating AI Coding Tools in Practice: Building an E-Commerce Site from Scratch

1. Why a “Benchmark Task”? Moving Beyond Single-Point Demos

AI coding assistants—whether AI-powered IDEs or autonomous coding agents—are proliferating. Yet most hands-on impressions stay at the level of “ask it to write a function” or “explain a stack trace.” Single-point tasks cannot reflect real productivity. In day-to-day development, the most valuable scenario is handing a tool a mid-to-large task and watching it deliver continuously. Generating a snippet tests a model’s knowledge and reasoning; delivering a project also tests planning, execution, bug-fixing, and engineering organization.

A sound benchmark task should satisfy three criteria:

1. Verifiable outcomes: The result must be obviously correct—it starts, it responds to clicks, and core flows run end-to-end.
2. Broad coverage: It should exercise routing, form validation, list rendering, detail views, state sharing, and page consistency together.
3. Engineering realism: It should involve mock-data organization, style separation, directory structure, and other real-world concerns that expose the gap between “the demo runs” and “the code is production-ready.”

“Build an e-commerce site from scratch” fits perfectly. Starting from zero code, the tool must independently handle information architecture, page implementation, data organization, and visual presentation—fully exposing gaps in completeness and consistency.

2. Benchmark Task Design: A 10-Page E-Commerce Site

Task description: Using a mainstream component-based front-end framework, build a shopping site with 10 pages: Home (product list), Product Detail, Product Purchase, Purchase Confirmation, Login, Register, User Dashboard, User Orders, User Wallet (coupons/cards), and User Settings.

A prompt can be structured like this:

Build a shopping site using a component-based front-end framework with these requirements:
1. Pages: Home (product list), Product Detail, Product Purchase, Purchase Confirmation,
   Login, Register, User Center (Dashboard / Orders / Wallet / Settings);
2. Use front-end routing with clear, named routes for navigation;
3. Login, register, and purchase forms must include basic validation (required fields, format, length);
4. All mock data goes in a standalone data-layer file—no hard-coding inside components;
5. Separate styles from logic; define global design tokens (primary color, spacing, border-radius, etc.);
6. The UI must be responsive for both desktop and mobile.

Two core evaluation points:

• Completeness: Are all 10 pages generated? Does the shopping flow (browse → detail → order → confirmation) close? Are form validations actually functional?
• Consistency: Are visual style (colors, component look), code style (naming, directory layout), and interaction patterns uniform across pages? Poor consistency means heavy rework later—costs that are routinely underestimated at first review.

For reliable conclusions, follow the “three same” rule: same task description, same prompt, same acceptance criteria, to avoid misjudgments caused by input variance.

3. Evaluation Framework: Five Core Dimensions

3.1 Interaction Mode and Autonomy

Mainstream tools offer roughly two modes: a high-autonomy fully-automatic agent that decomposes the task, runs multiple steps, and attempts self-repair; and a step-by-step conversational mode where key checkpoints require human confirmation. Some also offer a fast-generation agent that sits in between.

Three questions to answer:

1. Can the fully-automatic mode finish the whole task in one run, or does it “run but with errors”?
2. How costly is rollback and retry after a failure?
3. When multiple agents/modes are available, are the selection guidelines clear? If “choosing among modes” itself becomes a burden, the product’s mode design needs work.

Note: high autonomy is not always better. When planning capability lags behind autonomy, fully-automatic mode amplifies mistakes—messy directory structures, tangled component relationships. Step-by-step mode makes early correction easier. Evaluate “can it roll back cheaply after an error,” not simply “fully-automatic is always more advanced.”

3.2 First-Pass Completeness and Repair Cost

Track two quantitative metrics:

• First-run success rate: After the initial generation, can the project start without runtime errors?
• Repair rounds: How many conversational turns does it take to go from “has errors” to “fully usable”?

A typical finding: the first output starts but throws runtime errors, requiring repeated fixes just to see the complete result. Worse, multiple rounds of fixes still leave it broken, forcing a full restart. Repair cost is the dividing line between “looks impressive” and “actually useful.”

3.3 UI Quality and Consistency

UI is where consistency problems concentrate. Walk through five angles:

1. Layout bias: Some outputs default to mobile-first; desktop is merely “simple responsiveness,” and login/register pages visibly deform on wider screens.
2. Color and contrast: Extreme cases like white-on-white buttons—functionally present but invisible—indicate no basic usability self-check during generation.
3. Detail richness: Heavy use of placeholder images and sparse product-detail pages make the output “looks functional but feels thin.”
4. Form validation: Are required-field, format, and length checks all present for login/register forms?
5. Dual-device check: Capture screenshots on at least one desktop and one mobile width, then click through the entire shopping flow.

3.4 Code Engineering Quality

Open the generated source and inspect four things:

• Style-logic separation: Are styles independent of component logic, and ideally in separate style files?
• Data decoupling: Is mock data hard-coded inside components? Coupled data means every page must be rewritten when real APIs are plugged in.
• Architecture validity: Are reusable logic layers and global state management in place? Watch for “shell architecture”—folders exist but are empty, giving the illusion of layering without substance.
• Component reuse: Are there sensible component abstractions, or does each of the 10 pages repeat its own code?

Overly simple architecture (flat pages, no layers, no reuse) won’t break the first run but will tank future iteration speed—classic hidden technical debt.

3.5 Runnability and Service Stability

• Directory correctness: Real-world tests have seen nested folders with the same name, causing build failures and making the project unstartable.
• Service responsiveness: Does generation stall or queue up? Long waits slow the “generate → verify → fix” loop significantly.
• Fix success rate: Record cases where multiple repair rounds still produce no result—these must be documented honestly.

4. Common Issues and How to Avoid Them

Common Issue	Mitigation
All images are placeholders	Specify in the prompt: use consistent placeholders or local static assets
Contrast defects (e.g., white-on-white)	Require baseline contrast ratios (body text ≥ 4.5:1)
Desktop/mobile not responsive	Explicitly state breakpoints and layout requirements for both viewports
Mock data hard-coded in components	Require a standalone data-layer file; components import from it
Empty “shell” directories	Require every reserved directory to contain a minimal working implementation
Nested folder errors, build failures	Run a build step immediately after generation before any review
Login/register pages deformed on desktop	Add a per-page, dual-device walkthrough to the acceptance checklist

5. Turning Evaluation into a Quantified Scorecard

Suggested weighting (adjustable to team priorities):

• Task completeness (all pages, closed flow, working validation): 30%
• Runnability (first-run success rate, repair rounds): 25%
• UI quality (consistency, responsiveness, usability): 20%
• Code quality (separation, decoupling, architecture): 15%
• Efficiency & stability (response speed, queuing): 10%

Two veto items: the project cannot run, or the core shopping flow is broken. If either is true, do not adopt regardless of other scores.

6. Advanced Prompting and Collaboration Practices

1. Structured prompts: Write tech-stack constraints, page list, global design tokens, data-layer requirements, and acceptance criteria into the prompt to minimize “free interpretation” by the tool.
2. Phased generation: Produce routing and page skeletons first, confirm the structure, then deepen each page, and finally run a unified visual pass—this preserves consistency better than one-shot full code generation.
3. Post-generation manual checklist: Verify route reachability, closed core flow, dual-device responsiveness, no console errors, independent data layer, and style separation item by item—do not rely on overall impressions.
4. Check usage terms: Whether generated code may be used for model training (and if a privacy opt-out exists), free-tier limits vs. paid acceleration costs, and network accessibility—these often matter more for team tool selection than any single generation quality.

7. Conclusion

The value of a “build an e-commerce site from zero” benchmark is that it compresses tool differences into three observable signals: completeness, consistency, and repair cost. Multiple vendors, including Baidu AI Cloud, continue to refine their AI coding assistants. Developers and organizations should run their own tests with the same benchmark and scorecard so that tool-selection decisions rest on reproducible evidence, not single-task impressions. Tools will keep evolving, but the methodology—define the task first, then the evaluation dimensions, then quantify the scores—remains sound over the long term.