AI Coding Assistant Evaluation in Practice: Building an E-commerce Frontend from Scratch

1. Background: Three Forms of AI Coding Tools and Why Evaluation Matters

Over the past two years, AI coding tools have evolved rapidly into three broad categories:

1. IDE-embedded assistants: Focus on code completion, conversational Q&A, explanation, and refactoring. Developers remain in control of the coding process.
2. End-to-end agents: Offer an “auto-execute” mode inside the IDE. Given a natural language requirement, the tool autonomously plans, writes code, runs it, and fixes errors.
3. Autonomous agents: Take over entire development workflows at the task level, covering environment setup, coding, testing, and delivery.

These three forms differ significantly in maturity, and there is often a noticeable gap between marketing claims and actual delivery capability. In real-world adoption, additional constraints come into play: network accessibility (whether the tool works reliably without extra network configuration), data compliance (whether code is used for training and whether a privacy mode is available), response speed and stability, and cost structure (free tier and pricing model). Trial by impression is easily misled by a single impressive demo, which is why a reproducible, quantifiable evaluation method is essential.

This article uses one concrete task as the benchmark and breaks the evaluation process into actionable steps and scorable metrics.

2. Benchmark Task Design: Building an E-commerce Shopping Site from Zero

Task definition: Using a mainstream component-based frontend framework, build a shopping site with 10 pages from scratch (frontend only, mock data is acceptable):

• Home page (product listing)
• Product detail page
• Checkout page
• Purchase success page
• Login page
• Registration page
• User center — Dashboard
• User center — Orders
• User center — Wallet (coupons and cards)
• User center — Settings

Why an e-commerce site? Because it naturally covers the vast majority of typical frontend engineering scenarios: list rendering and pagination, forms and validation, routing and auth guards, cross-page state sharing (cart, user info), payment flow state transitions, and mobile responsiveness. A shopping site is essentially a “compressed biscuit” of frontend capabilities.

Two core difficulty criteria:

• Completeness: Are all 10 pages generated and runnable? Is the main flow (browse → login → checkout → payment success) closed-loop?
• Consistency: Is the visual style unified? Are components reused? Is state management centralized — rather than 10 pages written in 10 different ways?

To make results comparable across tools, assign weights to each dimension. For example: runnability 30%, functional completeness 25%, code quality and project structure 20%, UX details 15%, iteration and repair cost 10%.

3. Prompt Engineering: Writing Requirements “Into a Contract”

Most poor generation results stem not from the tool but from overly vague requirements. Use a structured prompt template:

[Goal] Use a mainstream component-based frontend framework to build a shopping site from 0 to 1 (frontend only, mock data is acceptable).
[Page list] 10 pages total:
1) Home (product listing) 2) Product detail 3) Checkout 4) Purchase success
5) Login 6) Registration 7) User center — Dashboard 8) Orders 9) Wallet (coupons and cards) 10) Settings
[Acceptance criteria]
- Installs and starts with a single command; no console errors.
- Main flow closed-loop: listing → detail → checkout → success page.
- Login/registration forms have basic validation (required fields, format, password confirmation).
- Unauthenticated access to user center redirects to the login page.
- Mobile-first across the site; works down to 375px width.
- Text-to-background contrast meets readability requirements.
[Engineering constraints]
- Pages, components, styles, and mock data organized in separate directories.
- Colors, spacing, and border radius unified through theme variables.
- Mock data decoupled from business logic, centralized in a single data layer.
- Routes registered centrally; user center routes protected by an auth guard.

Three key lessons:

1. Put acceptance criteria upfront: Write “runnable, closed-loop, responsive, readable” as hard requirements so the tool has clear optimization targets.
2. Make engineering constraints explicit: Directory structure, data layer location, and theme variables should be agreed upon in advance to prevent the tool from generating a chaotic structure.
3. Staged delivery beats one-shot generation: Ask for a directory skeleton and routing plan first, confirm it, then fill in pages one by one. Generating 10 pages in a single pass causes “quality collapse” in the later half with almost any tool.

4. Evaluation Dimensions in Detail

4.1 Runnability and Repair Cost

Record three numbers: whether the first generation runs directly, how many conversation rounds it takes to fix errors, and whether fixes introduce new problems. Two common failure patterns emerge in practice. One is structural failure — for example, creating a same-named subfolder inside a folder, causing module resolution to fail in a way that is hard to self-heal. The other is death by a thousand errors, where each fix surfaces a new error. If errors still do not converge after 3–4 repair rounds, the score should be significantly reduced.

4.2 Functional Completeness

Key checks: Does form validation actually work (empty values, format, consistency), or is the validation function written but never bound? Does the detail page have sufficient information density (title, price, specs, image and text description — not just a few lines)? Is the purchase flow closed-loop? Does login state actually affect routing? The most common problem is “pages exist, but logic is an empty shell.”

4.3 UX Details

Is mobile responsiveness actually effective? Login and registration pages are a high-frequency problem area at narrow widths. Is there a contrast disaster such as white text on a white background? How are image resources handled? Most tools use placeholder images everywhere, which is acceptable, but a better approach is to require a “configurable placeholder image component” in the prompt, leaving an interface for replacing images with real assets later.

4.4 Code Quality and Project Structure

Look at four things: separation of styles and logic; reasonable component granularity; whether extension points such as custom hooks and global state Context are reserved — note that reserving empty content is “fake structure,” worse than not reserving at all, and should be checked item by item during review; and whether mock data is coupled inside components. Mock data coupling is a major pain point that doubles the cost of connecting real APIs later.

4.5 Interaction Efficiency and Security Capabilities

These include response speed, visualization of the generation process, and whether a privacy mode is available (code not used for training). Also relevant is whether the tool works reliably without additional network configuration — often a prerequisite for a tool to enter a team’s daily workflow.

5. Common Failure Patterns, Root Causes, and Countermeasures

Symptom	Root Cause	Countermeasure
All images are placeholders	Tool cannot access real image hosting	Require a configurable placeholder component with image fields reserved
Login page breaks at narrow widths	Not mobile-first layout	Explicitly require “375px-first” and a unified form component
White text on white buttons	Hardcoded colors, no theme variables	Route all colors through CSS variables; run contrast self-checks before delivery
Same-named folder nested inside a folder	Directory structure left to the tool’s discretion	Provide the target directory tree in the prompt
Empty hooks/Context	Scaffolding to “look professional”	Require “implement it or delete it”; verify each item during review
Mock data scattered inside components	No data layer convention	Specify a unified data module and interface signatures

6. Engineering Best Practices

6.1 Provide the Directory Tree Before Generation

src/
├── app/            # App entry, global styles, theme variables
├── router/         # Route table and guards
├── pages/          # 10 pages, organized by module
│   ├── home/  product/  checkout/  auth/  account/
├── components/     # Cross-page reusable components (product card, forms, empty states)
├── services/       # Data layer: mock implementation shares signatures with real APIs
├── store/          # Global state (user, cart)
└── styles/         # Variables, mixins, utilities

6.2 Data Layer Abstraction: Making Mock and Real APIs Interchangeable

// services/productService.js
// Convention: pages depend only on this module, never holding mock data directly
const USE_MOCK = true;
const mockProducts = [
  { id: 'p001', title: 'Sample Product A', price: 19900, cover: null },
  { id: 'p002', title: 'Sample Product B', price: 29900, cover: null },
];
export async function fetchProducts({ page = 1, size = 20 } = {}) {
  if (USE_MOCK) {
    return { list: mockProducts, total: mockProducts.length, page, size };
  }
  // When connecting to a real API, only this part changes
  const res = await http.get('/api/products', { params: { page, size } });
  return res.data;
}

This approach quickly verifies whether the tool adhered to architectural conventions, and switching to real APIs later only requires toggling a flag.

6.3 Unified Theme Variables to Eliminate Contrast Failures

:root {
  --color-primary: #2f6bff;
  --color-text: #1f2329;
  --color-bg: #ffffff;
  --color-border: #e5e6eb;
  --radius-md: 8px;
  --space-4: 16px;
}
.btn-primary {
  background: var(--color-primary);
  color: #fff; /* White text on primary color for guaranteed contrast */
}

6.4 Delivery Acceptance Checklist

• One-command install and start; no console errors
• All 10 pages reachable; main flow closed-loop
• Form validation active; error messages readable
• No horizontal scrolling or layout breakage at 375px width
• Text contrast spot checks pass
• No structural errors such as same-named folder nesting
• Mock data centralized in the data layer; components fully decoupled
• Route guards in effect

7. Adoption Recommendations

1. Use a scoring matrix instead of impressions: Build a rating sheet based on Section 4, run the same prompt independently 2–3 times, and take the median to eliminate single-run luck.
2. Usability in your environment is a hard gate: If the tool does not work reliably without extra network configuration, lacks controllable code privacy, or suffers from unstable responses, it cannot enter daily team workflows regardless of raw capability. When evaluating infrastructure, consider the model services offered by cloud platforms targeting developers, as well as general-purpose capabilities such as object storage, containers, and continuous integration, as part of a unified assessment.
3. Clarify the human-machine division of labor: Agent mode is good for producing a “first runnable version,” while architecture design, critical component review, security, and performance remain human responsibilities. Integrate static checks and build validation into CI pipelines so machine output passes a “machine gate” first.
4. Build a prompt asset library: Turn validated prompts, directory trees, and acceptance checklists into team templates so evaluation results can be reused in real projects.

8. Conclusion

The gap between AI coding assistants is not visible in demo videos — it emerges in the process of “delivering a real project from zero to one.” Measuring tools against a fixed benchmark task, structured prompts, and quantitative scoring is the only way to move from “everything looks capable” to “we know who performs when it counts.” Even more valuable is that the directory tree, data layer conventions, and acceptance checklist produced during evaluation become the team’s engineering standards for AI-assisted development — evaluate once, benefit long term.