Evaluating AI Coding Assistants: A Hands-On Test From Zero to a Full E-Commerce Site

As AI coding assistants shift from simple code completion to executing entire development tasks, the key question for developers is no longer whether a tool can generate a function or a component, but whether it can deliver a complete, maintainable project. Rather than relying on third-party reviews, the most reliable approach is to run a reproducible benchmark yourself. This article walks through a practical evaluation using one consistent task — building an e-commerce site from zero — with a fixed prompt and clear acceptance criteria, and distills the process into a reusable assessment framework.

Why an E-Commerce Site Is an Ideal Benchmark for AI Coding Assistants

Generating a button or a form component is no longer a challenge for modern AI assistants. The real differentiator is the ability to deliver a complete project end to end. An e-commerce site is a particularly effective benchmark because it exercises the full range of front-end engineering challenges:

• Page scale: The task requires producing 10 pages: home (product list), product detail, purchase confirmation, payment success, login, registration, user dashboard, order list, wallet (coupons), and account settings. This tests the model’s long-range planning ability.
• Business flow: The complete journey — browse, detail, checkout, payment, order review — must work seamlessly. A single broken link makes the entire site unusable.
• State consistency: Login status, order data, and wallet balance must be shared across pages, requiring solid global state management.
• Engineering constraints: Directory structure, style organization, and the separation of data from views determine whether the codebase can be maintained.

The true difficulty of such a task lies in the completeness and consistency of the entire site. AI models excel at local generation but often struggle with global coordination. A high-quality single page does not mean the whole site is usable. Therefore, the evaluation should focus on whether the project runs, whether the user flow works, and whether another developer can take over the code — not on how elegant a single snippet looks.

Designing the Benchmark Prompt: Clarity Is Half the Battle

Testing shows that the difference in performance between prompts can be greater than the difference between products. A good benchmark prompt must include four types of information: tech stack constraints, a page list, acceptance criteria, and engineering conventions. The following template can be used directly:

Build an e-commerce shopping site using a mainstream component-based front-end framework:
1. Page list (10 pages total): Home (product list), Product Detail, Purchase Confirmation, Payment Success, Login, Registration, User Dashboard, Order List, Wallet (coupons), Account Settings.
2. Complete shopping flow: list → detail → checkout → payment success. Login state and order data must be shared globally.
3. All forms must have front-end validation: required fields, phone number format, password strength. Error messages must be displayed inline.
4. Engineering structure: organize the project into pages / components / router / store / services / mock / styles directories. All mock data must live in the mock directory and be accessed through the services layer. Hardcoding data inside components is forbidden.
5. Separate styles from logic. Centralize design variables such as primary color, spacing, and border radius.
6. Support both mobile and desktop breakpoints.

Three points deserve special attention:

1. Lock down the tech stack and target platform: Without constraints, the model may default to a mobile-first layout or introduce dependencies that were not agreed upon, leading to mismatched output.
2. Define testable acceptance criteria: Write “form validation must actually block invalid input” rather than vague phrasing like “implement good validation.”
3. Agree on the project structure upfront: Defining the directory layout and data layering in the prompt is far cheaper than refactoring later.

A Seven-Dimension Scoring System: From “Does It Run” to “Can It Be Maintained”

The following seven dimensions are recommended for evaluation:

Dimension	Weight	Core Observation Points
First-pass success and runnability	20%	Whether the first build succeeds and how many errors appear
Functional completeness	20%	Page coverage, flow closure, validation effectiveness
Visual and interaction consistency	15%	Unified design variables, responsiveness, contrast
Code quality	15%	Style separation, component granularity, naming conventions
Engineering architecture	15%	Directory clarity, data decoupling, extension points that actually work
Self-healing capability	10%	Number of fix rounds and convergence behavior
Generation efficiency and cost	5%	Total time, latency, and token consumption

Each dimension in detail:

1. First-pass success and runnability: The first gate is whether the project can build and start. In practice, “it runs but with errors” is common, indicating instability in dependency management and path resolution.
2. Functional completeness: Verify all 10 pages exist and the shopping flow works. Pay special attention to whether validation actually intercepts invalid input — a form that exists but does not validate should not count as complete.
3. Visual and interaction consistency: Check whether the entire site follows a consistent color and spacing system, whether key pages (especially login and registration) break on narrow screens, and whether there are contrast issues such as white text on a white background. Inconsistency usually means the model failed to establish global design variables.
4. Code quality: Assess whether styles are separated from logic, whether components are reasonably sized, and how much duplication exists. High-quality output reduces the cost of human handover.
5. Engineering architecture: Check if the directory structure is clear, data is decoupled from views, and whether the claimed extension points are real rather than cosmetic.
6. Self-healing capability: Feed the errors back to the assistant and observe whether it converges within a limited number of rounds. Weak repair ability turns a one-time generation into an endless debugging session.
7. Generation efficiency and cost: Measure total time, interaction rounds, waiting and lag, and whether the API quota is sufficient for daily use. A tool that is too slow to maintain flow is unlikely to be adopted by a team, even if the output quality is high.

Six Common Failure Patterns Observed in Testing

Across multiple assistants tested side by side, the following problems appeared most frequently and should be monitored closely:

1. Placeholder images and “demo-feel” interfaces: All images are placeholders, and product detail pages lack real content. A useful countermeasure is to instruct the model in the prompt to “use uniform color blocks with numbers to simulate images, wrapped in a shared placeholder component” so the visual rhythm of the page is preserved.
2. Missing responsiveness and contrast issues: The output leans mobile-first, desktop is only lightly adapted, login and registration pages break, and buttons can end up as white text on white backgrounds. The root cause is a lack of global design variables and contrast constraints. The prompt should explicitly define breakpoints and color rules.
3. Mock data coupled to components: Data is hardcoded inside components, so changing one piece of data requires editing multiple pages. The solution is to enforce a two-layer structure: a mock layer plus a service layer.
4. “Empty shell” architecture: The directory includes custom hooks, global state, or other extension points, but the files are empty. These “looks like architecture but isn’t” outputs are the most deceptive. Evaluation must sample file contents, not just directory names.
5. File system mistakes: For example, creating a subfolder with the same name inside a folder, causing path resolution failures and preventing the project from starting. Agent-style execution often lacks self-checking on file operations. A good practice is to require the assistant to print a directory tree for self-review before finishing.
6. Fixing without healing: The first generation runs but with errors, and after several rounds of fixes, it still fails or is eventually abandoned. The key is to observe whether the fix strategy targets the root cause or randomly changes approaches — the latter usually makes things worse.

Five Practices to Improve First-Pass Success

1. Deliver in phases — skeleton first, then flesh: Generate the routing skeleton and empty page layouts first, confirm the project runs, then fill in each page, and finally wire up global state. Asking for all 10 pages at once has a much higher failure rate than a three-step approach.
2. Define the data model first: Ask the assistant to produce the data structures and mock files first, review them, and only then write pages. This avoids reworking the data layer later:

// mock/products.js — data decoupled from components
export const products = [
  { id: 'p-001', title: 'Sample Product A', price: 19900, image: 'placeholder:p-001' },
  { id: 'p-002', title: 'Sample Product B', price: 29900, image: 'placeholder:p-002' },
];
// services/product.js — unified data access
import { products } from '../mock/products';
export const fetchProducts = () => Promise.resolve(products);

1. Define design variables first: Establish the primary color, spacing, and border radius in a centralized style file before writing page styles. This prevents style drift and inconsistent color choices:

src/
├─ pages/       # 10 page components
├─ components/  # shared components
├─ router/      # routing config
├─ store/       # global state (login, cart)
├─ services/    # data access layer
├─ mock/        # mock data
└─ styles/      # design variables and global styles

1. Verify after each phase: Require the assistant to run the build and static checks after every stage, resolving errors immediately rather than accumulating them until the end.
2. Write a negative list: Explicitly forbid hardcoded data, inline styles, and extra dependencies beyond those agreed upon. A clear list of what not to do is as important as the positive requirements.

Turning a Single Test into a Team Evaluation Process

A single experiment only gives a first impression. For a structured team evaluation, consider the following:

• Use a fixed benchmark task and prompt across multiple assistants, including those from vendors offering development-focused coding tools. Record the first-pass success rate, average fix rounds, total time, and latency to produce quantitative comparisons rather than subjective impressions.
• Pay special attention to the stability of agent-style execution. Most products now support breaking a natural language requirement into multi-step automated tasks. They are fast, but the project structure can easily get out of control. Always pair them with two checkpoints: a directory tree self-check and a build verification.
• Factor data compliance into the decision. Teams with strict confidentiality requirements should check whether the product offers a privacy mode and whether the code may be used for model training.
• In a second round, switch to a real business repository to evaluate partial-task capabilities: fixing a specific bug, adding tests to an existing module, or writing API documentation. This tests deep context understanding and the restraint of changes — often closer to daily development than a from-scratch build.

Conclusion

AI coding assistants have evolved from autocomplete tools to task executors, but their performance on engineering-level tasks remains uneven. Some produce polished interfaces and respond quickly but deliver poor architecture and tightly coupled data. Others have strong repair abilities but fail on file system details. The evaluation should return to engineering fundamentals: runnable, maintainable, and evolvable. Using a unified benchmark task, a quantitative scoring framework, and a phased generation strategy, teams can make rational product decisions in a fast-moving landscape — and turn AI coding assistants from a new source of uncertainty into a real productivity gain.