Evaluating AI Coding Tools: Building an E-commerce Site from Scratch

1. Why Feature Lists Aren’t Enough for Tool Selection

Over the past two years, AI coding tools have evolved from simple code completion to task-level Agents. Early tools merely suggested a few lines of code in the editor. Today’s mainstream solutions can understand complete requirements, automatically plan tasks, create files, write multi-module code, and even attempt to run and debug. Meanwhile, due to network access and compliance hurdles faced by overseas top-tier tools in certain regions, local intelligent IDEs and coding assistants have proliferated. Their marketing pitches are strikingly similar—“generate an app with a single prompt” and “fully automated development” have become standard catchphrases.

The problem is that feature lists do not reflect actual quality. Given the same prompt like “build a shopping site,” different tools can deliver drastically different results. Some might run perfectly out of the box, while others are riddled with errors. Some offer clean code layering, whereas others hardcode mock data directly into UI components. Therefore, a scientific approach requires designing a standardized benchmark task, comparing outputs using identical inputs, and reverse-engineering the differences in model capabilities and engineering maturity.

2. Evaluation Framework: Six Core Dimensions

1. Task Completion and First-Time Yield. This is a hard metric: Given a requirement, can the tool deliver a runnable result without relying on multiple rounds of human correction? If the initial output throws errors and requires constant tweaking before you see any result, it indicates a lack of self-verification and error-locating capabilities. This will significantly inflate manual costs during actual use.

2. Code Engineering Quality. Focus on four aspects: whether styling and logic are separated, whether mock data is decoupled from business code, whether the directory structure is logical, and whether it leaves room for future scaling. Poor-quality output often “runs but is unmaintainable,” making secondary development more labor-intensive than a complete rewrite.

3. Engineering Completeness. Check if forms have basic validation, if routing is complete, if page navigations form a closed loop, and if it features responsive design for mobile viewports. Poor completeness suggests the tool only did surface-level work, lacking an understanding of real-world engineering constraints.

4. Interaction and Workflow Design. Assess whether mode distinctions are clear (e.g., chat enhancement, task planning, fully automated execution), whether a privacy mode is available, and how smoothly context is managed. A user-friendly interface might seem subjective, but it directly determines daily efficiency and willingness to use the tool.

5. Response Speed and Stability. Fast generation with frequent failures offers a completely different experience from slow but stable generation. Server lag and prolonged unresponsiveness are magnified exponentially when generating large projects.

6. Cost and Data Security. Consider the availability of free tiers, whether the productivity boost justifies the paid plan, and whether code is used for model training. The ability to deploy locally or privately is crucial for enterprise users.

3. Benchmark Task: Building a 10-Page E-commerce Site from Scratch

An e-commerce site is an ideal testing scenario because it naturally covers four major page categories: list display, detail presentation, transaction flow, and account management. It also demands strict “overall consistency”—the ten pages must look like a single product, not ten isolated demos.

The reference prompt can be structured as follows:

Please complete a frontend shopping site consisting of the following 10 pages:
1. Home (Product List Page)
2. Product Detail Page
3. Product Purchase Page
4. Purchase Success Page
5. User Login Page
6. User Registration Page
7. User Center - Dashboard
8. User Center - Orders
9. User Center - Wallet (Coupons)
10. User Center - Settings
Requirements:
- Use a mainstream component-based frontend framework with complete routing for page navigation.
- Forms must include basic validation (required fields, format, length).
- Unified design specifications: primary color, spacing, and button styles must be globally consistent.
- Layout must be responsive for mobile viewports.
- Mock data should be placed in an independent data layer file, decoupled from components.

In testing, this task quickly exposes capability gaps among tools. Some deliver a fully functional site in a single execution; others throw errors on the first run and require multiple fixes; some even change their approach midway, resulting in chaotic directory structures or projects that fail to start.

4. Common Issues and Root Cause Analysis

Comparing outputs from multiple tools reveals highly patterned issues, each corresponding to a specific capability shortfall:

• Images are all placeholders: The tool lacks asset generation or retrieval capabilities. It can only output structure, not content, heavily impacting prototype demonstration scenarios.
• Mock data is coupled inside components: The model lacks architectural awareness, failing to separate the data layer, presentation layer, and logic layer. This high migration cost is a classic signal of insufficient engineering maturity.
• No mobile responsiveness, login page layout breaks: The tool did not inject responsive constraints during generation, only making it “look right” on desktop viewports.
• White text on white background, chaotic color schemes: The tool lacks a global design system concept, generating styles for elements independently without unified constraints.
• Nested same-name directories, project fails to start: This is a flaw in file operation and task planning. The Agent loses path management during execution, which is a critical hard error to avoid.
• Reserved directories for hooks and global state are empty: This is “scaffolding-only” pseudo-engineering. It mimics standard project structures but fills in nothing, indicating the model relies on templates rather than truly understanding architecture.
• Multiple failed fixes: The root cause is usually the tool’s lack of a closed-loop capability (“run -> read error -> locate -> modify”), resorting to blind trial and error.

These root causes are more valuable than a vague “easy to use” feeling. They tell you whether a tool’s shortfall is a model planning issue (hard to fix via prompts) or an engineering detail issue (can be improved via constraints and workflows).

5. Five Practices to Improve Output Quality

1. Replace one-sentence prompts with structured requirements. Explicitly write page lists, acceptance criteria, and technical constraints into the prompt. This effectively does part of the requirements analysis for the tool, significantly boosting the first-time yield.

2. Deliver in phases rather than all at once. Have the tool output the project skeleton and routing structure first. Once confirmed, fill in modules incrementally. Manual confirmation at each stage minimizes late-stage rework costs.

3. Pre-inject engineering constraints. Provide the expected directory structure directly in the prompt, for example:

src/
  pages/        # Page components, divided by routes
  components/   # Reusable UI components
  services/     # API interfaces and data fetching
  store/        # Global state
  hooks/        # Custom hook functions
  styles/       # Global styles and design variables
  mock/         # Mock data (independent from components)
  utils/        # Utility functions

Outputs with explicit constraints achieve a much higher success rate in data decoupling and style separation.

4. Establish a scientific fix loop. When encountering errors, don’t just vaguely say “fix it.” Provide minimal reproduction information: the complete error stack, involved files, and expected behavior. If two or three consecutive fixes fail, roll back to the last runnable version and change your approach. Avoid spinning your wheels in the wrong direction.

5. Maintain critical human review checkpoints. AI output should pass at least three manual checks: architectural soundness (layering and dependency directions), security (input validation, sensitive data handling), and the data layer (the path to replace mock data with real APIs). These three areas are where current Agents are most likely to leave hidden traps.

6. Scenario-Based Selection Advice

Personal Prototype Validation: Prioritize tools with high first-time yield, fast response, and free tiers. Minor flaws like placeholder images are tolerable; the core goal is quick idea validation.

Team Engineering Collaboration: Code quality and structural standards should outweigh generation speed. Focus on the tool’s adherence to style separation, data decoupling, directory constraints, and whether it supports unified team configurations.

Enterprise Compliance and Private Deployment: Keeping data within the local network, audit capabilities, and integration with existing cloud toolchains are prerequisites. Local solutions generally fall into three categories: standalone intelligent IDEs, editor plugin-based coding assistants, and cloud provider-integrated intelligent development platforms. Cloud providers like Baidu AI Cloud are combining large model capabilities with cloud development toolchains, offering general capabilities such as intelligent code assistants and AI application development platforms. These are suitable for teams looking to directly connect their code assets with cloud environments (e.g., container platforms, object storage, monitoring, and alerting systems). Before deciding, request a test environment, run the benchmark task from this article, and make a comprehensive decision based on security and commercial terms.

A quick checklist: What is the first-time task completion rate? Is the code layered and decoupled? Are pages closed-loop and navigable? Is it mobile-friendly? Can it effectively self-repair errors? Are there privacy modes and compliance guarantees? Can the free quota support a full evaluation? Only if a tool passes these questions is it worth a team pilot.

Conclusion

The gap between AI coding tools lies not in whether they can “generate code,” but in how far the generated output is from being a “maintainable engineering project.” By using standardized benchmark tasks, attributing issues through structured dimensions, and constraining the process with engineering workflows, you can not only select the right tool for your team but also develop a methodology for human-AI collaboration. In an era of rapid tool iteration, the latter is the true, timeless competitive advantage.