How to Choose an AI Coding Tool: A Hands-On E-Commerce Site Test

    This article presents a reproducible method for evaluating AI coding tools by building an e-commerce site from scratch. It covers fixed test conditions, five practical tasks, a six-dimension scorecard, a one-week POC plan, selection guidance by team size, and common evaluation mistakes.

    1. Why Selection Requires Hands-On Testing

    AI coding tools have evolved from code completion to agents that can modify multiple files. Their advertised capabilities and actual delivered results often differ, so teams need to compare tools on the same set of tasks. This article uses building an e-commerce site from scratch as the evaluation task and provides a reproducible evaluation method.

    2. Prerequisites for Evaluation

    Variables must be fixed before testing; otherwise, results from different tools cannot be compared. It is recommended to standardize the following:

    • Tech stack: Use the same frontend framework and version, and the same backend language and framework. Use the versions already used by the team’s existing project.
    • API contract: Write API documentation in advance. Define the paths, fields, and error codes for APIs such as login, registration, order list, and wallet transactions.
    • Mock data: Prepare fixed JSON samples, such as 50 orders, 3 users, and 2 coupons, so that each run uses the same input.
    • Acceptance criteria: List the must-pass functional points for each page. For example, the registration page must reject malformed phone numbers.
    • Recording method: For each generation, record the tool name and version, the original prompt, the list of generated files, test results, and the number of manual changes.

    3. Five Evaluation Tasks

    1. Purchase success page: Check page rendering, component reuse, and animation implementation.
    2. Login and registration pages: Check form validation, error messages, API integration, encrypted password transmission, and duplicate submission prevention.
    3. User center: Check data aggregation, chart component integration, and layout.
    4. Orders page: Check pagination, filtering and sorting, and the transitions among four statuses: pending payment, shipped, completed, and refunded.
    5. Wallet and coupon package: Check entity modeling, transaction consistency, and boundary conditions. For example, when the balance is zero, a debit request should be rejected.

    These tasks cover UI generation, business understanding, and engineering organization, and can distinguish the real capabilities of different tools.

    4. Six-Dimension Scorecard

    The total score is 100. Each dimension is scored against observed indicators. If any required item within a dimension fails, that dimension is marked as failed.

    Dimension Weight Observed indicators Pass criteria Test case
    Code generation quality 20 Component splitting, type annotations, empty states and error handling The registration page includes phone number validation and API failure messages, with no type errors Generate the registration page and run it
    Context and engineering understanding 20 Reuse ratio of existing components The orders page reuses the navigation component instead of implementing it again Generate the navigation component first, then generate the orders page
    Agent autonomous execution 20 Task decomposition, multi-file edits, self-test pass rate Passes self-test after one prompt, with no more than 2 manual changes See the note below
    Tech stack coverage 10 Framework, backend language, mini-program and H5 support All tech stack components currently used by the team are supported Create a new page in a mini-program project
    Data security and private deployment 15 Whether code is used for training, private knowledge base, permission audit Provides written data usage terms and supports private deployment or local indexing Review the contract and deployment documentation
    Cost and collaboration 15 Billing method, CI integration, team rule sharing Cost can be estimated from projected usage and can connect to existing code hosting and CI Calculate the three-month cost based on projected usage

    Agent test note: An example prompt is “Add filtering by payment status to the orders page and add unit tests for the filtering logic.” The expected changes are the orders page component, the filter utility function, and the corresponding test file. The login module should not be modified. The self-test command is npm test -- order. If the tool still fails the test after three repair rounds, it is judged as failed.

    The following code is pseudocode. It is only used to illustrate scoring points and cannot be run directly. isValidPhone, showToast, mapErrorMsg, api, redirect, and submitting need to be implemented according to the project.

    1. // Pseudocode: only for illustrating scoring points
    2. async function handleRegister(form) {
    3. if (!isValidPhone(form.phone)) { // Validate first
    4. return showToast('Invalid phone number');
    5. }
    6. if (submitting) return; // Prevent duplicate submission
    7. submitting = true;
    8. try {
    9. await api.register(form);
    10. redirect('/login');
    11. } catch (e) {
    12. showToast(mapErrorMsg(e.code)); // Handle errors
    13. } finally {
    14. submitting = false; // Restore state
    15. }
    16. }

    When scoring, check four points in sequence: whether validation happens first, whether duplicate submission is prevented, whether errors are handled, and whether state is restored at the end.

    5. One-Week POC Process

    Day Task Output
    Day 1 Build a unified project skeleton and import API documentation and mock data A runnable empty project
    Day 2 Complete the login and registration pages Generation records and test results
    Day 3 Complete the user center and navigation component Component reuse rate statistics
    Day 4 Complete the orders page and wallet page Status transition test results
    Day 5 Run Agent cases and fuzzy requirement cases File change list and self-test results
    Day 6 Evaluate data security and cost Contract terms summary and three-month cost estimate
    Day 7 Consolidate the scorecard and draw a conclusion Final score and pass or eliminate decision

    Elimination criteria: A total score below 60, or failure in the data security and private deployment dimension in an enterprise scenario, leads to direct elimination.

    6. Choosing by Team Size

    First answer one question: is there a requirement that data must not leave a specific region or country?

    • Yes: Evaluate only tools that support private deployment or local indexing, then follow the corresponding path below.
    • No: Go directly to the path for the relevant team size.

    The specific paths are as follows:

    • Individual developers: Focus on code quality and response speed. Use the login page and orders page for a two-day test, and choose a tool with sufficient free quota and complete support for the frontend framework you use.
    • Small and medium teams: Focus on Agent closed-loop capability and collaboration features. Run the full one-week POC, share prompt rules across the team, and retest once to compare how results change after rule reuse.
    • Large enterprises: Pass data compliance first, then compare capabilities. Confirm whether code is used for training and whether private deployment is supported. After compliance is confirmed, run the full POC, connect the internal component library and design specifications to the knowledge base index, and retest.

    7. Frequently Asked Questions

    How do I determine whether an AI coding tool is suitable for me?
    First run a POC using the e-commerce site tasks in this article, check the score against the six-dimension scorecard, and then consider the team’s tech stack.

    What should I pay attention to with private deployment?
    Confirm the hardware resources required for deployment, how model versions are updated, the log retention period, and whether administrators can view permission audit records.

    How should data security be evaluated?
    Request the data usage terms from the service provider, confirm whether code and prompts will be used for model training, and check whether data retention periods and data deletion are supported.

    What are common selection mistakes?
    First, watching only demo videos without hands-on testing. Second, using tasks that are too simple, such as generating only a button. Third, looking only at unit price without calculating actual usage. Fourth, failing to fix evaluation conditions, which makes results incomparable.

    8. Conclusion

    Some AI coding tools can already deliver usable code for standardized page development and form-based tasks. However, results vary by tool and task. Whether a tool can replace existing tools depends on data compliance requirements, how well it fits the team’s tech stack, and measured results on the team’s own tasks. By following the prerequisites and scorecard in this article, teams can reach reproducible conclusions.