Testing AI Agents on the Same Task: How Much Do Speed and Quality Differ?

    This article reports a hands-on test of three general-purpose AI agents on the same automotive sales data analysis task. The task covered wholesale sales rankings, popular model statistics, and a written conclusion report. It compares how quickly each agent finished, checks the accuracy of their deliverables, and explains the main sources of error, including confusion between wholesale and retail definitions. It closes with practical recommendations for developers who use agents for data analysis.

    1. Why Test Agents Hands-On

    Advances in large language models have produced a new group of general-purpose agents that can work autonomously. You give them a natural language instruction, and they break down the task, call tools, write code, process data, and produce a report on their own. This sounds promising, but vendor demonstrations alone do not show whether agents can actually replace people in real work. They need to be tested on the same tasks under the same conditions.

    This article uses a typical data analysis task. Given sales data for the automotive industry over a set period, each agent had to complete a wholesale sales ranking, count popular models, and produce a conclusion report. We chose this task for three reasons:

    1. The data structure is clear, but the definitions are complex. Sales are split into wholesale and retail. Wholesale refers to the volume that manufacturers sell to dealers, including domestic and export sales. Retail refers to the volume that dealers sell to end consumers. The task tests whether an agent understands the business meaning of these terms, not just its coding ability.
    2. The task chain is complete. It involves reading files, cleaning data, running aggregations, generating charts, and writing a report. Together, these steps cover the main capabilities of an agent.
    3. The results can be verified. The data has objective correct answers, which makes side-by-side comparison possible.

    2. Test Process: Speed Differences Were Large

    All three agents received the same prompt and the same input data. Their completion times differed significantly:

    • When the fastest agent had delivered a complete report, the other two were still printing command-line output, calling tools, and repeatedly debugging code.
    • The second-place agent delivered its report only after the fastest agent had finished about one-third of its total time.
    • The slowest agent took nearly twice as long as the fastest agent or more.

    The speed differences came from three main sources:

    1. Different planning ability. Some agents read the data structure and worked out the steps before acting, so they got it right in one pass. Others made changes while running and looped through debugging, which wasted many rounds.
    2. Different code execution strategies. Some agents tended to write a complete script in one go. Others worked in small steps and waited for feedback after each one, which naturally required more rounds.
    3. Different tool call efficiency. Frequent file reads and writes and repeated data exploration noticeably slowed the overall pace.

    However, speed is only a process metric. Delivery quality is the final metric. This test showed a serious gap between the two.

    3. Quality Review: Fast Does Not Mean Correct

    We checked each of the three deliverables one by one. The problems fell into two categories.

    1. Numerical Accuracy

    The best-performing agent got the overall totals and the general market trend mostly right. However, it made clear errors in the details. It miscalculated the wholesale figures for one major automaker, and its ranking of popular models was also inaccurate. If someone used this report for decisions, the conclusions would be misleading.

    2. Misreading of Statistical Definitions (More Serious)

    This was the most important problem in the test. In the industry context, “sales” in the prompt mainly refers to wholesale volume, meaning vehicles shipped from manufacturers to dealers, including domestic and export sales. Retail is a separate statistical system with different rules. The agents did not distinguish between them and mixed data from different definitions in their aggregations.

    As a result, the overall figures looked roughly correct, but errors in the detailed items were magnified by the mixed definitions. This kind of error is more dangerous than a simple calculation mistake. The report looks logically consistent and well formatted, so readers have difficulty spotting the problem at a glance.

    3. Other Deliverables

    In one of the other two reports, the agent dropped some records during data cleaning, so its ranking was incomplete. The other report completed its calculations, but its conclusion section stated results that did not match the data. This is a typical example of a generative hallucination.

    4. Sources of Error

    Based on these problems, errors in agent-based data analysis come from four stages:

    Stage Typical Problem Impact
    Task understanding Business terms were not clarified Wrong scope or wrong direction
    Data exploration Field meanings and missing values were not fully checked Data lost during cleaning
    Calculation logic Aggregation dimensions or filter conditions were wrong Errors in detailed figures
    Conclusion generation Written conclusions did not match the calculated results Hallucinated statements

    The first stage deserves the most attention. Agents lack a mechanism for asking the user questions. When they encounter an ambiguous term, they tend to apply their own default interpretation. These defaults usually come from general usage in training data and may not match your specific business context.

    5. Practical Recommendations for Developers

    Based on this test, we summarize four recommendations that you can apply directly:

    1. Write definitions into the prompt to avoid ambiguity. Do not assume the agent knows industry jargon. Instead of “count sales,” write “count wholesale sales, defined as the number of vehicles shipped from manufacturers to dealers, including domestic sales and exports, excluding retail.” The more precise the definition, the smaller the error.

    2. Provide reference answers for self-checking. Give the agent one or two known correct benchmark values. Ask it to recalculate the benchmarks and confirm that its definitions are correct before running the full dataset. An example structure:

    1. Task: Rank automakers by wholesale sales volume
    2. Definition: Wholesale = number of vehicles shipped from manufacturers to dealers (including exports), excluding retail
    3. Self-check: The March 2024 wholesale volume for Company X should be 123,456 vehicles.
    4. If your result deviates from this value by more than 1%, stop and check the definition.

    3. Require traceable intermediate outputs. Ask the agent to output intermediate statistical tables and key filter conditions, not just the final report. A person only needs to spot-check a few detailed items to judge whether the definitions were applied correctly.

    4. Build an evaluation that measures both speed and quality. When choosing an agent, do not look only at which one finishes first. We recommend a fixed set of benchmark tasks with standard answers. Score each agent on four dimensions: completion rate, numerical accuracy, definition correctness, and time taken. This is the only way to draw reliable conclusions.

    6. Conclusion

    The main finding of this test is clear. Current general-purpose agents are quite mature at running a complete workflow, and their speed exceeded our expectations. However, they still have clear weaknesses in understanding business meaning and in the accuracy of detailed data. Agents can take over mechanical data processing work. Tasks that require domain knowledge, such as defining metrics and verifying results, remain the responsibility of people for the foreseeable future. A more accurate way to position agents is to treat them as efficient junior data analysts, with people acting as strict technical leads. Give the agent precise requirements through well-written prompts, and back its output with spot-check processes.