How to Evaluate the Real Capabilities of Next-Generation General-Purpose Agents

Evaluating Agents: Turn Marketing Claims into Verifiable Questions

When a new AI tool is described as having “significantly improved general capabilities,” the most important question is not whether it can hold a fluent conversation, but whether it can reliably complete multi-step tasks in real work. For enterprises and developers, evaluation typically focuses on whether the agent can understand ambiguous requirements, select appropriate tools, check its own execution results, recover from errors, and how much time and cost a task requires.

Benchmarks provide a useful cross-sectional reference, but they do not directly represent real delivery capability. Test sets tend to have clear boundaries and standardized inputs, while real tasks usually involve incomplete materials, format variations, permission restrictions, and repeated revisions. A more reliable approach is to design a set of reproducible real-world tasks with explicit inputs, acceptance criteria, and permitted resources, then observe the entire process rather than only checking the final answer.

Evaluation should also distinguish three types of capability: the model’s own understanding and reasoning ability, the execution ability of external tools, and the collaboration between the model and those tools. For example, a final file may look correct either because the model generated it well or because downstream programs applied extensive corrections. Without recording the intermediate process, it is hard to know what the system actually solved.

Move from Single Demos to Complete Task Chains

One difference between an agent and a traditional question-answering system is that an agent may break a goal into steps, call external tools, and continue acting based on results. Testing therefore should not stop at asking one question and reading one response; it should cover task planning, execution, verification, and delivery.

A practical task evaluation workflow can be divided into four steps:

1. Prepare baseline tasks: Select representative work materials and record the steps, time, and common errors involved in manual processing.
2. Define acceptance criteria: Specify what the result must contain, which errors are unacceptable, and whether the original file structure or a specific format must be preserved.
3. Record the execution process: Save key operations, tool call results, human approval checkpoints, and recovery methods after failures.
4. Retest and compare: Change inputs or repeat execution to observe whether results are stable, avoiding the mistake of treating a single accidental success as general capability.

Pay particular attention to the fact that an agent may produce content that looks plausible mid-task but is actually wrong. If acceptance is based only on the visual appearance of the final page or file, issues such as calculation errors, missing fields, and unit confusion may be overlooked. Checks should cover content, format, and operation results according to the task type.

CAD Tasks: From “Explaining the Method” to “Producing an Inspectable File”

CAD processing is a typical scenario for testing an agent’s practical execution capability. In the past, a system might only explain drawing approaches or generate operating instructions; a more advanced automated workflow might attempt to read drawing information, write drawing scripts, invoke drawing programs, and export files that can be viewed. The value of this shift lies in reducing repetitive operations, but it does not mean the generated results are exempt from professional review.

When evaluating such tasks, consider the following dimensions:

• Geometric accuracy: Whether key dimensions, positional relationships, proportions, and connections meet requirements.
• Structural completeness: Whether layers, annotations, components, or other necessary objects are missing.
• File usability: Whether the file opens, displays, and remains editable normally, and whether the exported page is complete.
• Process traceability: Whether the generation logic and key parameters are clear, and whether problems can be traced to specific steps.

A safer approach is to validate the workflow first with small-scale, low-risk drawings, then gradually increase complexity. For drawings involving engineering safety or downstream production, agent-generated content should be treated as a draft awaiting review, not as a final design used without verification. Human approval should not be placed only at the start of a task: appropriate confirmation mechanisms should also be set when the system is about to overwrite original files, perform high-impact operations, or submit final deliverables.

Dynamic Pages: Verify State and Data, Not Just the Interface

Building a static page mainly tests layout and styling; building a page that updates as data changes also involves state management, component interaction, data formats, and exception handling. Even if elements such as maps, metric cards, time series, and filters look correct in a screenshot, that does not mean they work together correctly during interaction.

Acceptance for dynamic pages should therefore cover multiple states: initial load, filter changes, empty data, delayed data, and partial content load failures. Check whether the page display matches the input data, whether interactions affect the expected areas, whether data update times are clearly indicated, and whether understandable messages are shown when exceptions occur.

If the page displays real-world scenarios or operational data, pay extra attention to data sources and refresh logic. Visually “real-time” does not mean the data is truly real-time, and the page should not create an illusion of accuracy through animation or frequent changes. Evaluation should include the data update mechanism, display time, and data meaning together in the acceptance scope.

Presentations: Beyond Correct Information, Layout Remains Difficult

Generating a presentation is not simply splitting a report into several pages. Materials intended for speaking usually emphasize key points and concise text; research-report-style materials intended for reading need to carry more background, data, argumentation, and citations, placing higher demands on information organization and layout control. The two goals are different and cannot be evaluated with the same standard.

For report-style presentations, focus on checking:

• Argument structure: Whether the problem, method, evidence, and conclusion are coherent.
• Content reliability: Whether numbers, facts, and citations are traceable, and whether unverified additions are present.
• Page hierarchy: Whether titles, body text, charts, and notes have a clear visual priority.
• Text-chart alignment: Whether charts genuinely support the argument, and whether decorative charts or images disconnected from the text are present.
• Editability: Whether text and graphics can be easily adjusted during later revisions, rather than being fully flattened into images.

When the user does not provide a template or style requirements, the output may lean toward generic layout. This does not necessarily indicate weak content capability, but it does show that layout control still needs to be achieved through templates, explicit design constraints, or human polishing. If the task involves large amounts of text and complex information, it is best to confirm the outline and page plan before batch generation, avoiding discovering after a long processing run that the overall structure does not meet expectations.

Cost, Speed, and Approval Are Also Part of Capability Evaluation

Complex tasks may require multiple rounds of reasoning, file processing, and tool calls, with time and resource consumption often higher than a single ordinary Q&A. Comparing only per-call pricing or task completion speed can easily overlook the combined costs of retries, manual corrections, and failed rework.

It is advisable to record end-to-end metrics for each task, such as success rate, average completion time, number of human interventions, rework ratio, and the total cost required for the final output to meet acceptance criteria. Task budgets and time limits should also be set; when progress stalls, retries repeat, or resource consumption becomes abnormal, the system should be able to pause and explain why rather than continuing indefinitely.

When local files, software operations, data export, or other impactful actions are involved, follow the principle of least privilege. Clarify which operations can be performed automatically and which must be confirmed by a human; back up original materials and retain operation logs. Such a process reduces the risk of misoperations and makes it easier to audit and reproduce issues after they occur.

Build Your Own Evaluation Conclusion

To judge whether a general-purpose agent is worth adopting, do not only ask “Is it smarter?” Also ask “On which tasks is it stable, where does it need supervision, and is delivery quality worth the cost?” The most effective approach is to start with high-frequency, verifiable tasks where errors have controllable impact, continuously accumulate local evaluation samples, and retest regularly.

The final conclusion should be based on task results and process evidence: benchmark scores indicate potential capability, real tasks indicate fit, and stability and cost determine whether it can be used at scale. For complex file generation, dynamic application building, and high-information-density material production, agents can significantly reduce mechanical labor, but professional judgment, result verification, and risk control remain essential parts of the delivery chain.