An Engineering Guide to Evaluating AI Coding: From Code Generation to Multi-Agent Collaboration
This guide explains how to evaluate intelligent coding systems by shifting focus from answer quality to delivery quality. Using a financial report monitoring application as a concrete evaluation scenario, it covers the four capability layers (generation, engineering, execution, delivery), constrained feedback loops for long-horizon agent tasks, testing across business behavior and failure recovery, the trade-offs of multi-agent collaboration, cost accounting around effective delivery, how to assess open-source models, and how to build a reproducible evaluation process.
1. Evaluating Intelligent Coding: From “Answer Quality” to “Delivery Quality”
In software development, a well-structured explanation, a visually complete page, or even a successful demo run is not enough to prove that an intelligent system can handle real engineering work.
The reason is that software delivery is continuous: requirements must be clarified, code must fit into an existing project, features must be tested, errors must be located and fixed, and the final result must satisfy deployment, security, and maintenance requirements. Failure at any stage can void the value of everything generated before it.
Therefore, evaluations of intelligent coding capability should be organized into four layers:
- Generation capability: Can it convert requirements into reasonable code and interfaces?
- Engineering capability: Can it understand an existing project, respect architectural constraints, and control the scope of changes?
- Execution capability: Can it invoke authorized tools, run tests, and fix problems based on the results?
- Delivery capability: Can it provide a reproducible way to run the result, verification evidence, and a clear statement of risks?
These four layers are not interchangeable. A polished page does not imply a reliable backend; a high number of tool calls does not imply a high task completion rate. What truly matters is whether the system, within its constraints, continuously narrows the gap between requirements and delivered results.
2. Using a Financial Report Monitoring App as a Complete Evaluation Scenario
A financial report monitoring application is well suited as a comprehensive task because it involves UI interaction, data processing, scheduled jobs, and information traceability all at once. Compared with simply asking for a display page, this kind of task is far more likely to expose weaknesses in the system’s engineering loop.
2.1 Define user goals first, instead of asking to “build an app”
The requirement can be stated as:
Users maintain a watchlist of companies, view expected disclosure dates, receive update notifications after public reports are released, and view key financial metrics together with their sources.
Next, acceptance criteria should be made explicit:
- Can the watchlist be added to, deleted from, and persisted?
- Does the timeline distinguish between expected dates and actual release dates?
- Does the page handle loading, empty-data, and request-failure states?
- Are metrics annotated with reporting period, currency, unit, and source?
- Does repeated data collection produce duplicate records?
- After a task failure, can it be retried with a traceable log?
- Can users only access data they are authorized to view?
These criteria turn a vague “looks good” into verifiable engineering requirements.
2.2 Clearly separate a frontend demo from a complete system
A page that can filter, sort, and render charts may rely entirely on built-in sample data. This is not without value: it can validate interaction flows, align product expectations, and inform subsequent API design.
But the delivery notes must clearly state:
- Which data is mock data;
- Which operations only change the current page state;
- Which features are connected to real services;
- Which modules are not yet implemented.
If there is no persistence, access control, data collection, or background scheduling, it should not be described as a complete business system. Distinguishing an “interactive prototype” from a “running service” is the first step in avoiding misjudgments of intelligent coding capability.
2.3 Data accuracy takes priority over summary fluency
Financial report data is especially prone to caliber confusion. For example, the same metric may correspond to a cumulative quarterly value, a single-quarter value, or an annual value, and it may not be directly comparable across currencies, units, or accounting standards.
A layered data-processing pipeline is recommended:
Retain raw sources → Extract fields → Validate format and caliber → Store structured data → Analyze and summarize.
Intelligent models can assist with extraction and interpretation, but key figures should be traceable back to the original source. For missing values, conflicting sources, and unverifiable calibers, uncertainty should be displayed explicitly, rather than producing conclusions that only look complete.
3. The Core of Long-Horizon Task Capability Is a Constrained Feedback Loop
The main difference between an agent and one-shot code generation is that an agent can execute multiple steps toward a goal and use execution results to correct subsequent behavior.
A reasonable development loop typically includes:
- Reading the project documentation, directory structure, and runtime constraints;
- Inspecting the current code state and existing tests;
- Breaking the goal into verifiable subtasks;
- Making clearly scoped changes;
- Running relevant checks and tests;
- Deciding whether to fix issues based on error evidence;
- Producing a change summary, verification results, and unresolved problems.
What matters is not the number of iterations, but whether each round produces valid evidence.
3.1 Do not mistake “continuous execution” for “continuous progress”
Common inefficient behaviors include: repeatedly retrying the same failing operation, expanding the modification scope without locating the root cause, weakening assertions to make tests pass, and masking exceptions by replacing real logic.
An engineered system should define stopping conditions. For example, when several consecutive rounds produce no new useful information, or when the agent is about to touch high-risk resources, it should pause and request human judgment.
Stopping conditions can be based on time budget, resource consumption, change scope, and risk level, but the specific thresholds should be configured per project rather than copied from a uniform set of values.
3.2 Use checkpoints to control context and rollback risk
Long tasks are vulnerable to context bloat. Early constraints may be forgotten, and temporary workarounds may be mistaken for final designs.
It is recommended to keep a concise state record at each stage, including:
- Current goal and completed items;
- Unresolved issues;
- Confirmed interface and data constraints;
- The most recent verification result;
- The next action and its rationale.
At the same time, keep rollbackable code checkpoints. This way, even if a task is interrupted, work can resume from a clear state instead of relying on a long conversation history to reconstruct the full context.
3.3 Verification permissions should match operational risk
Reading source code, running isolated tests, and modifying local files are usually lower-risk than changing production configuration, deleting data, or performing paid operations.
Agents should therefore follow the principle of least privilege. Operations involving credentials, production data, external publishing, or irreversible actions should require a separate authorization step. The stronger the autonomous execution capability, the clearer the permission boundaries need to be — not a default grant of full access to all resources.
4. Testing Should Cover Business Behavior, Not Just Whether Code Runs
“Started successfully” is only the most basic check. For the financial report monitoring application, testing should cover at least four levels.
4.1 Business rule tests
Verify deterministic logic such as date sorting, metric conversion, duplicate-record detection, and subscription state changes.
For financial figures, also check unit conversion, negative values, missing values, and abnormal formats, to prevent cases where the page displays correctly but the numeric meaning is wrong.
4.2 Module integration tests
Verify the interface contracts between data collection, storage, query, and notification. For example, when the same report is processed twice, the system should update the existing record or skip the duplicate, rather than sending the notification again.
4.3 User flow tests
From the user’s perspective, verify flows such as adding a watched company, viewing reports, filtering by time range, and handling failure prompts. A clickable page does not mean the business chain is complete; tests need to confirm that an operation actually produced the expected result.
4.4 Failure and recovery tests
Simulate a temporarily unavailable data source, task timeouts, incomplete responses, and service restarts, and observe whether the system can retry, degrade gracefully, and preserve state.
Test reports provided by the agent should include, as far as possible, the execution scope, results, and reasons for failures. A text statement that “testing is complete” cannot replace actual verification evidence.
5. Multi-Agent Collaboration: Parallelism Has Benefits — and Coordination Costs
Multi-agent collaboration is suitable for parallelizing work with relatively independent boundaries, but it is not simply a matter of adding more executors.
Taking financial report processing as an example, the work can be divided into:
- Source collection role: gathers and archives permitted public materials;
- Data extraction role: extracts metrics with source locations attached;
- Caliber validation role: checks reporting periods, currencies, units, and outliers;
- Analysis role: produces interpretations based on confirmed data;
- Aggregation role: merges results and flags disagreements and gaps.
The value of this division lies in clear responsibilities, not in the number of role names.
5.1 Agree on delivery contracts before running in parallel
Every subtask should define its inputs, outputs, and completion conditions. For example, an extraction result should include the metric name, value, unit, reporting period, source location, and validation status.
If outputs are free-form text, the aggregation stage has to re-interpret and reorganize everything, which easily cancels out the gains from parallelization.
5.2 Multiple identical judgments do not count as independent verification
If multiple agents use the same sources, similar prompts, and the same erroneous premise, they can consistently produce the wrong result. Majority agreement therefore does not directly prove a conclusion is correct.
A more reliable approach is to introduce different verification paths: validate units with deterministic rules, cross-check figures against the raw source, and independently recompute rates of change. “Independent” here refers to independence of evidence and method, not merely the number of execution instances.
5.3 Parallelize only when the benefit exceeds the coordination overhead
Actual elapsed time depends not only on subtask execution speed, but also on scheduling, communication, result merging, and rework.
For small changes to interdependent code, a single agent may be more efficient; for organizing many independent documents, checking multiple modules, or running different test suites, multiple agents are more likely to pay off. The collaboration mode should be decided by task dependencies, not by defaulting to cluster execution.
6. Cost Evaluation Should Center on “Effective Delivery,” Not Per-Call Price
Agent tasks often involve multiple rounds of reasoning, tool calls, and failed retries. A low per-call price does not mean a low final delivery cost, and a large available quota does not guarantee more output.
A simple accounting formula can be used:
Cost per effective delivery = total task cost ÷ number of deliveries that pass acceptance.
Total task cost should include model calls, execution environment, storage and networking, human review, and the cost of failed rework.
During evaluation, also record:
- Total elapsed time from task start to acceptance;
- Number and duration of human interventions;
- First-pass acceptance rate;
- Regression behavior after defect fixes;
- Cost variation across task sizes.
If a trial environment provides extra quota or special resource configurations, these should be distinguished from normal usage conditions. A single experience should not be used to infer long-term cost, service stability, or concurrency capability.
7. How to Objectively Assess the Value of Open Models
Model openness increases options for deployment and research, but “open” by itself does not mean production-ready.
During selection, check separately:
- Whether the open content is sufficient for the intended use;
- Whether the license permits the target business scenario;
- Whether the runtime environment meets compute and memory requirements;
- Whether suitable inference, monitoring, and operations solutions exist;
- Whether tool calling and long-horizon task workflows run reliably;
- How data security, access auditing, and update maintenance will be implemented.
It is also important to distinguish base model capability from surrounding system capability. Code modification outcomes can be influenced simultaneously by the model, prompt design, project indexing, tool permissions, and the test environment. Good performance in one integrated environment does not mean swapping in a different base model alone will reproduce the same results.
8. Building a Reproducible Evaluation Process
For development teams, a phased evaluation is recommended, rather than migrating all tasks to agents at once.
Phase one: Choose clearly scoped, low-risk tasks, such as adding tests, fixing isolated defects, and generating interactive prototypes.
Phase two: Introduce multi-file modifications within an existing project, focusing on constraint compliance, change scope control, and regression test quality.
Phase three: Evaluate long-horizon tasks, multi-agent collaboration, and external system integration, adding permission management, failure recovery, and cost limits.
Each evaluation should, as far as possible, fix the requirement description, initial project state, available tools, time budget, and acceptance criteria. For tasks with inherent randomness, run multiple observations, and avoid treating a single success or failure as stable capability.
Ultimately, what matters is not whether an intelligent system can produce an impressive first result, but whether it can continuously complete tasks under real constraints, with a traceable process, controllable cost, and verifiable quality. Shifting the center of evaluation from “how much content was generated” to “how much engineering uncertainty was reduced” is the only way to truly judge the practical value of intelligent coding and multi-agent collaboration.
