From Code Generation to Multi-Agent Collaboration: An Engineering Evaluation Guide
The practical value of large language models is shifting from answering questions to completing verifiable, end-to-end tasks. This guide presents an evaluation framework for models, AI agents, and multi-agent systems using a financial reporting monitoring application as a running example. It covers long-context use, code delivery, iterative execution, collaboration boundaries, cost measurement, open-model deployment, security, and acceptance criteria.
The practical value of large language models is moving beyond whether they can answer questions. For developers, generating an attractive page is useful, but delivery efficiency depends more on whether a model can understand an existing codebase, modify multiple files, run tests, locate errors, and complete an acceptable deliverable within budget and permission limits.
For this reason, evaluations of new models and their available capabilities should not focus only on launch announcements, leaderboards, or subjective impressions. A more useful approach is to convert expectations into verifiable technical questions:
- Is long context being used effectively?
- Does generated code implement a complete business workflow?
- Can an agent establish a reliable execution loop?
- Does multi-agent collaboration produce a measurable net benefit?
- Can total cost support sustained use?
This guide develops an evaluation method for research trials, technology selection, and enterprise pilots, using a financial reporting monitoring application as the running example.
1. Define What Is Being Evaluated: Models, Agents, and Delivery Systems Are Different
The capability users experience usually comes from several layers and should not be attributed entirely to the underlying model.
The first layer is the model’s basic capability, including text understanding, reasoning, code generation, and structured communication. It determines whether the model can understand requirements, identify code logic, and propose reasonable changes.
The second layer is the agent execution framework, which includes task planning, tool use, context management, failure recovery, and process logging. The same model can produce significantly different results when connected to different execution frameworks.
The third layer is the runtime environment. This includes file permissions, dependency installation, test infrastructure, network policies, and compute resources. Even a correct modification may be impossible to verify if the environment does not support it.
The fourth layer is the product’s operational constraints, such as request frequency, concurrency limits, context limits, task timeouts, and usage policies. These factors determine whether a capability can be used consistently over time.
Evaluation reports should therefore distinguish among:
- The model did not propose a correct solution.
- The execution framework did not complete the operation.
- The environment did not allow verification.
Without this distinction, teams may overestimate a system or incorrectly reject a capability that is effective under suitable conditions.
2. Long-Context Capability: The Goal Is Accurate Retrieval and Use
Long inputs can contain requirements, historical records, financial reports, and source code. However, input capacity is only a baseline condition. It does not guarantee effective understanding.
2.1 Check Cross-Document Consistency
Financial reporting analysis often involves multiple reports and different accounting scopes. One document may disclose quarterly revenue, another cumulative annual revenue, and another adjusted operating metrics. The model must distinguish reporting periods, currencies, units, and metric definitions. Similar names alone are not enough to justify comparison.
Useful evaluation tasks include:
- Extract revenue, profit, and cash flow data for a specified period.
- Preserve the document name and precise location for every data point.
- Record the original unit and the converted unit.
- Explain differences between values reported in different materials.
- Clearly identify uncertain definitions instead of filling them in automatically.
These tasks test evidence integration rather than simple summarization.
2.2 Use Traceable Structures to Reduce Error Propagation
A recommended analysis chain is:
Raw materials → Evidence snippets → Structured fields → Deterministic calculations → Analytical conclusions
Each step should retain its relationship to the previous one. For example, a program can calculate a growth rate from confirmed data, while the model explains the result and its applicable scope. This reduces the risk of asking the model to perform numerical reasoning while simultaneously generating prose.
2.3 Do Not Assume That One Large Input Is Always Best
When materials become large, teams can first build a catalog and retrieval index, extract relevant sections for each task, and return to the source when necessary. Broader context can then be used for issues that require full-document verification.
This layered approach cannot always replace long context, but it is often easier to control for cost, trace evidence, and locate omissions. The right choice depends on whether the task requires global relationships and whether retrieval might miss important conditions.
3. Coding Capability: From “Page Complete” to “Business Workflow Complete”
Generating a visually complete interface and delivering a usable application are different achievements.
A financial reporting monitoring application might include a company list, date filters, metric cards, and charts. Yet these features may rely only on static sample data. Without persistence, update jobs, and error handling, the result should not be considered a complete business system.
3.1 Define the Delivery Boundary Before Development
A verifiable minimum version might include:
- Users can maintain a list of monitored entities.
- The system can enter or import data within the authorized scope.
- Data is validated by field and persisted.
- The interface can filter data by reporting period and entity.
- Failed updates display an error state.
- Persisted data remains available after the application restarts.
If the immediate goal is only an interactive prototype, state clearly that it uses sample data and does not include a real update workflow. A clear boundary is more valuable than superficial completeness.
3.2 Test Both Functional and Failure Paths
In addition to the normal workflow, evaluate at least the following cases:
- Does the system incorrectly display empty data when the source is temporarily unavailable?
- Does importing the same record twice create duplicate statistics?
- Does the system reject a calculation or request conversion when units differ?
- Are missing fields incorrectly replaced with zero?
- Can date boundaries cause records to be omitted?
- Can an interrupted update be safely retried?
These issues are difficult to detect from screenshots but directly determine whether the system is ready for use.
3.3 Do Not Measure Contribution by the Size of the Change
More generated files or more lines of code do not necessarily mean higher quality. In an existing project, excessive refactoring can introduce greater maintenance costs than the original issue.
A better standard is to require changes proportional to the request, preserve existing behavior, explain the necessity of new dependencies, and provide reproducible verification steps. For localized defects, prefer the smallest change that can be explained, tested, and rolled back.
4. Long-Running Tasks: Feedback Must Come from Execution Evidence
Agents are suitable for multi-step work, but prolonged execution does not necessarily mean continuous progress. Reliable iteration should form the following loop:
Understand the goal → Create a plan → Execute actions → Collect results → Identify gaps → Adjust the approach
Result collection is the critical part. Test output, process exit status, runtime logs, interface interaction records, and data validation results are more persuasive than an agent’s statement that a task is complete.
4.1 Break Large Tasks into Independently Acceptable Stages
A financial reporting monitoring application can be divided into:
- Define the data structure and metric definitions.
- Implement data entry and validation.
- Implement querying and presentation.
- Add update scheduling and failure handling.
- Run tests and prepare delivery documentation.
Each stage should preserve a change summary, verification results, and known issues. If execution is interrupted, the work can resume from the latest valid checkpoint.
4.2 Set Boundaries for Automatic Repair
An agent may repeatedly attempt the same fix or bypass a failure by weakening tests. Define explicit stopping conditions:
- Reanalyze the cause when the same type of error appears repeatedly.
- Request confirmation before broad refactoring.
- Do not delete critical tests to produce a passing result.
- Do not perform destructive operations without authorization.
- After the time or cost budget is reached, report the current state and blocking reasons.
High-quality automation must know not only what to do next, but also when it should stop.
4.3 Separate Verification from Generation When Appropriate
A separate verification phase can reread the requirements and inspect the resulting changes instead of accepting the development phase’s summary. However, independent verification can also miss issues and cannot replace executable tests or human review.
For important business workflows, acceptance should still be based on explicit behavioral requirements, test results, and necessary human confirmation.
5. Multi-Agent Collaboration: Design Boundaries Before Parallelization
Multi-agent systems are useful for work that can be decomposed and recombined, such as reading multiple reports in parallel, reviewing separate modules, or running different categories of tests.
Collaboration also introduces costs. Task assignment, information sharing, conflict resolution, and result integration all consume resources. When a task is small or requires frequent synchronization, a single agent may be more efficient.
5.1 Assign Explicit Responsibilities
For financial reporting analysis, responsibilities might include material extraction, definition checking, anomaly detection, and result consolidation. Each agent must know its inputs, outputs, and when to report uncertainty.
Instead of asking agents to “analyze these materials,” use a precise instruction such as:
“Extract operating cash flow for the specified period and preserve the source location. If it is not disclosed, mark it as missing and do not estimate it.”
5.2 Standardize the Output Format
Parallel tasks can use shared fields:
- Analysis subject
- Reporting period
- Metric name
- Original value
- Unit
- Source location
- Uncertainty notes
A common structure reduces integration effort and makes it easier to detect duplicate records, missing fields, and unit conflicts programmatically.
5.3 Prevent Shared Assumptions from Creating Consistent Errors
Multiple agents producing the same answer does not make that answer reliable. If they rely on the same incorrect source or definition, the error may be amplified consistently.
Cross-checking should therefore use different evidence or verification methods where possible. For example, one stage can extract data, another can check the source text, and a deterministic program can perform the calculation. Simply asking several agents to generate the same conclusion and then voting is weaker.
6. Cost Evaluation: Measure the Cost per Successful Task
Agents can generate multiple reasoning steps, tool calls, and retries. A low per-call price does not necessarily mean low delivery cost, and generous usage limits do not automatically produce higher productivity.
A useful calculation is:
Cost per successful delivery = (model usage cost + execution environment cost + human review cost + rework cost) ÷ accepted tasks
Evaluations should also record task completion rate, total elapsed time, number of human interventions, retry count, and defects discovered after delivery.
It is particularly important to distinguish test conditions from normal operating conditions. Experiences based on temporary accounts, short-term offers, or special resource configurations should not be treated as long-term cost estimates. Enterprise evaluations should also confirm resource limits, service stability, and data-handling requirements.
7. The Value of Open Capabilities: Verify Licenses and Runtime Requirements
Available model weights do not necessarily mean that all training materials, data, or engineering components are public. They also do not imply that every usage scenario is permitted.
Teams planning self-hosting or further development should verify:
- The permitted scope of use under the license.
- Commercial-use and redistribution restrictions.
- Compute and storage requirements for inference.
- Compatibility with the existing execution framework.
- Responsibilities for logging, monitoring, and security maintenance.
- Compatibility and rollback plans for future versions.
Open capabilities can provide more deployment and integration choices, but they also create operational responsibilities. Whether self-hosting is worthwhile should be determined by data requirements, workload size, and maintenance capacity, not by the label “open” alone.
8. Design Security and Acceptance Together
Agents with file and command execution permissions should run in isolated environments with least-privilege access. They should not have default access to production credentials.
External files, reports, and web pages should be treated as data to analyze, not as instructions to execute. This helps prevent external content from inducing unauthorized operations.
A final delivery package should contain at least four categories of material:
- A requirements-to-implementation mapping
- A change summary
- Test and runtime records
- Known limitations and rollback procedures
For financial analysis, preserve data sources and calculation steps as well. Unreviewed generated content should not flow directly into important decision-making processes.
Conclusion
Whether a large language model or agent is worth adopting cannot be determined by visual quality in a single demonstration or by one benchmark score. The important questions are whether it can understand complex materials, deliver complete functionality, correct itself based on execution evidence, and remain stable within security and cost constraints.
A sound evaluation process starts by defining acceptance criteria, then designing representative tasks, recording success rates, costs, and human involvement, and expanding usage gradually. This approach turns capabilities that merely appear powerful into engineering productivity that can be measured, maintained, and used sustainably.
