How to Evaluate Open-Source LLMs: Coding Agents and Multi-Agent Collaboration

    This article provides a practical framework for evaluating open-source large language models beyond simple answer quality. It covers three evaluation layers (model, agent execution, and engineering environment), a four-level completeness check for coding tasks, metrics for long-horizon iterative work, testing discipline, multi-agent collaboration design, effective delivery cost analysis, open-source governance, and a reusable selection process for developers and enterprises.

    How to Evaluate Open-Source LLMs: Coding Agents and Multi-Agent Collaboration

    1. From “Answering Well” to “Getting Tasks Done”

    When discussing the technical value of a new generation of open-source LLMs, two misconceptions come up repeatedly: treating the quality of a single response as evidence of full engineering capability, and treating a smooth demo experience as proof that the system is usable at scale.

    For developers, the meaningful question is not whether a model can generate a snippet of code, but whether it can understand an existing project, make changes under explicit constraints, use tools to verify the results, and continue to locate and fix problems after a failure. For enterprise users, evaluation must go further, covering access control, operating cost, result traceability, and the burden of deployment and maintenance.

    The evaluation focus should therefore shift from “what the model outputs” to “what the entire system delivers.”

    Three layers must be distinguished:

    First, model-layer capability, including requirement comprehension, code reasoning, information extraction, and context utilization.

    Second, agent execution-layer capability, including task planning, tool invocation, state maintenance, failure recovery, and termination decisions.

    Third, engineering-environment capability, including file access, test execution, permission isolation, logging, and resource scheduling.

    A strong task experience is usually the combined result of all three and should not be attributed entirely to the underlying model. Conversely, a misconfigured tool or a missing environment dependency should not be judged directly as a model deficiency.

    2. Coding Capability Is About the Engineering Loop, Not Page Generation

    Take an “earnings season monitoring application” as an example. Users may want the system to support a watchlist of companies, a disclosure calendar, key metric dashboards, anomaly alerts, and historical record queries.

    A model that produces a well-laid-out, smoothly interactive page in a short time genuinely reduces prototyping cost. But if the page only uses fixed sample data, with no persistent storage, no real data integration, and no access control, it remains an interactive prototype, not a continuously running business application.

    Completeness can be evaluated across four levels.

    2.1 Presentation layer: does the page express the requirement correctly?

    Check information hierarchy, navigation, table filtering, detail expansion, and usability across screen sizes. Also cover loading, empty-data, request-failure, and invalid-input states.

    A page that only demonstrates the happy path usually overestimates actual usability. Developers should explicitly require the model to show boundary states, not just produce a polished homepage.

    2.2 Data layer: does information have a source, a structure, and constraints?

    The source, update time, and field meaning of each data category should be made explicit. For example: is a disclosure date a planned schedule or a formal announcement; what currency do amounts use; how are quarterly figures distinguished from year-to-date cumulative figures.

    Sample data is acceptable for prototype demonstrations, but it must be clearly isolated from actual production data. The mere appearance of concrete numbers on a page does not prove the system has reliable data acquisition.

    2.3 Business layer: can the rules actually execute?

    An alert feature cannot be just a toggle. It should define trigger conditions, repeat-notification policy, failure-retry behavior, and how user configuration changes take effect.

    Likewise, metric comparisons must unify period, unit, and calculation methodology. Otherwise, even without any program errors, the results may lead to misleading business conclusions.

    2.4 Operations layer: can the system be maintained over time?

    Check configuration management, protection of sensitive information, access permissions, operation logs, and fault-diagnosis methods. For data changes, backup and recovery should also be considered.

    These four levels help teams identify a common bias: the model completes a large amount of visible work while omitting the small amount of critical work that determines whether the application can actually go live.

    3. Long-Horizon Iteration: Can It Build a Verifiable Execution Loop?

    The key value of a coding agent is its ability to continuously execute an “understand–modify–verify–fix” loop, rather than generating all code in one shot.

    A reliable long-task process typically includes:

    First, inspect the project directory, existing documentation, and how the project runs, to form a basic understanding of the code structure.

    Second, break requirements into independently verifiable sub-tasks, with explicit inputs, outputs, and acceptance conditions for each step.

    Third, make localized modifications, run the relevant checks, and adjust the approach based on real errors.

    Finally, complete regression verification and summarize the scope of changes, verification results, and remaining risks.

    The critical point is not “more loops,” but whether each iteration reduces uncertainty. Repeatedly editing the same file or retrying the same command does not indicate stronger autonomy; it may mean the system lacks effective error attribution.

    Long-task performance can be measured with the following metrics:

    • Time to first effective modification: how long from receiving the task to producing a code change in the right direction.
    • Verification coverage: whether checks relevant to the changes were executed, rather than merely confirming the program starts.
    • Error recovery ability: whether the agent can adjust its path when facing missing dependencies, interface changes, or test failures.
    • Proportion of ineffective operations: the share of repeated reads, repeated edits, and pointless retries among all operations.
    • Number of human interventions: how often a developer must add judgment, correct direction, or handle risk.
    • Final delivery completeness: whether code, verification records, and a list of unfinished items are all provided.

    In practice, agents should have explicit stop conditions. For example: pause execution when the same type of error occurs repeatedly; require human confirmation before deleting data, modifying permissions, or publishing externally; submit partial results when budget or time limits are reached.

    Continuous running is not the goal. Completing tasks in a controlled manner is.

    4. Testing Capability: Distinguish “Executed” from “Verified”

    The fact that an agent can invoke a testing tool only proves it has an execution entry point. It does not prove the test conclusions are trustworthy.

    Reliable tests should correspond to requirements item by item. For the earnings monitoring application, acceptance relationships can be established as follows:

    The watchlist needs verification of adding, duplicate adding, deletion, and persistence. The disclosure calendar needs verification of date ordering, timezone handling, and missing dates. Metric display needs verification of unit conversion, null-value handling, and period consistency. The alert feature needs verification of trigger boundaries, repeated execution, and failure recovery.

    Additionally, watch out for the model modifying test standards just to make checks pass — for example, deleting failing assertions, skipping key test cases, or replacing real logic with fixed outputs. These operations may raise the apparent pass rate while degrading system quality.

    A safer practice is to separate acceptance rules from the implementation process to a reasonable degree:

    The requesting party pre-defines key test cases; the agent may add tests but must not weaken existing assertions on its own; execution output and failure records are retained; core paths receive independent review.

    Test reports should also distinguish three states: executed and passed, executed but failed, and not executed due to environment limitations. “No errors found” must not be equated with “proven correct.”

    5. Multi-Agent Collaboration: Parallelism Has Value, but More Is Not Better

    When a task can be split into relatively independent sub-problems, multi-agent collaboration can improve throughput. For example, when batch-analyzing multiple earnings reports, different execution units can handle document collection, metric extraction, anomaly checking, and result aggregation respectively.

    However, multiple agents do not automatically improve accuracy. They introduce new engineering problems: whether task boundaries are clear, whether input conventions are consistent, who arbitrates conflicting results, whether shared files may be overwritten, and how to recover when one execution unit fails.

    Tasks suited to parallelization usually share two characteristics: low inter-dependency between sub-tasks, and outputs that can be validated with unified rules. Batch document extraction, independent module review, and grouped testing are generally better suited to parallelism than tightly coupled sequential code changes.

    A reasonably robust collaboration workflow can be designed as follows:

    A task coordinator defines unified fields and acceptance rules; multiple execution units process non-overlapping data partitions; an independent validation step checks units, periods, sources, and missing values; the aggregation step only accepts results that pass validation and preserves unresolved conflicts.

    Earnings-related scenarios in particular need to preserve the correspondence of “conclusion — original text location — disclosure time.” A summary without an evidence chain, however fluent its language, is not suitable as the sole basis for important decisions.

    Note also that parallelism can reduce waiting time without necessarily reducing total cost. Task distribution, duplicated context, and cross-validation all consume resources. Both wall-clock time and total resource consumption should be recorded, rather than showcasing only speed improvements.

    6. Cost-Effectiveness Should Be Measured by Effective Delivery, Not Per-Call Price

    In agent-based tasks, a low price does not necessarily mean low cost. A system with cheap per-call pricing that frequently misses the target may require more retries and more human repair.

    A more practical formula is:

    Effective delivery cost = model invocation fees + execution environment fees + human review and repair costs + deployment and maintenance costs.

    For tasks with significant failure risk, the business loss caused by incorrect results should be assessed separately rather than hidden inside average costs.

    When evaluating subscription quotas or usage allowances, short-term experience alone is also insufficient. Test accounts, promotional arrangements, concurrency limits, and task types can all affect consumption patterns. Complete tasks should be recorded under consistent conditions, rather than extrapolating long-term cost from a handful of usage screenshots.

    It is advisable to build a fixed task suite covering small-scope fixes, medium-complexity feature development, cross-module refactoring, and long-running analysis tasks. For each task, record completion status, total time, cost, human intervention, and final quality, and re-run the suite multiple times.

    For open-source models, deployment cost must also be counted. A model being downloadable does not mean running it locally is economical. Hardware resources, inference efficiency, concurrency requirements, and operations capability all affect the actual investment.

    7. Open-Source Value: From “Accessible” to “Governable”

    When evaluating an open-source model, first confirm the specific scope of what is open: model weights, inference code, training-related code, or only part of these. Different scopes imply different degrees of reproducibility and room for secondary development.

    At the same time, review the license and related terms, clarifying commercial use, modification and redistribution, and usage restrictions. The word “open source” alone should not be taken to mean all uses are permitted.

    Enterprises should also pay attention to post-deployment governance capability, including:

    Can sensitive data remain within a controlled environment; can model and configuration versions be tracked; do tool invocations follow least-privilege principles; are execution records auditable; can regression evaluation be performed after upgrades.

    In particular, agents capable of reading files, executing programs, or connecting to external systems should run in isolated environments. Content from web pages, documents, and code comments should be treated as data to be processed, never as content that directly acquires system-instruction privileges. Operations such as sending information externally, deleting files, or modifying important configurations should require additional approval.

    8. Building a Reusable Selection Process

    For developers and enterprises, the most effective evaluation method is not to first ask “which model is the strongest,” but to first define your own tasks.

    Step one: select representative tasks that are real but controllable, with explicit inputs, constraints, and acceptance criteria.

    Step two: fix the runtime environment and tool permissions, avoiding environment differences that distort judgment.

    Step three: separately test single-shot answers, single-agent execution, and multi-agent collaboration, identifying which stage the performance gains come from.

    Step four: independently review the results, focusing on correctness, completeness, security, and maintainability.

    Step five: record full-process cost and re-test across different times and different samples, avoiding the mistake of treating occasional success as stable capability.

    Finally, set different thresholds according to business criticality. Prototype exploration can emphasize generation speed and interaction quality; production business development must emphasize regression testing, change review, and permission control; high-risk information analysis must strengthen source verification and human decision-making.

    What is worth watching in the new generation of open-source LLMs is not merely smoother code generation, longer context, or larger-scale parallelism, but whether these capabilities can be combined into a delivery process that is reliable, verifiable, and cost-controlled. Only by placing model performance inside a real engineering loop can you judge whether it is producing demo effects or continuously improving productivity.