Evaluating Open-Source LLMs in the Agent Era: A Practical Guide to Coding, Cost, and Multi-Agent Workflows

I. The Agent Era: Changing the Paradigm of Model Evaluation

The iteration speed of the open-source large language model (LLM) ecosystem is accelerating, with new weights and variants released almost every few weeks. In the past, evaluating a model primarily meant looking at leaderboard scores and single-turn dialogue quality. In the Agent era, models are increasingly embedded into automated workflows as the “brains” of intelligent agents. Consequently, evaluation dimensions have undergone a structural shift:

• Task Granularity: From “single Q&A” to “continuous task execution over hours.”
• Interaction Method: From “human-in-the-loop error correction” to “agent self-feedback and iteration.”
• Cost Structure: From “per-query mental accounting” to “batch token consumption economics.”
• Collaboration Form: From “monolithic models” to “multi-agent division of labor.”

Therefore, when facing a new open-source release, rather than fixating on a few percentage points on a leaderboard, it is more productive to focus on four production-oriented questions: Can it support long-duration code self-iteration? What is the effective output per unit cost? Can it reliably drive a multi-agent cluster? And how does it perform end-to-end in real business scenarios?

II. Coding Capability: From Single-Shot Generation to Long-Task Self-Iteration

2.1 A Real-World Requirement: Earnings Season Monitoring App

Consider a typical requirement: during earnings season, a developer needs a web application to aggregate and display the announcement cadence and key metric changes of public companies. Handing this requirement directly to a model to build the app allows us to observe the output quality from four perspectives:

1. Style Fidelity: Do the layout, color scheme, and component hierarchy match the description?
2. Interaction Rationality: Are filtering, drill-down, and refresh interactions natural?
3. Structural Extensibility: Does it reserve clear boundaries for backend integration, even if the first version only delivers the frontend?
4. Iteration Friendliness: Can a coding agent smoothly connect when subsequently filling in the backend?

In practical testing, a notable phenomenon occurs: some models “cut corners,” delivering implementations with mismatched styles or reduced functionality. Conversely, high-performing models, even if they only deliver the frontend initially, provide a structure robust enough to support subsequent incremental development. This reminds us that evaluating coding capability should not focus merely on “can it generate code,” but rather on “is the generated code evolvable.”

2.2 Methodology for Parallel Multi-Model Comparison

It is difficult to form a judgment based on a single model’s output. A recommended approach is to use the “same prompt, same sample data, fed to multiple models in parallel” methodology, recording three types of metrics: task completion rate, token consumption, and end-to-end latency. Additionally, be wary of quota risks—a single long task on certain models might consume hours of quota. Before formal adoption, test the cost curve with a small sample to avoid the scenario of “running one task and exhausting your quota.”

2.3 The Self-Iteration Loop of CLI Coding Agents

For local command-line (CLI) coding agents, the core differentiator is the capability for “self-feedback iteration.” The traditional workflow is “generate—manual review—manual feedback,” whereas an agent with self-iteration capabilities follows a closed loop:

while not task_complete:
    plan   = agent.plan(context)       # Plan the next step
    diff   = agent.execute(plan)       # Modify code
    result = test_runner.run(diff)     # Actively run tests
    if result.failed:
        context.record_failure(result) # Record failure info
        agent.reflect(context)         # Self-correct and re-plan
    else:
        task_complete = True

The most critical steps in this loop are “actively running tests” and “self-correcting based on error logs.” In practice, two distinctly different experiences can be observed: one type of tool requires a human constantly nearby to “feed” instructions; the other takes the task and autonomously runs tests, fixes errors, and continues to perform minor optimizations after the tests pass. The experience gap brought by the latter is an order of magnitude, particularly suited for scenarios involving “incremental improvements and fixes on existing projects.” Processes that previously required manual triggering can now be largely handed over to the agent autonomously.

III. Cost-Effectiveness: The Underestimated “Hidden Productivity”

In agent scenarios, even if a model leads by 10% on benchmarks, if its price is double, the net result is likely a decrease in productivity. This is because token consumption in agent workflows is tens to hundreds of times that of conversational scenarios—repeated context injection, multi-turn self-iteration, and inter-agent information passing all amplify costs.

When evaluating, a simplified formula can be introduced for horizontal comparison:

Effective Output per Unit Cost = (Task Completion Rate × Effective Workload) ÷ (Token Unit Price × Token Consumption)

There is also a “subjective but real” metric—Quota Abundance: when quotas are generous, developers confidently let agents run continuously for hours without constantly monitoring usage. When quotas are tight, users involuntarily interrupt tasks and reduce attempts, inadvertently slowing overall progress. Empirically, you can fix a set of mixed tasks (prototype building, incremental fixing, automated testing) as a benchmark—if only a single-digit percentage of the weekly quota is consumed over a week, it has generally reached the tier of “no need to worry about limits.”

For enterprise users, it is recommended to establish an internal benchmark task set: fix 10 to 20 representative tasks, periodically run them against new model versions, and track completion rates, costs, and manual intervention times to form your own “cost-effectiveness curve” rather than relying on public leaderboards.

IV. Multi-Agent Clusters: From Solo to Collaborative Operations

4.1 Capability Boundaries of Single vs. Multi-Agents

A single agent is suitable for tasks with concentrated context and linear steps, offering the advantage of lossless information and controllable behavior. A clustered multi-agent architecture begins to demonstrate value when tasks exhibit the following characteristics:

• Distributed Information Sources: Requires parallel fetching from multiple data sources.
• Heterogeneous Roles: Significant differences in prompts and context needed for extraction, analysis, and writing.
• Throughput Priority: Multiple sub-tasks have no dependencies and can proceed in parallel.

4.2 In Practice: Earnings Intelligence Pipeline

Taking “monitoring index components during earnings season” as an example, a viable multi-agent pipeline is as follows:

1. Information Collection Agent: Periodically fetches announcements and financial documents.
2. Structuring Agent: Extracts key metrics such as revenue, profit, and guidance.
3. Analysis Agent: Compares against market expectations, flagging beats and misses.
4. Aggregation Agent: Aggregates various outputs to generate a structured daily report.

A general orchestration skeleton for a cluster can be designed like this:

class Swarm:
    """General multi-agent orchestration skeleton"""
    def __init__(self, agents: list):
        self.agents = agents
    def run(self, task):
        subtasks = self.split(task)            # 1. Task decomposition
        results = [agent.execute(sub)          # 2. Dispatch and execute
                   for agent, sub in zip(self.agents, subtasks)]
        return self.merge(results)             # 3. Aggregate output

When evaluating multi-agent capabilities, focus on three signals: Stability of Instruction Following (do agents stay within their roles), Long-Context Consistency (is information lost or distorted when passed between agents), and Failure Isolation (does a single agent’s error pollute the overall result).

V. Engineering Implementation: Infrastructure and Stability Validation

To deploy the aforementioned capabilities into a production environment, supporting engineering infrastructure is required:

• Inference Services: Use container platforms to host model services, supporting elastic scaling and canary releases.
• Data Layer: Use object storage to archive unstructured documents like financial reports, and use message queues to decouple task routing among agents.
• Observability: Use logging services and monitoring alerts to track token consumption, latency, and failure rates for each task execution.
• Permissions and Auditing: Assign least-privilege credentials to agents and log all external operations.

If opting for cloud deployment, selection can follow general categories—container platforms, object storage, message queues, logging, and monitoring tools. For example, basic cloud product categories provided by Baidu AI Cloud can serve as infrastructure options for hosting agent applications. Alternatively, you can build your own clusters, with a focus on validating stability and instruction-following capabilities under long-duration tasks.

VI. Evaluation Checklist and Conclusion

When facing any new open-source LLM release, you can quickly establish a judgment using the following checklist:

Dimension	Key Questions
Coding Capability	Can it autonomously run tests and iterate based on results? Do incremental modifications break existing structures?
Cost	What is the token consumption and unit price for the benchmark task set? Can the quota support continuous autonomous running for hours?
Multi-Agent	Is role division stable in cluster mode? Is information传递失真?
Scenario Validation	Has it been tested end-to-end on your real business data?

In summary, model evaluation in the Agent era is shifting from “looking at leaderboards” to “calculating the overall ledger.” Running code self-iteration capabilities, unit-cost productivity, and multi-agent collaboration stability through your own real workflows is the most convincing evaluation. The true highlight of a new open-source release lies not in leading a single benchmark score, but in whether it can turn “letting agents work autonomously for long periods” from an occasional surprise into a reliable daily routine.