Evaluating Open-Source LLM Agents: A Practical Framework for Long-Horizon Coding and Cost Control

Introduction

Over the past two years, more and more large models have been released with open weights. The question developers face has shifted from “which model is the strongest” to “which model is the most cost-effective and stable within my workflow.” This is especially true in agent scenarios, where a model no longer answers a single question but plans, codes, runs, reads error messages, and fixes issues continuously for hours. The evaluation dimensions change fundamentally: how polished a single generation looks is no longer the point. What determines real productivity is whether the agent can maintain direction over long tasks, self-correct along the way, and keep quota consumption under control.

This article presents a reusable evaluation framework covering four areas—long-horizon coding, local command-line agents, multi-agent clusters, and cost accounting—along with a complete case study and a verification checklist you can apply directly when selecting a model.

1. Calibrate Your Evaluation Mindset First: A 10% Benchmark Win at Double the Cost Is a Net Loss

In single-turn chat scenarios, public benchmarks are enough to compare models. In agent scenarios, a single task often involves dozens to hundreds of model calls, forming a loop of “plan → code → run → read errors → fix → run again.” This brings three changes:

1. Token consumption jumps by orders of magnitude: Compared with chat workloads, quota and unit price shift from “secondary factors” to “decisive factors.”
2. Judge the closed loop, not individual calls: A quality defect in one call may be amplified by the loop or absorbed by self-correction. What matters in the end is closed-loop capability.
3. Risk of runaway quota usage: A single task that goes off the rails can burn through the entire current quota within a few hours, interrupting the workflow outright.

This leads to a simple conclusion: if model A scores 10% higher than model B on benchmarks but consumes more than twice the quota to complete the same task, it will actually slow you down in production. The first step in evaluation is to put “capability” and “cost” into the same equation.

2. Long-Horizon Coding: Judge the Closed Loop, Not Single Shots

Anyone can produce a one-shot generation. Long tasks truly test the complete loop of “plan—execute—get feedback—iterate.” A competent coding agent should demonstrate four capabilities:

• Requirement decomposition: Before writing code, it should produce a product plan, a module breakdown, and sample data—not just dive in blindly.
• Runnable deliverables: The output should be code that runs directly, not fragments that need manual stitching.
• Self-feedback: It should proactively run the code, read errors, locate problems, and fix them in a loop, rather than handing testing off to the user.
• Context consistency: Deep into the task, it should still remember the original constraints and not silently simplify requirements.

Case Study: Building an Earnings-Season Monitoring Web App from Scratch

Use a real requirement for a side-by-side test: build a small web application that tracks earnings season, including a data dashboard, filtering, and detail interactions. Evaluate in four steps:

Step 1: Give only the requirement and see whether it first produces a credible product plan and sample data. Plan quality directly reflects how deeply the model understands the business: Is the data structure sensible? Is the page breakdown restrained? Does it confuse “monitoring” with “analysis”?

Step 2: Have it produce the frontend prototype directly. This step best reveals completeness: Does the styling match the requirements? Are the interactions usable? Are empty states and edge cases handled? Ideally, even before the backend is ready, the frontend should run independently on mock data—this leaves a clean interface boundary for later enhancements.

Step 3: Add backend requirements and check how targeted the additions are. If the artifacts from the first two steps are well structured, backend integration should follow the existing design. If the model starts over from scratch, the earlier “plan” was just for show.

Step 4: Record quota consumption and interaction rounds throughout the process, as cost-side data.

Two Typical Failure Modes in Side-by-Side Comparisons

Giving the same task to different models tends to reveal two tendencies:

• Corner-cutting: Interactions get simplified, components get omitted, and you have to repeatedly remind it to “follow the requirements.”
• Runaway consumption: The agent digs too deep into a single task, burns hours of quota in one go, and the output may end up broken to the point of not running.

The ideal behavior lies in between: small commits, each step verifiable, able to pause and resume at any time, with a restrained change scope.

3. Local Command-Line Agents: Environment Awareness and Self-Iteration

Web chat and a local command-line agent are two different experiences. The latter can sense the file system, execute commands, and read test output. Its core is a self-iteration loop:

while not task.done:
    plan   = model.plan(task, memory)     # Plan the next step using accumulated memory
    diff   = model.edit(plan, repo)       # Make restrained changes to the repo
    result = sandbox.run(diff)            # Validate in a sandbox
    if result.failed:
        memory.append(result.error)       # Record the failure info into memory
        continue                          # Proceed to the next fix round
    task.advance()

For evaluating local agents, “enhancing and fixing an existing project” is a more revealing scenario than building from scratch, because it is closer to day-to-day work:

• Does it read the code before acting, rather than making assumptions out of thin air?
• Are its changes restrained, without breaking existing logic?
• Does it proactively run tests and drive fixes from test results?
• After an error, can it close the loop itself instead of throwing the problem back at the user?

Among these, “proactively running tests” is the most telling dividing line: an agent that can discover problems on its own can genuinely self-iterate; one that leaves verification to the user is, at its core, still an advanced autocomplete.

4. Multi-Agent Clusters: Only Worth It for Parallelizable, Information-Dense Tasks

A single agent suits linear tasks; the value of a multi-agent cluster lies in parallelism and division of labor. A good test is an information-dense scenario, such as earnings intelligence gathering: distribute the extraction of earnings release dates and key financial metrics for a batch of listed companies across multiple agents running in parallel, then merge everything into a unified view.

There are four observation points:

1. Task allocation: Is the split even, or does one agent get overloaded?
2. Context sharing: How are intermediate results passed along, and are key constraints lost?
3. Merge quality: When combining results, can it deduplicate, sort, and detect conflicts?
4. Coordination cost: Does orchestration overhead cancel out the parallel gains—a cluster is not free.

If the task itself has no parallelism, forcing in a cluster only increases the failure surface and debugging difficulty.

5. Cost Accounting: Turn “Cost-Effectiveness” into Measurable Numbers

“Generous quotas at low prices” may sound like a marketing slogan, but in the agent era it is a hard metric: if quotas are too tight, people will not dare let agents run freely—which amounts to giving up the core value of the agent paradigm. Keep a simple ledger for every evaluation:

Task Type	Call Count	Token Usage	Quota Share	Human Interventions
Frontend prototype generation	…	…	…	…
Existing project fixes	…	…	…	…
Multi-agent information extraction	…	…	…	…

The ideal state: after running several medium-complexity tasks, weekly usage still stays in the single-digit percentage range. The peace of mind of “never having to check your quota” is itself part of productivity. Conversely, if a single task can eat hours of quota, then even with better output quality, you should question its sustainability in a real workflow.

6. Verification Checklist After Integrating an Orchestration Framework

After plugging the model into a local or cloud-based agent orchestration framework, verify the following item by item:

• Stability: Does it drop connections during long runs, or fall into repetitive busywork?
• Instruction following: Does it silently simplify requirements or skip specified steps?
• Handling of surprises: Can it self-heal from network interruptions, missing dependencies, or test failures?
• Environment awareness: Does it probe the runtime environment before acting, rather than assuming based on training memories?

If any of these four falls short, it will be amplified into an incident in unattended scenarios.

7. Implementation Advice: Tiered Usage and Infrastructure Choices

1. Tier your tasks. Prototype generation, large-scale refactoring, and information extraction place very different demands on a model. There is no need to chase “one model to rule them all”—matching models to task risk and complexity makes it much easier to balance capability and cost.
2. Choose deployment based on general capabilities. Whether self-hosted or cloud-managed, the core question is whether the general infrastructure—container platform, model inference service, object storage, logging, and monitoring/alerting—is complete, and whether metering is transparent. Take Baidu AI Cloud as an example: its general cloud capabilities such as containers, object storage, logging, and monitoring can support the deployment and observability needs of agent applications, and developers can choose according to their needs.
3. Roll out gradually. Run it on low-risk tasks for two weeks first, and only expand usage after you have stable cost and quality data.

Conclusion

The value of an open-source model ultimately has to be verified in your own workflow. Capability, iteration, collaboration, and cost—none of these four dimensions can be missing: a benchmark lead will not save you from runaway quota consumption, and generous quotas will not make up for missing closed-loop capability. Build a ledger with real tasks and roll out in small, measured steps—that is how “might be good” becomes “genuinely useful.”