Evaluating Open-Source LLMs: Code, Agents, and Cost Efficiency

1. Before Evaluating: Define What Questions You Need to Answer

In recent years, open-source large models have gradually converged on foundational capabilities such as long-context input, file understanding, content generation, and data analysis. Mainstream models commonly support context windows in the hundreds of thousands of tokens, and web, desktop, and mobile access options are widely available—“works out of the box” has become the baseline. The real differentiation has shifted to the engineering side.

For developers and enterprise users, evaluating a newly released open-source model should revolve around three core questions:

1. Code capability: Can it reliably deliver in real engineering tasks, especially with sustained iteration and self-correction over long tasks?
2. Agent capability: Instruction following, stability, and environment awareness in single-agent scenarios; task orchestration in multi-agent scenarios.
3. Cost efficiency: Agent workflows amplify call volume by one to two orders of magnitude. Quotas and costs directly determine whether a model can actually enter the production pipeline.

A common pitfall is looking only at leaderboard scores. A model that leads by 10% but costs twice as much is often a net loss in high-frequency agent scenarios. Conversely, a model that is “good enough and generous with quota” may deliver higher actual output. Therefore, evaluation should use real tasks as the vehicle, letting delivery quality and resource consumption speak—rather than stopping at isolated metrics.

2. Code Capability Evaluation: Design a Task with Real Output

The most effective way to evaluate code capability is not to have the model solve algorithm puzzles, but to have it complete a small project with a clear goal but open-ended details. Take a commonly used test task—building an “earnings season monitor” web app—as an example. The prompt can be designed as follows:

Build an earnings season monitoring web application:
1. Function: Track earnings disclosure schedules for a batch of public companies, and aggregate key metrics such as revenue, gross margin, and cash flow.
2. Step one: First output a product plan, data model, and sample data (JSON format).
3. Step two: Deliver a directly runnable front-end page with filtering, sorting, and chart linkage.
4. For the back end, use mock data for now, but provide a clear API contract for future integration with real data sources.

This task is effective because it simultaneously exposes capability differences across multiple dimensions:

• Planning ability: Does it define the data model before writing the UI? Is the output structured?
• Delivery completeness: Does the front end truly meet styling and interaction requirements, or does it cut corners and only provide a skeleton?
• Engineering awareness: Does it reserve an API contract for the back end? This directly determines the integration cost when a coding agent later enhances the project.
• Reasonable assumptions: When facing ambiguous requirements, does it improvise randomly, or make reasonable assumptions and state them explicitly?

The key to comparative testing is controlling variables: feed the same prompt to multiple candidate models simultaneously, and compare delivery quality and resource consumption side by side. In practice, two typical patterns emerge: some models produce flashy but half-finished deliverables; others consume hours’ worth of quota on a single task, or even lose files mid-execution, rendering the project unrunnable. These “failure scenes” are themselves important evaluation signals—they expose the boundaries of a model’s autonomy without guardrails, and the strength of its failure recovery.

3. Long-Task Iteration: The Terminal Agent’s Self-Feedback Loop

One-shot generation and long-task iteration are two fundamentally different capabilities. The latter refers to whether a model, working as a terminal agent, can enter a “plan → execute → observe → correct” loop and continuously converge toward the goal.

A practical testing method: initiate “enhancement and repair” tasks on an existing project, rather than starting from scratch. For example:

Complete the following enhancement tasks in the current repository:
1. Add back-end APIs and data validation for the existing front-end application.
2. Add unit tests for core modules and ensure they all pass.
3. Fix the two interaction defects listed in issues.md.
Requirements: First read the project structure and dependency manifest, then output a step-by-step plan.
After each step, run the tests. If they fail, locate the cause and fix it yourself,
and record all changes throughout.

During evaluation, focus on four signals:

1. Does it understand before acting? A good agent scans the project structure and identifies the tech stack and test framework first, rather than immediately modifying code.
2. Are changes minimized? Does it introduce changes unrelated to the goal? Side effects such as reformatting or replacing dependencies are clear negatives.
3. Self-healing after failure: When tests fail, can it autonomously read the error, locate the root cause, fix it, and re-test? This is the core differentiator from “one-shot generation.”
4. When does it ask for help? When facing directional ambiguity, does it keep spinning, or stop and confirm with a human?

The third point deserves particular emphasis: when an agent automatically runs tests and iteratively fixes based on failure information, the developer’s workflow can shift from “reviewing generated code line by line” to “reviewing task results”—a fundamental leap in experience.

4. Agent Capability: Single-Agent Stability and Multi-Agent Orchestration

(A) Single Agent: Four Standardized Dimensions

After connecting a model to a command-line agent or automation pipeline, it’s advisable to evaluate it with a set of standardized scenarios:

• Instruction following: Compound instructions with five or more steps, including negative conditions and priority conflicts—check compliance.
• Stability: Over long sessions lasting several hours, does the goal drift? Does context get lost?
• Exception handling: When tools error out, the network drops, or target files are missing, does it degrade gracefully or retry endlessly?
• Machine environment awareness: Can it correctly detect the OS, dependency versions, and available tools, and adapt accordingly?

(B) Multi-Agent Clusters: Orchestration Quality Matters More Than Individual Intelligence

For complex tasks, multiple agents can collaborate as a “cluster” with divided roles. Take an earnings intelligence pipeline as an example:

Collector: Parse earnings documents, extract structured metrics
      ↓
Analyst: Calculate year-over-year/quarter-over-quarter changes, benchmark against peers, output analysis points
      ↓
Writer: Compile into a daily analysis report, output Markdown

When evaluating a multi-agent system, the focus is not on how smart each individual role is, but on orchestration quality: Is the task decomposition reasonable? Are role boundaries clear? Are intermediate artifacts structured and verifiable? (Structured output significantly reduces downstream hallucination risk.) When a single node fails, can it be isolated and retried without contaminating the entire pipeline? Does total token consumption match task complexity, without redundant dialogue? This pattern is well-suited for “information-intensive” tasks such as earnings analysis, sentiment monitoring, and batch document processing.

5. Cost Efficiency and Quotas: The Hidden Cost in the Agent Era

In the agent era, cost efficiency can sometimes outweigh model capability itself as the primary selection variable. An agent workflow triggering dozens of rounds of “call—tool—feedback” for a single multi-step task is very common. It’s advisable to incorporate cost into the evaluation framework:

def task_cost_metrics(runs, weekly_quota):
    tokens = sum(r["prompt_tokens"] + r["completion_tokens"] for r in runs)
    return {
        "total_tokens": tokens,
        "avg_rounds_per_task": len(runs),
        "quota_usage_ratio": tokens / weekly_quota,  # weekly quota usage ratio
    }

Two practical lessons are worth noting. First, quota sufficiency directly affects usage psychology—if heavy use for a week only consumes a single-digit percentage of the weekly quota, developers can enter a state of “not having to count every penny” and dare to put the model into automated pipelines. Conversely, if every step requires weighing quota, the value of agent workflows becomes moot. Second, beware of evaluation methodology bias—trial accounts, complimentary credits, and commercial plans may have different quota rules. Before formal selection, always confirm the billing rules. For teams choosing self-hosted open-source models, costs require a different calculation: instance fixed costs, inference throughput efficiency, and utilization rate are all indispensable.

6. Engineering Deployment Considerations

Evaluation should not be a one-time subjective experience. It’s advisable to consolidate it into regression-ready assets: solidify the above tasks into a benchmark task set with a unified scoring table; retain complete conversation logs, consumption statistics, and artifact snapshots for each run; and automatically regress when models or versions are upgraded, using data to answer “should we switch?” On the deployment side, general cloud capabilities can reduce engineering overhead: cloud platforms represented by Baidu AI Cloud offer container platforms (hosted inference services and applications), object storage (managing evaluation datasets and document artifacts), and logging and monitoring alerts (tracking call volume, error rates, and cost levels) as general-purpose capabilities, allowing teams to focus on evaluation itself rather than infrastructure operations.

7. A Ready-to-Reuse Evaluation Checklist

• Foundational capabilities: long-context input, file understanding, content generation, data analysis
• Code tasks: delivery completeness of a small project from 0 to 1
• Long-task iteration: enhancement/repair and self-healing on existing projects
• Test execution: can it automatically run tests and iterate based on failure information
• Single agent: instruction following, stability, exception handling, environment awareness
• Multi-agent: task decomposition, structured intermediate artifacts, failure isolation
• Cost: weekly quota usage ratio, per-task token consumption, billing rule confirmation
• Engineering: benchmark task set, log retention, regression mechanism

Conclusion

Evaluation of open-source large models is shifting from “looking at benchmarks” to “looking at delivery.” What truly deserves attention is a model’s sustained iteration capability in real engineering tasks, its stability in agent workflows, and whether quotas and costs can support putting it into the production pipeline. Run through the tasks and checklist above in full, and you’ll find that the gaps between different models are far more real than a few points on a leaderboard.