Open-Source LLM Evaluation in Practice: Code Generation and Long-Horizon Agent Task Assessment

1. Why Benchmark Leads Do Not Equal Productivity Leads

Model evaluation has long relied on benchmark scores, but benchmarks measure single-turn answer accuracy. Agent-era workloads are chained: a real task strings together planning, retrieval, coding, execution, verification, and remediation—each step consuming context windows and generation quota.

A simple cost model follows:

Cost per task ≈ (interaction turns) × (tokens per turn) × (price per token) + manual rework time

If Model A scores 10% higher on a benchmark yet costs 100% more per task, it is a net regression in productivity terms. This explains recent community frustration with some leading commercial models: verbose output, rapid quota burn, marginal capability gains, and disproportionately amplified cost.

Open-source models fill this gap precisely because open weights give you deployment flexibility and a controllable cost structure. The evaluation question therefore shifts from “how high does it score?” to three practical checks:

1. Can it finish a real task end-to-end?
2. For the same quality, how many tokens, how much quota, how much manual rework?
3. On long tasks, does it self-correct, or does it drag you into a “read logs → tweak prompt → rerun” loop?

2. Evaluation Framework: Six Reusable Dimensions

Break the evaluation into six dimensions, score each 1–5 under the same task and prompt, and compare models side by side to avoid anecdotal impressions.

• Single-turn code generation quality: requirement understanding, structural completeness, UI/UX fidelity, and out-of-the-box runnability.
• Long-task self-iteration: in a CLI coding agent, watch whether the “execute → feedback → fix” loop converges automatically.
• Instruction following: adherence to multi-step and negative constraints.
• Multi-agent collaboration: task decomposition quality, information transfer loss, and result consistency.
• Cost and quota: tokens per task, rate-limit headroom, per-task cost.
• Stability and exception handling: network jitter, timeouts, dirty data, and machine-environment awareness.

Score results should be stored as structured data for long-term tracking:

{
  "task": "earnings-season monitoring app",
  "model": "model-under-test",
  "dimensions": {
    "single_turn_codegen": {"runnable": 0, "ui_fidelity": 0, "data_contract": 0},
    "long_task_iteration": {"auto_converge": 0, "test_fix_loop": 0},
    "instruction_following": {"multi_step": 0, "negative_constraint": 0},
    "swarm": {"task_success": 0, "info_coverage": 0, "transfer_loss": 0},
    "cost": {"tokens_per_task": 0, "quota_used_pct": 0},
    "stability": {"retry_on_timeout": 0, "env_awareness": 0}
  },
  "notes": "Score 1-5; cost fields filled from actual measurements"
}

3. Practice 1: Single-Turn Generation with an Earnings Monitoring App

Task background. During earnings season, analysts must track release schedules, revenue, margins, and guidance for dozens of companies. Manually monitoring spreadsheets is inefficient, so a local web app for aggregation and visualization is needed.

Test method. First ask the model to produce a product plan and sample data, then use those as input to generate the app in one shot. Prompt template:

Role: Full-stack engineer.
Task: Based on the product plan and sample data below, generate an earnings-season monitoring web app.
Requirements:
1. Deliver a standalone frontend with built-in static data in the first pass: earnings calendar, KPI cards, YoY/QoQ charts.
2. Support filtering and sorting by company and date, plus one-click refresh.
3. Frontend data structures must map one-to-one to the sample fields, leaving a clean boundary for future backend integration.
4. Output the full directory structure and local run instructions.

Acceptance criteria:

• UI/UX fidelity: does the layout match the plan, and are filter/sort interactions actually functional?
• Data contract: do frontend fields match the sample exactly—this directly drives rework when a backend is added later.
• Runnability: can the code be copied locally and run in one attempt?
• Completeness trap: some models cut corners—pages exist but components are empty shells or charts use dummy data. These must be checked item by item. Other models chase completeness with extremely long output, burning hours of quota or getting truncated and producing incomplete files that cannot run. “Quota headroom” should therefore be a hard scoring metric.

A common misconception: “frontend only in the first pass” is not a flaw. Single-turn limits and context-window constraints usually restrict the first pass to a frontend skeleton. The correct workflow is to accept the frontend contract first, then let the agent iterate on the backend over subsequent turns—this tests engineering rhythm better than demanding a full-stack delivery upfront.

Horizontal comparison. Feed the identical prompt to multiple models and record four data points uniformly: runnable (yes/no), completeness percentage, rework rounds needed, and quota consumed. Consistent recording is essential; otherwise the comparison is misleading.

4. Practice 2: Long-Task Self-Iteration with a CLI Coding Agent

Unlike the “one-shot generation” of a chat interface, a CLI coding agent operates in a local repository context. The evaluation focus shifts from single-turn quality to closed-loop convergence.

Typical test tasks (starting from an existing project exposes real capability better than starting from scratch):

• Fix legacy bugs.
• Add automated tests to an existing module and get the full suite green.
• Small feature additions, such as adding report-export capability to the monitoring app.

Watch whether an automatic loop forms:

while not at goal:
    plan   = agent plans next step based on repo context
    diff   = apply change / run command
    result = run tests and build in sandbox
    if result fails:
        agent reads error log → locates issue → fixes   # key: no human intervention
    else:
        converge and output change summary

The dividing line is self-feedback. A weak agent stops every turn for human confirmation; a strong agent treats the error log as new input and decides the next action autonomously. In testing, the ability to “run tests and keep fixing based on failures” varied dramatically—some agents stall after one failure, others chain multiple “execute → read error → fix” cycles until all tests pass, a qualitative difference never seen in one-shot generation.

Engineering red lines (follow regardless of which agent you use):

• Execute in a sandbox or isolated branch; never write directly to main.
• Every change must be auditable: output a diff summary each round.
• Use tests as guardrails: the agent may modify code but must not lower test standards (e.g., deleting assertions to “go green”).

5. Practice 3: Multi-Agent Collaboration (Swarm) for Earnings Intelligence

Single-agent tests measure individual ceiling; swarm tests organizational capability. For “tracking earnings intelligence across a major-index constituency,” split work into four roles:

• Coordinator: decompose tasks, assign work, aggregate results, arbitrate conflicts.
• Collector Agent: scrape each company’s release schedule and core metrics.
• Cleaner Agent: standardize fields, unify units, deduplicate and correct errors.
• Analyst Agent: generate summaries, flag beats and misses.

Orchestration sketch:

swarm:
  coordinator:
    duties: [task decomposition, result aggregation, conflict arbitration]
  workers:
    - id: collector
      job: earnings schedule and core metric collection
    - id: cleaner
      job: data standardization and deduplication
    - id: analyst
      job: summary generation and anomaly flagging
  bus:
    type: message queue or shared document
    schema: fixed-field contract

Four evaluation metrics to watch:

1. Task success rate: is the end-to-end output directly usable?
2. Information coverage: are any companies or metrics missing?
3. Transfer loss: what proportion of upstream fields or semantics are lost downstream?
4. End-to-end latency and total token consumption.

Common pitfalls. Upstream and downstream agents each “hallucinate completions,” amplifying errors. The same company gets processed by multiple agents, producing conflicting conclusions. Both issues require constraints at the field-contract and arbitration level—don’t rely on the model to self-police.

6. Treating Quota and Cost as Engineering Metrics

In the agent era, cost-effectiveness sometimes matters more than raw capability. Build a cost ledger per task type:

Cost per task = input tokens × input price
              + output tokens × output price
              + (self-hosted) compute amortization + ops labor

Two practical tips:

• Track quota headroom, not just usage. If routine tasks are a single-digit percentage of weekly quota, you won’t be rate-limited during production runs—critical for agents that must operate continuously.
• Calibrate your numbers. Trial or promotional quota may differ from standard plans. If you evaluated on a trial account, convert to the standard-plan equivalent before drawing cost-effectiveness conclusions, or you will systematically overestimate value.

Open weights add another option: high-frequency, high-concurrency task chains can be self-hosted, converting “per-token billing” to “compute amortization.” But honestly account for inference optimization, version upgrades, and ops labor—don’t count only electricity.

7. Stability Checklist Before Wiring Models into an Agent Framework

Before connecting a model to a persistent agent runtime, run a stress test:

Instruction following
  [ ] Give 5+ constraints at once; verify each is honored.
  [ ] Do negative constraints ("do NOT do X") get violated?
Unexpected conditions
  [ ] After an external-request timeout, does it retry or degrade gracefully?
  [ ] If a long task is truncated, can it resume rather than restart?
  [ ] With missing fields or malformed input, does it validate before acting?
Environment awareness
  [ ] Can it correctly read working directory, dependency versions, and OS info?
  [ ] Does the output stay stable when moved to a different machine?

Any failing item should be guarded at the orchestration layer (timeout fallback, input validation, idempotent retries)—don’t assume the model is always stable.

8. From Evaluation to Production: Engineering the Earnings Monitor

When evaluation results become production, a typical architecture for an earnings-monitoring app looks like this:

• Containerized web frontend and API service.
• Scheduled jobs for data refresh and anomaly scanning.
• Object storage for historical data and generated reports.
• Message queue to decouple collection from cleaning.
• Logging and alerting on task failures, quota anomalies, and API errors.

These are standard cloud building blocks—container platforms, object storage, message queues, logging and monitoring services—available from providers such as Baidu AI Cloud among others. Because the model layer is naturally decoupled from the deployment layer, the application need not be tied to any specific model, preserving flexibility to swap or mix models later.

9. Conclusion: Three Reusable Evaluation Questions

Across these practices, assessing the “highlights” of a new open-source LLM boils down to three questions:

1. Quality: On a real task, what is the first-pass completeness and the rework cost?
2. Rhythm: On long tasks, can the model self-correct and auto-converge, freeing you from repetitive loops?
3. Economics: Can per-task cost and quota headroom sustain high-frequency production use?

Benchmarks answer only half of Question 1. Questions 2 and 3 require running the model inside real workflows. The value of open source is this: once you have the answers, you also gain the option to self-host and control your own cost structure.