How to Evaluate Long-Horizon AI Agents: Real Capability and Cost-Effectiveness

From answering questions to delivering results: how agent evaluation is changing

Large models have typically been scored on knowledge Q&A, code generation, reasoning problems, and single-turn instruction following. Those tests capture local skill, but they do not answer a more practical question: when a job lasts hours or days, requirements keep being decomposed, tools are called repeatedly, code is revised many times, and a complete artifact must be delivered, does the system stay reliable?

The core value of a long-horizon agent is not how much text or code it emits in one shot. It is whether it can close a stable task loop:

1. Understand the goal: final deliverable, constraints, and acceptance criteria.
2. Decompose the work: break a large goal into executable, checkable subtasks.
3. Assign work: route tasks to different sub-agents or tools.
4. Keep executing: coding, tests, retrieval, file operations, and environment setup.
5. Detect problems: failures, conflicts, omissions, and results that miss requirements.
6. Roll back and correct: keep intermediate state and replan after errors.
7. Integrate and deliver: produce a structured result that can be reviewed and accepted.

Whether an agent is “strong” cannot be decided from a single demo of a complex task. You also need clear task boundaries, independently verified results, a defined role for human intervention, and acceptable running cost.

Why multi-day tasks are harder

A short-task failure usually hits one local step. A long-task failure may surface much later, and earlier mistakes keep propagating.

A software prototype that runs for days might follow this chain:

• requirements analysis and architecture
• filesystem, compiler, and test environment setup
• core modules
• drivers, UI, networking, or storage
• automated tests and bug fixes
• docs, install flow, and a final demo

A misjudgment at any step can put later work on a false foundation. Long-horizon execution therefore has to address four issues.

1. Context continuity

The system must know what is done, which conclusions were verified, and which work is still in progress. Chat windows alone are unreliable. Engineering systems typically need task lists, state files, change records, and structured logs.

2. State management

Repos, dependencies, test data, and running processes change. The agent must record state and resume after interruption instead of inferring everything from scratch.

3. Failure recovery

Long jobs will fail. Reliability is not “never err.” It is detecting errors, isolating impact, restoring a stable checkpoint, and re-running the affected steps.

4. Acceptance of results

“The program started” or “the page rendered” only shows a basic path works. It does not prove stability, compatibility, security, or maintainability. The final artifact must map to predefined acceptance criteria.

What public experiments say about capability bounds

One agent system published two kinds of experiments.

The first started from an empty directory, coordinated thousands of sub-agents over multiple days, and produced a bootable OS prototype: kernel, storage and keyboard drivers, networking, persistent files, windows, login UI, install flow, and built-in intelligent features.

The second used about a thousand agents and, in roughly thirty hours, formally reproduced an existing mathematical proof. Formalization turns an argument into a machine-checkable form, with each inference step confirmed by a verifier. Similar work previously took longer, suggesting multi-agent parallelism plus automatic checking can compress parts of a research workflow.

These experiments show potential to:

• hold a goal over long periods
• decompose complex work automatically
• process relatively independent subproblems in parallel
• filter results with tests or verifiers
• assemble many local outputs into one whole

They do not imply agents can take over every project. A bootable OS prototype is not broad hardware compatibility. A formally checked proof is not a new theorem. Success in a demo environment is not the same as production stability.

Treat such results as an upper bound or trend signal, not a substitute for project review.

More agents does not mean linear efficiency

More sub-agents can raise parallelism, but the gain depends on task structure.

Work that parallelizes well usually has:

• clear subtask boundaries
• explicit input/output formats
• few dependencies between subtasks
• verification by automated tests or rules
• easy isolated retries after failure

If every subtask hangs on the same unstable reading of requirements, more agents only amplify error. They may duplicate features, produce conflicting designs, or generate hard-to-merge code without shared conventions.

A reasonable collaboration layout looks like:

Controller agent
 ├─ Requirements analysis and acceptance criteria
 ├─ Architecture and task orchestration
 ├─ Implementation sub-agents
 ├─ Testing and defect-analysis sub-agents
 ├─ Security and quality-review sub-agents
 └─ Aggregation, regression, and delivery

The controller keeps global state, executors do the work, and reviewers check independently. Avoid “the implementer proving itself correct,” or tests inherit the same wrong assumptions.

How to judge real cost-effectiveness

Price cannot be measured only by API calls or output tokens. Long-horizon jobs also consume context, tools, compute, concurrency, human review time, and retry cost.

A simplified total-cost model:

Total cost =
model invocation cost
+ tool and compute cost
+ storage and logging cost
+ failure-retry cost
+ human review cost
+ maintenance and security-governance cost

Convert value into a comparable metric:

Cost per effective output =
total cost ÷ number of accepted, effective deliverables

“Effective deliverables” are not file counts. They are results that pass tests, meet requirements, and can be maintained by the team.

At minimum, record:

• elapsed time from start to delivery
• automated runtime vs. human wait time
• first-pass rate
• defect density and rework count
• share of generated code actually kept
• pass/fail per acceptance item
• security blocks or permission denials
• time to resume after interruption
• fully loaded cost per accepted feature or module

If an agent dumps a lot of code that engineers rewrite line by line, high volume is not high value. Smaller output with high test pass rates, complete docs, and low follow-on maintenance may be better for production.

Evaluate safety boundaries and access separately

Long-horizon agents can drive more tools, so risk concentrates. Cybersecurity, biological research, sensitive data, and production changes usually need stricter permissions and human approval.

Keep three things distinct:

1. Theoretical model capability: whether the model can reason and generate for the task.
2. Actual product permissions: whether the current account may use the relevant tools, data, and environments.
3. Policy limits: whether high-risk requests are blocked, degraded, or rerouted.

A high score in an internal test environment does not mean ordinary users get the same power. Organizations, account tiers, and use cases can have different access.

More conservative enterprise patterns include:

• least privilege by default
• isolation of development, test, and production
• human approval for write, delete, and publish actions
• auditing of tool calls
• limits on sensitive files and network egress
• time, budget, and call-count caps per task
• full intermediate artifacts for replay and accountability
• independent test systems for final verification

Start enterprise rollout with controllable tasks

Do not hand an agent an entire core project first. Begin with clear boundaries, limited risk, and easy verification, for example:

• generating test cases
• static analysis of existing code
• writing API docs
• log triage and incident clues
• regression after dependency upgrades
• first prototypes of internal tools
• converting existing proofs or rules into machine-checkable form

In a pilot, use a fixed task set with the same inputs, environment, and acceptance criteria across multiple runs. The point is not the best demo, but average performance, worst-case performance, and failure modes.

Grade outcomes in four levels:

• Level 1: reference only — needs heavy human rewrite
• Level 2: assistive — humans still own the core result
• Level 3: basically shippable — humans mainly review and integrate
• Level 4: automatable — stable verification, rollback, and audit

Expand only when work stays at level 3 or 4 over time and fully loaded cost beats a human-only approach.

Conclusion: judge the delivery loop, not the demo moment

The progress that matters for long-horizon agents is sustained decompose–execute–check–integrate on complex work. Multi-day OS prototypes and hour-scale formal reproductions show the unit of work moving from “an answer” to “a checkable artifact.”

Capability demos still sit far from production use. Selection should cover:

• whether the task parallelizes
• whether intermediate state can be saved and restored
• whether failures can be found and rolled back
• whether results have independent acceptance
• whether the real account has the needed permissions
• whether security policy meets organizational requirements
• whether human review and infrastructure cost are acceptable

Cost-effectiveness is not “stronger model, better value.” It is whether, inside clear boundaries, verifiable process, and reasonable permissions, cost per accepted deliverable is lower. For engineering teams, the most reliable method remains a real-task benchmark, multiple consecutive runs, recorded failure cost, and evaluation against maintainable, auditable, reusable results.