Evaluating Multi-Day Agent Delivery Capability and Cost Effectiveness

Long-horizon agents are moving from “one conversation, one answer” to splitting work across multiple days, executing it, integrating artifacts, and delivering results that can be inspected. The real question is not whether a demo looks impressive. It is whether the system can take over a real project, whether deliverables can pass acceptance, whether total cost can be accounted for, and whether safety and access boundaries are explicit. If those conditions are not recorded together, a prototype or a single benchmark score can easily overstate real usefulness.

Multi-day work is a capability divide, not the end of delivery

Public reports include two notable experiments. In one, a model starts from an empty folder, coordinates thousands of sub-agents to write, test, and review code, and eventually boots a prototype operating system on a laptop, including kernel, disk and keyboard drivers, networking, persistent file storage, a window system, a login screen, and an installer. In another, thousands of agents formalize an existing mathematical proof in about 30 hours, compared with an earlier process that took about 11 days. Formalization here means writing the proof so that every step can be checked by a computer.

These results show substantial potential in task-graph management, role specialization, artifact integration, and automated verification. Each result still answers a narrow question. A prototype OS that boots is not the same as long-term stability and compatibility. A formalized proof that passes a checker is not the same as general mathematical research ability. For selection, the conditions of success must be read together. Local success should not be extrapolated into general capability.

Evaluation framework: turn “can demo” into “can accept”

Long-horizon agent evaluation can be split into five layers:

1. Task decomposition. Can the agent break the goal into a dependency-aware task graph and define inputs, outputs, acceptance criteria, and stop conditions for each subtask? Multi-day work fails most often when progress looks real but never converges. Check whether the graph is persisted, supports checkpoints, and allows local retry after failure.

2. Execution and integration. Thousands of sub-agents do not automatically mean higher efficiency. More roles increase the need for interface contracts, conflict resolution, artifact versioning, and rollback. Evaluation should look for explicit contracts between subtasks, validators at integration time, and a final artifact that can be checked as a whole.

3. Acceptance criteria. Distinguish “boots successfully,” “proof checker passes,” “tests pass,” and “runs stably over time.” A prototype that boots only proves a functional loop in a specific environment. A proof that passes machine checking only proves formalization of a specific claim. A selection sheet should state the verification scope and what is out of scope.

4. Cost accounting. Cost effectiveness is not unit price. It is effective delivery value divided by total cost. Total cost should include model calls, tool calls, storage and network, failed retries, human review, audit trails, and opportunity cost. Multi-day work also includes state persistence, log storage, and interrupt recovery.

5. Safety and access. The same underlying capability can be scoped differently by guardrails and access rules. Record capability items, protection status, organizational eligibility, and fallback paths together. If a high-risk capability is restricted in an open version, scores from another version cannot be used as a selection basis.

A minimal evaluation pipeline can be expressed as:

def evaluate(task, agent, policy, budget):
    graph = agent.decompose(task)
    assert graph.has_acceptance_criteria()
    artifacts = []
    for step in graph.schedule():
        if not policy.allow(step):
            return blocked(step)
        result = agent.execute(step, budget)
        artifacts.append(result)
        if not verifier.check(step, result):
            retry_or_abort()
    deliverable = agent.integrate(artifacts)
    report = verifier.audit(deliverable)
    cost = budget.actual()
    return {deliverable, report, cost}

The point is not a specific API. It is a closed loop of decompose, policy check, execute, verify, integrate, audit, and bill. Missing any link distorts cost-effectiveness judgment.

Case reading: boundaries of the prototype and formalization results

The prototype OS experiment shows how far multi-day work can go: the model does more than emit code snippets; it coordinates many sub-agents through writing, testing, review, and integration. Its verification scope does not include long-term stability or compatibility. It answers whether an empty directory can become a bootable prototype, not whether the result can run as a production system.

The formalization experiment shows the strength of automated checking: every step can be machine-verified, shrinking the space of “looks correct.” It still targets formalization of an existing proof. It does not automatically mean proposing new proofs or solving open problems. Treat these experiments as evidence for specific task shapes, not as a general capability certificate.

Safety and access: record capability, safeguards, and eligibility in separate columns

Safety evaluation for long-horizon agents cannot stop at intercept counts. Public materials may show fewer blocks, but whether high-risk tasks can still be performed must be confirmed item by item against the protection scope. A selection table should at least include:

• Capability: task type, inputs and outputs, verification method
• Protection status: open, restricted, approval required, unavailable
• Access eligibility: whether the organization qualifies
• Fallback path: which general capability takes over if access is restricted
• Audit evidence: logs, artifact versions, verification reports
• Cost basis: calls, storage, human review, failed retries

If eligibility is ignored, a leading score may correspond to a service scope the team cannot use. The same underlying model does not mean the same service scope on every account.

Adoption advice: close a small loop before expanding horizon

For enterprise adoption, start with medium-horizon tasks that are verifiable, rollback-capable, and auditable: generate a bootable prototype from an empty directory, formalize an existing proof, or add tests and review on a fixed codebase. Each round should record task-graph size, number of sub-agents, acceptance pass rate, human review time, and total bill. Expand horizon and automation only after acceptance criteria are stable, failures can be localized, and cost is predictable.

On infrastructure, general object storage can hold artifacts and versions, a message queue can decouple subtasks, and logging plus monitoring can support audit and anomaly detection. The goal is not to stack tools. Every deliverable should have provenance, verification, and a bill.

Conclusion: the cost effectiveness of multi-day agents cannot be decided by a single demo or a single score. Capability, safeguards, access, acceptance, and billing belong on the same table. Delivering inspectable results is the starting point for production evaluation of long-horizon agents.