Choosing an AI Agent for Long-Horizon Tasks: Capability, Access Tier, and the Full Cost Bill

## 1. First, Get the Evidence Straight: What the Two Experiments Actually Prove

A model provider recently published two experiments on long-horizon autonomous coding.

In the first, the model started from an empty directory. Upper-level orchestration logic scheduled thousands of sub-agents through repeated loops of coding, testing, and code review, and the result eventually booted on a single laptop. The delivered scope covered a kernel, disk and keyboard drivers, a network protocol stack, persistent file storage, windowing and a login UI, an installer, and a built-in assistant capability.

In the second, an array of sub-agents at the same scale ran an earlier version of the model and completed a previously published formalization in roughly 30 hours, where comparable prior work had taken 11 days. “Formalization” here means breaking a proof into steps that a machine can check one by one.

Together these two data points support one judgment: for tasks with clear goals, machine-checkable verification, and separable parallel work, long-running autonomous execution has moved from demonstration into a deliverable stage. But each result answers only one narrow question — a prototype boots; a proof passes the checker. To make a selection decision from them, you also have to read the conditions under which they hold, the guardrail boundaries, and the real bill.

2. Three Dimensions That Decide Whether the Conclusion Holds

Dimension 1: Verifiability Determines the Strength of the Conclusion

Formal verification returns a binary verdict: the checker either accepts or rejects, with no room for interpretation. That is precisely why it works as a capability measure. “Boots from an empty directory” is a different kind of claim — booting is a necessary condition, not a sufficient one. The provider’s own material states that the validation scope excludes long-term stability and compatibility.

Abstracted out, tasks fall into roughly three classes:

Task type	How it is judged	Conclusion strength	Engineering advice
Clear goal + machine-decidable	Checker or tests return a verdict directly	Strong	Safe to hand to autonomous execution; optimize verification cost
Clear goal + expensive verification	Requires long runs or manual comparison	Medium	Narrow the verification scope first, then widen authorization
Ambiguous goal + subjective acceptance	Requirements still need clarification	Weak	Have a human produce the spec and acceptance criteria before delegating execution

Dimension 2: The Same Base Model Is Not the Same Service

The published material states that the consumer-facing version and the version offered to approved organizations are defined as “the same base model,” sharing an identical model identifier. The differences come from guardrails and access rules.

The version open to general users carries stronger cybersecurity and biology guardrails: penetration testing, exploit generation, binary vulnerability analysis, and most research-grade biology tasks are restricted, or answered by a model of a different tier. The version restricted to organizations that have passed verification does not carry equivalent restrictions in these two areas.

This leads to two inferences that are easy to get wrong.

First, a lower interception rate does not mean higher capability. Early testing reported by the provider shows the open version intercepting roughly 55% fewer requests in total than the previous-generation minor release, with about 60% fewer in the cybersecurity category and about 40% fewer in the biology category. That number describes fewer requests actually blocked. It reflects adjustments to the interception strategy and to classification accuracy, not permission to perform high-risk tasks. What you can and cannot do still has to be confirmed item by item against the guardrail scope.

Second, benchmark scores cannot be borrowed across versions. In the official benchmarks, the composite cybersecurity benchmark of 76.3%, the vulnerability-reproduction benchmark of 89.6%, and the de novo protein binder design score of 85.1% are all labeled for the approved-organization access version. The base model is the same, but the service scope an account can actually reach is not. A seemingly leading score may well correspond to a version you cannot obtain.

Dimension 3: The Complete Bill Includes the Parts That Get Ignored

Thousands of sub-agents running continuously cost far more than inference alone: planning and scheduling overhead, repeated context injection, failed retries, occupied verification environments, review labor, log and storage retention, and security and compliance overhead all land in the total. The more important issue is the denominator — cost only becomes comparable when it is measured per accepted result.

3. Move Acceptance Criteria Upstream: From Demo Metrics to Engineering Metrics

Demo metrics (does it boot, does it pass the checker) are not the same thing as engineering metrics (reproducible builds, regression coverage, crash rate, compatibility matrix, maintainability). Before authorizing any long task, fix an acceptance checklist:

• Reproducible build: builds with a single command in a clean environment, with a stable artifact hash.
• Functional coverage: every requirement mapped to a test case; anything unmapped counts as incomplete.
• Stability: several rounds of continuous running and cold starts, with failure rate and recovery time recorded.
• Compatibility: combinations outside the declared support range explicitly marked as unsupported.
• Security and licensing: dependency provenance, vulnerability scanning, license compliance.
• Maintainability: module boundaries, documentation, and the context cost required for handoff.

Writing down what “passing” means in a sample config is a precondition for long tasks being deliverable at all:

task: prototype-kernel-boot
goal: boot from an image inside an isolated VM and reach the login screen
verification:
  - type: clean_build          # build after clearing the cache
    timeout: 30m
  - type: boot_smoke
    attempts: 10
    pass_ratio: 0.9            # at least 9 of 10 boots must succeed
  - type: input_output         # keyboard input, disk read/write, network handshake
acceptance:
  required: [clean_build, boot_smoke]
  score_weights: {boot_smoke: 0.6, input_output: 0.4}
  reject_if: [any_unlisted_hardware_dependency]

The scoring logic itself should be a few lines of code rather than a discussion in a review meeting:

def evaluate(run) -> dict:
    checks = {c["type"]: verify(c, run) for c in run.task.verification}
    score = sum(w * checks[t] for t, w in run.task.acceptance["score_weights"].items())
    accepted = all(checks[t] for t in run.task.acceptance["required"]) and score >= 0.8
    return {"accepted": accepted, "score": round(score, 3), "checks": checks,
            "retry_count": run.retry_count, "token_cost": run.token_cost,
            "human_minutes": run.human_review_minutes}

4. Shadow Runs: How to Design a Small Pilot

Do not draw conclusions from a single task and a single run. A workable sequence:

1. Pick 10–20 historical tasks from a real codebase, where the human baseline outcome is already known.
2. Fix the environment and a budget cap, then run autonomous execution and the human baseline in parallel.
3. Record pass rate, average duration, retry count, human intervention time, and unit cost.
4. Classify failures: requirement misunderstanding, insufficient context, missing verification, tool limitations — each points to a different improvement.

Three metric families are the easiest to omit: first-pass rate (the share accepted without human rework), rework cost (the additional human and machine effort needed to reach acceptance), and verification coverage (how much human judgment the automated checks can replace). The last one determines the ceiling on scaling.

5. Doing the Math: Put Unit Cost and Yield Side by Side

A workable estimate is: cost per accepted result = (model inference + orchestration and retry overhead + verification environment + human review) ÷ number of accepted results. Two conclusions here run counter to intuition.

One, raising parallelism can push unit cost up. The more sub-agents you run, the faster coordination and repeated-context overhead tends to grow relative to useful output — this holds only when the task is genuinely separable and verification is cheap. If verification is expensive, adding parallelism just amplifies waste.

Two, review labor is usually the most underestimated line item. Autonomous execution shifts cost from writing code to reviewing code, and without an automated acceptance checklist, human review becomes the bottleneck. On the engineering side, general-purpose cloud capabilities help bring observability and rerun costs down — object storage for retaining build artifacts, a message queue for decoupling task scheduling, logging services with monitoring and alerting to track anomalous tasks. When choosing, evaluate the specific platform against your own compute profile and data compliance requirements.

6. Collaboration Structure and Guardrail Boundaries: Two Red Lines for the Selection Process

Collaboration structure. The value of a sub-agent array is context isolation and parallel exploration; the price is coordination overhead and duplicated information. The constraints that are actually operable: cap the context and step count per agent; split planning, execution, review, and verification into distinct roles; land every artifact in a file rather than in conversation; and place a human-accessible checkpoint at the end of each phase. The human role shifts from writing the implementation to writing the spec and the acceptance criteria, which demands a substantially higher precision in the spec itself.

Guardrail boundaries. If your work involves security testing or life-science research, do not infer the available scope from “same base model.” Instead, determine the service tier your account actually receives, and verify item by item whether restricted requests are blocked, degraded, or routed to a different model. Record this in your compliance review material. Data such as a lower interception rate is evidence that the policy is more precise — it is not evidence that capability has been unlocked.

7. Conclusion: What a Selection Table Should Record

Deciding whether a system can take on your project requires answering four questions at once: is the task machine-decidable; does the service tier and guardrail state your account actually receives match the requirement; were the acceptance criteria fixed before work started; and has the complete bill, including review labor and verification environments, been calculated. Accordingly, a selection table needs at least these fields: model identifier and service tier, available benchmarks with the version each corresponds to, guardrail restriction list, access eligibility, cost per accepted result, first-pass rate, and average rework hours.

One final judgment is worth emphasizing: in long-horizon scenarios, a “stronger model” usually determines success or failure less than “stricter acceptance criteria + finer parallelism constraints + a more complete cost basis.” Demo footage proves possibility; the acceptance checklist is what determines deliverability.