Evaluating Long-Horizon Agent Tasks: What Cross-Day Collaboration Experiments Reveal About Real-World LLM Delivery

Why Long-Horizon Tasks Are the New Benchmark for LLM Capability

Historically, LLM evaluation focused on single-turn reasoning benchmarks and code completion accuracy—“sprint metrics.” But what truly determines whether a model can enter production is its “marathon ability”: can it maintain task decomposition, autonomous execution, and progress integration over hours or days, and finally return a coherent, verifiable deliverable?

Two recent representative experiments offer a useful lens:

1. Cross-day software build: Starting from an empty folder, a model coordinated over a thousand sub-agents to continuously code, test, and cross-review. The final output was a prototype operating system that could boot on a standard laptop—including a kernel, basic drivers, network stack, persistent file system, window system, login screen, and installer. Note that validation only covered “ability to boot,” not long-term stability or hardware compatibility.
2. Formal proof reproduction: Over a thousand sub-agents collaborated to rewrite an existing proof of a famous mathematical theorem into a form where every step can be automatically checked by a computer. This took about 30 hours, whereas a human team had previously needed roughly 11 days for similar work.

The value of these experiments is not the sensational “the model wrote an OS” headline, but that they validate a critical chain: task decomposition → parallel multi-agent execution → result integration → overall acceptance. This is exactly the core requirement that enterprise-grade software engineering places on AI collaborators.

Multi-Agent Orchestration: The Technical Backbone Behind Long Tasks

To understand why such experiments are feasible, we need to look at the underlying technical structure:

• Task graph planning: The main model first generates a directed task graph. Nodes are independently verifiable subtasks (e.g., “implement the disk read/write interface”), and edges define dependencies.
• Parallel sub-agent execution: Each subtask is assigned to a sub-agent with its own context, preventing the context window from being overwhelmed by the long task. Sub-agents do not directly talk to each other; they interact only through artifacts (code, documents, test cases).
• Cross-review mechanism: A module written by one agent is reviewed and tested by another, creating a check-and-balance similar to human code review.
• Failure rollback and retry: If any subtask fails acceptance, the main model re-plans or re-assigns that subtask instead of starting everything from scratch.

For engineering teams, the key takeaway is this: if you want to reuse this pattern in your own business, the most important investment should be formalized acceptance criteria. For example, you can define completion conditions for each subtask using pseudocode like:

task = {
    "name": "implement_kv_store",
    "acceptance": [
        "unit_tests_pass: tests/kv/*.py",
        "coverage >= 80%",
        "benchmark: p99_latency < 5ms"
    ],
    "on_fail": "replan"
}

The more machine-checkable the acceptance criteria, the more reliably the agents can operate autonomously. Vague acceptance criteria will cause a cross-day task to “drift” midway.

Beyond Raw Capability: Safety Tiers Determine What You Can Actually Use

The same base model is often offered in different product forms, with differences mainly in safety guardrails and access rules. Industry practice typically includes:

• Public-facing versions carry strong cybersecurity and biosecurity protections. Categories such as penetration testing assistance, exploit generation, and high-risk biological experiment design are blocked or degraded. Newer versions often reduce “false block” rates via more refined classifiers, but a lower overall block count does not mean that all high-risk tasks are allowed.
• Versions offered to vetted organizations may relax some guardrails. Official benchmark scores in high-risk areas (e.g., network attack/defense simulations, protein design tasks) are often attributed to these tiers.

The direct implication for model selection: You cannot directly transplant a score from a different version of the same base model into your own evaluation table. The correct approach is to record three columns in your selection matrix: capability score, safety tier, and your organization’s eligibility for that tier. Without the latter two, a seemingly superior score may correspond to a product you have no access to.

Cost Accounting: Including the Full Bill in Your Evaluation

The cost structure of long-horizon tasks is fundamentally different from traditional API calls:

1. Token compounding effect: In a multi-agent architecture, the main model’s planning, sub-agents’ execution, and reviewers’ re-checking all multiply token usage exponentially. A “~30 hour” experiment could correspond to hundreds of times the cost of a normal chat session.
2. Human acceptance cost: The model delivers “a result that can be checked as a whole,” but the checking itself still requires senior engineers’ time. Between “the prototype OS can boot” and “the prototype OS is production-ready,” there is a huge engineering gap.
3. Retry cost for failed tasks: Failures in long tasks often occur late in the process, sinking the earlier investment. Evaluations should measure end-to-end task success rate, not per-step accuracy.

A pragmatic cost formula is:

True cost per task = API cost + (human review hours × hourly rate) + expected retry cost

Only by calculating this can you answer whether the model can take over your project.

Actionable Recommendations for Engineering Teams

• Start with tasks that have formal verification: Work like test case generation, documentation completion, and proof-checking is naturally suited to long-running agent patterns.
• Build a layered acceptance system: Put machine-checkable items (compilation passes, test passes) into an automated pipeline; keep human review for items that are difficult for machines to judge (architecture soundness, security design).
• When selecting a model, check the conditions behind any demo: For any capability demonstration, confirm its supporting conditions—acceptance scope, safety tier, organizational access—all three are indispensable.
• Leverage cloud tooling for production deployment: In an enterprise environment, you can combine agent orchestration with general cloud services such as object storage (for task artifacts and checkpoints), message queues (for subtask distribution), logging services (for full auditability), and monitoring/alerting (for detecting abnormal interruption of long tasks). Mainstream cloud platforms offer these generic categories, and you can build a controlled long-running task foundation by combining them.

Conclusion

The significance of long-horizon agent experiments is to push the question from “how complex a thing can the LLM do” to “over what period can the LLM reliably deliver acceptable, verifiable results.” For engineering teams, chasing benchmarks matters less than building your own acceptance criteria, collaboration structure, and cost model—these three are the real levers for converting model capability into productivity.