Long-Task AI Agents: Engineering Practices, Verifier Loops, and Cost Governance

Introduction: The Real Bottleneck Is Convergence, Not a Single Benchmark

Long-task AI agents have become a major focus in AI engineering. Instead of asking which model performs best in a single turn, the more important question is whether a system can converge reliably across long-running, multi-step, and highly dependent workflows.

According to an official demonstration from a leading AI lab, anonymized in the source material, about 1,000 agent instances worked together for roughly 30 hours to convert an existing proof of Fermat’s Last Theorem into a machine-verifiable formal proof. Compared with the lab’s earlier approach, the new demonstration reportedly used about one-third of the action tokens and produced a proof about one-seventh the length. These figures are specific to the demonstration task and should not be treated as universal capability for all long tasks.

This article summarizes the engineering essentials: how to decide whether a task suits long-running agents, how to organize multi-agent collaboration, how to use verifiers as a closed feedback loop, and how to manage budgets, evaluation, safety, and cost.

1. What Is a Long-Task Agent, and What Tasks Fit?

1.1 Four Characteristics of Long Tasks

A long task usually meets the following conditions:

1. Large time span: It can last from tens of minutes to several days and involve hundreds of thousands or even millions of model calls.
2. Complex state: It depends on many intermediate artifacts, such as code repositories, proof files, experiment records, and design documents.
3. Verifiable feedback: In principle, each step can be checked by a program, such as compilation, tests, proof checkers, or schema validation.
4. Accumulating errors: If each step is 95% correct, 20 consecutive steps yield an overall correctness rate of about 0.95^20 ≈ 36%.

This is where traditional conversational AI is weak. Conventional models are usually optimized for the current turn, while long-task agents must complete a full loop: planning, execution, verification, error correction, and convergence.

1.2 Which Tasks Are Suitable?

Tasks suitable for long-running agents usually have three properties:

• Acceptance criteria can be verified: Completion can be judged by programs, tests, checkers, or clear rules.
• The goal can be decomposed: The overall goal can be broken into subgoals with dependencies or parallel opportunities.
• Results can be composed: Subgoal outputs can be mechanically merged into the final result.

Examples include code repair, test-case generation, formal proofs, data-analysis pipelines, engineering configuration generation, automated operations scripts, and scientific computing assistance.

Tasks that are less suitable include those with vague acceptance criteria, heavy subjective judgment, very high verification cost, low error tolerance, and no reliable rollback mechanism.

2. Case Study: 1,000 Agents Tackling Formal Verification

2.1 What Was Demonstrated?

The figures below come from a public demonstration by a leading AI lab:

• Task: Convert an existing mathematical proof of Fermat’s Last Theorem into a machine-verifiable formal proof.
• Input: The model received only limited prompts and a set of intermediate theorem statements.
• Autonomy: Collaboration mechanisms, task allocation, use of formal proof languages, verification, and correction were handled by the agents themselves, with very limited human intervention.
• Reported metrics: Completion time, action-token consumption, and final proof length.

Three types of claims should be distinguished:

1. Official demonstration figures: Results produced under a specific task, prompt setup, and model version.
2. Third-party hands-on reports: Independent experiences from external testers, which may reflect practical impressions but are not necessarily standardized benchmarks.
3. Production engineering results: Results measured under a company’s own tasks, data, constraints, and acceptance criteria.

Therefore, demonstration numbers should not be treated as universal performance guarantees.

2.2 Why Formal Verification Is a Strong Testbed

Fermat’s Last Theorem states that for any integer n > 2, the equation x^n + y^n = z^n has no positive integer solutions. It was once called one of the hardest problems in mathematics.

Andrew Wiles began working secretly on the problem in 1986, announced a proof in 1993, and spent more than a year fixing a gap found during review. The final proof was published in 1995. The paper is over 100 pages long and builds on a large amount of advanced mathematics.

“Formalization” means rewriting a natural-language proof into a formal proof language so that every inference step can be mechanically checked by an interactive theorem prover. One Dutch mathematician estimated that formalizing proofs in a mathematical library takes about three to four times the effort of writing them on paper.

Importantly, the AI in this case was formalizing and verifying an existing proof, not discovering a new proof. Whether the proof is correct is not decided by the model; it is decided by the proof checker. This gives long-task agents an ideal testing environment: the judge cannot be persuaded.

2.3 Comparison with Earlier Approaches and Control Groups

According to the demonstration:

• About 1,000 agents completed the task in roughly 30 hours.
• An earlier, unpublished research version reportedly took about 11 days.
• A control group using models of similar agent scale did not achieve a clear breakthrough within 30 hours. Progress nearly stalled after 18 hours, and about 26% of the formalization work had been completed by that point.

The comparison suggests:

• The new approach took about one-eighth of the time of the earlier approach.
• It used about one-third of the action tokens.
• The final proof was about one-seventh the length of the earlier proof.
• In the control group, progress nearly stalled after 18 hours despite a similar agent scale.

This suggests that efficiency in long-task agents is not determined by how much compute is used, but by how much actual consumption translates into task completion. Gains come from better planning, fewer ineffective attempts, more direct proof paths, and stronger error recovery.

2.4 Transferability to Scientific Computing

According to the demonstration and related reports, newer models also completed calculations for two open problems in black-hole physics, one of which involved holographic duality.

Holographic duality relates a gravitational theory to a lower-dimensional quantum field theory on its boundary. This can transform some difficult gravitational problems into more tractable ones. The demonstration reportedly found, under small-scale spacetime assumptions and specific model conditions, a phase-transition-like process resembling the sudden formation of a black hole.

These results should still be treated as demonstration or research reports. They should be evaluated alongside papers, reproducible experiments, and expert review before being generalized.

3. Three Engineering Essentials for Long-Task Agents

3.1 Verifiable Task Decomposition

A formal proof is essentially a dependency tree: a top-level theorem depends on lemmas, lemmas depend on smaller lemmas, and so on until axioms, existing theorems, or machine-decidable basic propositions are reached.

Each lemma can be judged independently by a checker, regardless of whether other lemmas are complete. The overall task therefore becomes a large set of parallel subtasks that can be accepted immediately after submission.

A subgoal graph can be used to organize this structure:

from dataclasses import dataclass, field
@dataclass
class SubGoal:
    id: str
    statement: str
    depends_on: list[str] = field(default_factory=list)
    verify_cmd: str = ""
    status: str = "pending"
    attempts: int = 0
    max_attempts: int = 8
def ready_to_run(graph: dict[str, SubGoal], max_parallel: int) -> list[SubGoal]:
    done = {gid for gid, g in graph.items() if g.status == "verified"}
    return [
        g for g in graph.values()
        if g.status in ("pending", "failed")
        and g.attempts < g.max_attempts
        and set(g.depends_on) <= done
    ][:max_parallel]

For any long task, three questions can be used to test suitability:

1. Can the task be split into units whose correctness can be judged by a program?
2. Do those units offer significant parallel opportunities?
3. Can the final result be mechanically composed from the outputs of those units?

If all three answers are yes, the task is a good candidate for long-task agents.

3.2 Verifier-Centered Feedback Loops

A long-task agent is not a one-shot answer generator. It operates as a loop: generate, verify, and correct.

Verifiers may include:

• Compiler type checks and static analysis;
• Unit tests and integration tests;
• Schema validation and data-consistency checks;
• Conservation checks for energy, mass, and other quantities in physical simulations;
• Formal proof checkers.

In practice, only context relevant to the target should be provided to the model to avoid context inflation. Only concise summaries of structural errors should be written back to the context, preventing irrelevant output from drowning out key information.

Verifier quality determines the ceiling of agent capability. Any area that verifiers cannot cover becomes a space where the model can “persuade itself.”

3.3 Humans Should Intervene Only at High Levels

A reasonable division of labor is:

• Humans define direction, set standards, and perform spot checks.
• Agents execute, experiment, and converge.

Task descriptions must be machine-verifiable. For example:

• Not verifiable: “Write a fully functional operating system.”
• Verifiable: “A bootloader that passes a specified test set, starts within X seconds, and has zero crashes.”

Humans should retain control over three key checkpoints: requirements definition, high-risk operations, and final acceptance.

4. From Demonstration to Production: Topology, Budget, and Evaluation

4.1 Choosing a Topology

Common topologies include:

Topology	Suitable Scenarios	Advantages	Risks
Single-agent loop	Small tasks with long dependency chains	Simple to implement; coherent context	Serial and slow; may fall into local loops
Orchestrator–worker	Tasks can be split into subgoals	Clear global planning; easy budget allocation	Orchestrator can become a bottleneck
Shared blackboard	Many subtasks need shared intermediate artifacts	Results can be reused; reduces duplicate work	Requires concurrency control and versioning
Pipeline	Clear stages with strong handoffs	Each stage can be optimized independently	Upstream errors propagate downstream

Production systems often use hybrid topologies: an orchestrator handles global planning and acceptance, workers advance subgoals in parallel, results are written to shared storage, and a verification pipeline makes the final decision.

4.2 Common Failure Modes

Long-task agents commonly fail in the following ways:

1. Ineffective loops: Repeatedly trying the same path without convergence.
2. Misunderstood intent: Interpreting acceptance criteria incorrectly, producing unusable results.
3. Tool misuse: Calling tools with excessive permissions, wrong parameters, or unintended effects on external resources.
4. Weak verifiers: Checkers fail to detect errors, leaving room for model self-persuasion.
5. Context inflation: Irrelevant information consumes model budget and reduces judgment quality.
6. Runaway cost: The task fails to converge, while token consumption continues.

4.3 Budgeting and Convergence Control

The biggest cost risk in long tasks is non-convergence. A practical system should define four budgets:

• Token budget;
• Time budget;
• Cost budget;
• Global retry budget.

In addition:

1. Allocate budgets by subgoal: Prioritize high-value, low-cost subgoals.
2. Monitor progress: Trigger human intervention when no new “verified” results appear for a long time.
3. Fail gracefully: If a subgoal fails repeatedly, try another path; if that also fails, escalate to a human.
4. Persist state: Write all intermediate artifacts to disk to support checkpoint recovery.

Data-loss and accidental-deletion risks should also be addressed. Intermediate artifacts should be versioned, critical directories should enable snapshots or trash, deletion operations should be audited and alerted, and interrupted tasks should resume from the latest checkpoint.

4.4 Evaluation Should Use Task-Level Metrics

Enterprise selection should not rely only on public benchmark scores. Teams should build their own task-level evaluation system:

• Task pass rate: The share of tasks that fully meet acceptance criteria.
• Number of human interventions: Reflects autonomy.
• Cost per successful task: Total tokens and cost divided by successful tasks.
• Execution efficiency: Token consumption and runtime under equivalent completion quality.
• Failure-mode distribution: Proportions of hallucination, intent misunderstanding, ineffective loops, tool misuse, and other failures.

4.5 Why Unit Price Alone Is Insufficient

A lower unit price does not necessarily reduce total spending. Token consumption in long tasks grows with task duration and may increase linearly or even superlinearly.

Comparisons should use cost per successful task.

Example: Task A consumes 10 million tokens per attempt with a 50% success rate. Task B consumes 8 million tokens per attempt with an 80% success rate. To complete 1,000 successful tasks:

• Task A requires approximately: 1000 ÷ 0.5 × 10 million = about 20 billion tokens.
• Task B requires approximately: 1000 ÷ 0.8 × 8 million = about 10 billion tokens.

Therefore, even if Task A has a lower unit price, its total cost may be higher.

Common optimization methods include:

• Model routing: Use lightweight models for simple subgoals and flagship models for complex subgoals.
• Caching and reuse: Avoid regenerating repeated context and intermediate artifacts.
• Elastic resource scheduling: Balance cost with spot instances, batch jobs, and reserved capacity.
• Parallelism control: Parallelism reduces wall-clock time but can amplify peak resource usage; quotas and rate limits are needed.

4.6 Safety and Governance

High-capability models require tiered governance. Recommended measures include:

1. Permission and purpose controls: Distinguish scenarios such as internal R&D and external collaboration, and define accessible capabilities clearly.
2. Data isolation: Sensitive code and data should not leave dedicated environments. Prompts and outputs in logs should be masked as needed.
3. Auditability and traceability: Record every agent tool call and artifact to facilitate diagnosis.
4. Sandboxing and least privilege: Code execution, file access, and network access should follow the principle of least necessity.
5. Protection against data loss or accidental deletion: Use versioning, snapshots, trash, and deletion alerts for critical artifacts.
6. Prompt-injection defense: Treat content from external web pages, files, and tool returns as untrusted data.

Prompt-injection defenses should include:

• Separating content from instructions;
• Isolating untrusted data;
• Preventing tool output from directly triggering high-privilege operations;
• Requiring confirmation for critical actions;
• Allowlisting commands and network access;
• Alerting on anomalous calls.

5. Best-Practice Checklist

1. Verify decomposability first: Only tasks that can be split into independently verifiable units are suitable for long-task agents.
2. Build verifiers before deploying agents: Do not give a task to long-task agents unless a reliable judge exists.
3. Set budgets first: Define token, time, and cost limits, as well as failure-degradation policies, before launch.
4. Make state recoverable: Persist all intermediate results and support checkpoint resumption.
5. Build an evaluation system: Measure cost per successful task using real business tasks, not just benchmark scores.
6. Build security alongside capability: Permissions, auditing, sandboxing, injection defenses, and data protection are all required.
7. Keep humans at key checkpoints: Human confirmation is required for requirements definition, high-risk operations, and final acceptance.
8. Label claims with context: Demonstration figures must state their source, task definition, and measurement method, and should not be treated as universal conclusions.

Conclusion

The competitiveness of long-task agents does not come from a single impressive demo. It comes from whether the system can reliably, reproducibly, and auditably complete real tasks under budget constraints, verification loops, and human supervision.

For developers and enterprises, the real opportunity is not chasing a single release. It is making three things solid: verifiable decomposition, verifier-based closed loops, and disciplined budgeting and governance. That is how demonstrations become productivity.