Engineering Long-Running AI Agents and Multi-Agent Clusters: State, Scheduling, and Fault Recovery

    As LLM applications evolve from question-answer tools into long-running execution engines, the engineering challenge shifts from inference latency and context length to state management, task scheduling, fault recovery, and capacity governance. This article walks through three capability milestones—continuous coding tasks that run for hours, parallel coordination of hundreds of sub-agents, and day-level unattended operation—and the concrete practices each one demands: layered memory, checkpoints and idempotency, self-correction loops, DAG-based decomposition, heterogeneous roles, workspace isolation, backpressure-aware scheduling, budget degradation, observability, task-level evaluation, private deployment checklists, and a phased rollout roadmap.

    1. The Shift in the Problem: From “Information Container” to “Execution Engine”

    Over the past two years, the typical shape of LLM applications has been question and answer: feed in a chunk of context, produce a chunk of content, and destroy the state when the session ends. The engineering complexity of such systems concentrates on inference latency and context length.

    When the product form shifts to an “execution engine”—where the model can autonomously decompose a goal, call tools, write code, run tests, read logs, and self-correct based on failures—the nature of the system changes. It is no longer a stateless inference service but a long-running, stateful distributed system.

    This shift has three direct engineering consequences:

    Dimension Conversational apps Long-horizon agent systems
    Single lifecycle Seconds Minutes to days
    State Disposable Must be persisted and recoverable
    Failure cost Ask again Hours of compute and cost lost
    Concurrency model Request-level Task-level + subtask-level
    Core metric Time to first token Task success rate / human intervention rate / cost per task

    This article uses three typical engineering goals as reference points: continuous coding tasks that run for hours up to tens of hours, parallel coordination of on the order of a hundred sub-agents, and day-level unattended autonomous operation. These three goals are not the targets of any single product; they are capability thresholds that agent infrastructure in general must clear today.

    2. Capability Line One: Long-Horizon Coding—Making “Several Continuous Hours” a Recoverable Process

    The hard part of a goal like “continuous coding for 13 hours” is not “how many tokens the model can generate” but the fact that context will inevitably overflow, errors will inevitably occur, and processes will inevitably be interrupted. The real engineering problem is: how do you make a process that might fail ten thousand times eventually converge to a usable result?

    2.1 Layered Memory: Turning Context from “Linear Growth” into a “Retrievable Structure”

    A single ultra-long context fails simultaneously on three fronts: high cost, attention decay, and noise accumulation. A workable approach is a three-layer structure:

    • Working layer (Working Context): the local code for the current file, the most recent error stack, and the next planned step. Strictly bounded to a few thousand tokens.
    • Summary layer (Episodic Summary): after each phase completes, compress the actions taken into structured entries (what was done, which files changed, verification results, open issues) rather than a natural-language stream of consciousness.
    • Retrieval layer (Long-term Memory): codebase indexes, interface documentation, and historical decision records, retrieved and injected on demand rather than kept resident.

    The key engineering constraint: summaries must be structured and machine-consumable, for example {goal, touched_files[], verified: bool, open_issues[]}. This lets the scheduler decide whether execution can continue, instead of forcing the model to interpret a vague narrative.

    2.2 Checkpoints and Idempotency: Making Tasks Resumable from a Breakpoint

    The core of fault tolerance in long-horizon tasks is state persistence plus re-entrant execution. The recommendation is to abstract every side-effecting operation as an action with a unique identifier, so that re-execution produces no additional consequences.

    1. @dataclass
    2. class TaskState:
    3. task_id: str
    4. plan: PlanDAG # task decomposition graph
    5. completed: dict[str, StepResult] # step -> result summary
    6. revision: int # monotonically increasing, for optimistic concurrency control
    7. budget: Budget # triple limits: duration / tokens / cost
    8. checkpoints: list[str] # object keys in persistent storage
    9. class AgentLoop:
    10. def step(self, state: TaskState) -> TaskState:
    11. if state.budget.exhausted():
    12. return self.suspend(state, reason="budget") # suspend, do not discard
    13. nxt = state.plan.next_ready(state.completed)
    14. if nxt is None:
    15. return self.finish(state)
    16. try:
    17. result = self.executor.run(nxt, idem_key=f"{state.task_id}:{nxt.id}")
    18. except TransientError as e:
    19. return self.retry_with_backoff(state, nxt, e)
    20. except PermanentError as e:
    21. return self.escalate(state, nxt, e) # trigger human / stronger-model review
    22. state.completed[nxt.id] = result
    23. return self.persist(state, revision=state.revision + 1)

    Design points:

    1. Suspension (suspend) and failure (fail) must be distinguished. An exhausted budget or waiting on external approval is a suspension and should preserve all state for later recovery; only an unrecoverable logic error terminates the task.
    2. Every state change must be persisted, and the write must be atomic (write a new object, then atomically swap the reference), so that a killed process never leaves a half-written state.
    3. Idempotency keys (idem_key) must cover every external side effect, including file writes, code commits, ticket creation, and message sending.

    2.3 Self-Correction Loop: Turning Failure into Input

    The value of a long-horizon task comes mainly from “being able to climb out of a hole on its own.” A minimal usable correction loop has four steps: capture the complete failure signal (stack, exit code, log excerpt) → localize it to a specific assumption (which precondition on which line of code was wrong) → generate a minimal fix → re-run the original verification command to confirm. The fourth step is mandatory after the third; otherwise the model falls into a hallucination loop of “declaring the fix successful.” At the same time, you should set an upper bound on the number of repeats for the same error; exceeding it escalates to a stronger model or a human.

    3. Capability Line Two: Multi-Agent Clusters—Scheduling and Isolation at Hundred-Agent Concurrency

    Supporting “300 agents coordinating in parallel” is, from an engineering standpoint, equivalent to running a batch-processing cluster with hundreds of tasks, uneven lifecycles, and heterogeneous resource demands. It is not a matter of turning up the concurrency number; it requires solving four problems: decomposition, scheduling, isolation, and merging.

    3.1 Task Decomposition: Parallelism Comes Only After the DAG

    Parallelism is not a configuration knob; it is a product of the task decomposition structure. Only by breaking a goal into a directed acyclic graph, and keeping nodes as data-dependency-free as possible, can you obtain real parallelism.

    1. Goal: add a paginated query endpoint to the service
    2. ┌─ Read existing data-access layer ─┐
    3. ├─ Review API conventions ────────┤ → parallel
    4. ├─ Research similar implementations ┘
    5. ↓
    6. Generate implementation plan (join node)
    7. ↓
    8. ┌── Write code ──┬── Write unit tests ──┬── Write docs ──┐ → parallel
    9. └────────────────┴─────────────────────┴────────────────┘
    10. ↓
    11. Integration test / conflict merge (join node)

    The join node is the key: it determines how parallel results converge into a single trusted artifact. Without an explicit join-and-verify stage, a multi-agent system merely produces mutually contradictory results faster.

    3.2 Heterogeneous Roles: Different Nodes, Different Capabilities and Budgets

    The essence of heterogeneous-collaboration ideas like “Claw Groups” is role differentiation: a planner needs long context and strong reasoning, an executor pursues throughput and low cost, and a reviewer needs rigor and an adversarial perspective. In engineering terms, a role should be abstracted as a four-tuple of “model tier + system prompt + toolset + resource quota” rather than writing separate code for each agent.

    Role Capability focus Tool permissions Budget strategy
    Planner Strong reasoning, long context Read-only A few high-value calls
    Executor High throughput, low cost Codebase read/write, sandbox execution Many short calls
    Verifier Strict, adversarial Read-only + test execution Triggered on demand
    Merger Conflict resolution Version-control write access Single serial point

    3.3 Isolation and Conflict: The Inevitable Cost of Parallel Writes to the Same Codebase

    When multiple agents edit the same repository at once, conflicts are not exceptions but the norm. Isolation techniques, in increasing order of strength:

    1. Logical isolation: assign each subtask a non-overlapping set of file paths, and have the scheduler check for path intersection at allocation time.
    2. Workspace isolation: give each sub-agent an independent working copy/branch, ultimately integrated serially by the merger.
    3. Execution isolation: run generated code inside disposable container sandboxes that restrict network egress, filesystem scope, and CPU/memory limits.

    The third cannot be skipped. Letting model-generated code run directly in the host environment is equivalent to handing control of the production environment to an uncertain input source.

    3.4 Scheduling and Backpressure: Concurrency Must Be a Controllable Variable

    The scheduling loop needs to explicitly control three quantities: the global concurrency ceiling, the per-tenant quota, and the carrying capacity of downstream dependencies.

    1. def schedule(dag: PlanDAG, state: TaskState, limiter: ConcurrencyLimiter):
    2. ready = [n for n in dag.ready_nodes(state.completed)]
    3. for node in sorted(ready, key=lambda n: n.priority, reverse=True):
    4. # If tokens are insufficient, queue instead of blocking the main loop: this is backpressure, not a crash
    5. if not limiter.acquire(tenant=node.tenant, kind=node.kind):
    6. break
    7. try:
    8. dispatch_async(node, on_done=lambda r: on_node_done(state, node, r))
    9. except Exception:
    10. limiter.release(node.tenant, node.kind) # a dispatch failure must return the token
    11. raise

    A design principle worth stressing: backpressure beats timeouts. When resources are short, letting tasks queue and wait is far more economical than letting them time out, fail, and retry—the retry cost of a long-horizon task is two to three orders of magnitude higher than that of an ordinary request.

    4. Capability Line Three: Day-Level Autonomous Operation—Making Reliability Infrastructure

    When the expected runtime of a task reaches the day scale, failures are no longer exceptions but certainties: processes get restarted, containers get evicted, nodes go offline, and upstream APIs rate-limit. The system’s design premise should shift to “interruption may happen at any time, and execution must be able to continue.”

    4.1 Three Governance Problems That Must Be Closed

    • Side-effect governance: over a day of runtime, an agent performs many irreversible actions (sending messages, modifying external systems, consuming quota). You must establish a tiered approval gate: low-risk actions execute automatically, medium-risk actions execute after batch confirmation, and high-risk actions require mandatory human sign-off. Approval policies should be externalized as auditable rules, not scattered across prompts.
    • Cost and budget governance: set triple limits of duration, tokens, and external-call count for each task, and support graceful degradation—when the budget reaches 80%, automatically switch to a more economical model tier and shut down non-critical subtasks, rather than hard-cutting off at 100% and leaving an unusable half-finished product.
    • Observability governance: debugging day-level tasks cannot rely on digging through logs. At minimum you need: a trace identifier that runs through the whole chain (a single task_id linking all sub-agents and tool calls), a structured event stream (one event per state transition), and stall detection for “stuck, not progressing” (alert and intervene when no state change occurs for more than N minutes).

    4.2 Recovery Drills Matter More Than Recovery Code

    Having checkpoints does not mean you can recover. It is advisable to fold the following scenarios into routine drills: a process killed mid-file-write, a dependency returning a non-retryable error, a budget exhausted during subtask execution, a version conflict from concurrent writes, and a human approval that goes unanswered for a long time. Every drill should validate one concrete metric: whether recovery produces duplicate side effects. This is the only true test of whether idempotency design actually works.

    5. The Serving Side: Traffic Peaks, Rate Limiting, and Quota Compensation

    When a traffic surge beyond the expected capacity causes members to queue, features to fail, and an emergency quota reset to be required, the event usually exposes not a problem with model capability but a maturity problem in traffic governance and the quota system.

    5.1 Tiered Strategies Under a Traffic Peak

    1. Ingress layer: a global token bucket plus per-user/per-tenant quotas, prioritizing paid and business-critical requests so they are not crowded out.
    2. Queue layer: convert synchronous calls into asynchronous tasks plus result polling/push. Route long-running tasks and interactive requests through separate channels so long tasks do not saturate connections.
    3. Degradation layer: preset degradation switches that can progressively disable high-cost capabilities (deep-reasoning tier, ultra-large context, number of parallel subtasks) to preserve basic availability.
    4. Elasticity layer: define the trigger metrics for scaling out. CPU utilization is often a poor signal; closer indicators are queue wait time, concurrent task count, GPU memory occupancy, and queue rejection rate.

    5.2 Key Points for Designing a Quota Compensation Mechanism

    “Resetting quota for everyone” is a correct emergency action, but as a long-term mechanism it needs to be engineered, or it introduces new problems (duplicate issuance, reconciliation mismatches, and abuse). The key points:

    • Idempotent issuance: use compensation batch ID + user ID as the unique key, so a repeated delivery does not trigger a second issuance.
    • Reconcilable: retain issuance logs and before/after balance snapshots, supporting balance reconstruction and discrepancy reconciliation at any point in time.
    • Reclaimable and rate-limited: set compensation caps for abnormally high-frequency accounts, so the compensation itself does not become an attack surface.
    • Notifiable: proactively inform affected users through message channels to reduce support load and duplicate appeals.

    5.3 The Basic Formula for Capacity Planning

    A practical rough calculation:

    1. Required concurrency ≈ peak QPS × average occupancy time per request (seconds)
    2. Required GPU memory ≈ weight footprint + KV cache (concurrency × context length × per-token cache overhead) + runtime overhead
    3. Required instances ≈ ceil(required concurrency / stable concurrency per instance) × (1 + redundancy factor)

    KV cache is usually the resource that runs out first, and is most easily underestimated, in long-context scenarios. If GPU memory becomes the bottleneck, evaluate general optimizations such as paged KV-cache management, prefix-cache reuse, and quantization before considering scaling out.

    6. Evaluation: How to Objectively Measure “Long-Horizon Capability”

    Traditional benchmarks mostly measure single-turn answer quality and cannot reflect the level of long-horizon execution. The recommendation is to build a task-oriented evaluation set with the following metrics:

    • Task success rate: the proportion that meets a defined acceptance standard end to end (such as all tests green), not the proportion that “looks reasonable.”
    • No-intervention rate: the proportion completed without any human intervention throughout.
    • Average number of interventions and their timing: the earlier an intervention occurs, the weaker the planning phase.
    • Rework rate: number of steps rejected by the verifier / total number of steps.
    • Cost per unit of effective output: the total cost consumed to complete one task that passes acceptance—this is the metric closest to business value.
    • Wall-clock time and parallelism efficiency: parallelism efficiency = estimated serial time / actual wall-clock time, used to check whether DAG decomposition and scheduling are actually effective.

    Evaluation must be regression-based: run the same task set after every model or prompt change, and retain failure cases as permanent regression cases. It is also advisable to validate new versions with shadow traffic—dual-write real production requests, record the new version’s results without returning them, for comparison only, without affecting users.

    7. Delivery Forms: Open Weights and Private Deployment

    A dual-track form of “open weights + enterprise-oriented cluster solutions” imposes requirements on infrastructure in two directions.

    7.1 The Engineering Checklist for Private Inference

    • Serving layer: dynamic and continuous batching, request queuing and prioritization, streaming output, and timeout and cancellation propagation.
    • GPU memory optimization: paged KV-cache management, prefix-cache reuse, and weight quantization (you must assess the impact of accuracy loss on long-horizon task success rate—a 1% drop in short Q&A may be acceptable, but a long-horizon task can fail entirely over a single misjudgment).
    • Heterogeneous compute adaptation: operator-library coverage, compile-time graph optimization, accuracy alignment (output differences of the same model across different backends must be included in regression tests), and fallback paths (degraded implementations when an individual operator is unsupported).
    • Metering and limits: per-tenant usage collection, quota validation, and billing generation, decoupled from the inference gateway so that a metering failure cannot drag down the main path.

    7.2 How Changes in the Billing Model Impose Reverse Requirements on Architecture

    When subscription pricing shifts from “whether a feature is available” to “the number of agents that can be invoked, the degree of parallelism, and the data-access tier,” billing changes from a marketing problem into an architecture problem. The system must have:

    1. A resource-level quota model: concurrent sub-agent count, parallel task count, storage tier, and access bandwidth must all be independently measurable and limitable.
    2. Near-real-time usage collection: a single long-horizon task may consume hours of resources, so usage must be reported incrementally per phase, not counted once at task completion.
    3. An explainable usage view: let users clearly see which resource category their quota was spent on—this is a prerequisite for high-value users to renew, and the key to reducing disputes.

    8. A Phased Rollout Roadmap

    Phase 1: Single-agent closed loop (1–2 months)
    Wire up “plan → tool call → execution sandbox → verify → state persistence.” The goal is not the capability ceiling but recoverability: after killing the process, resume from a checkpoint without verification producing duplicate side effects.

    Phase 2: Limited parallelism (2–3 months)
    Introduce DAG decomposition, concurrency ceilings, path/workspace isolation, and the merger with conflict resolution. First keep parallelism in the single digits, polish the merge path thoroughly, and only then talk about scaling up.

    Phase 3: Cluster operation (3–6 months)
    Fill in tenant quotas, priority queues, backpressure, budget degradation, full-chain tracing, and stall detection; integrate container orchestration and elastic scaling; establish a task-level evaluation set and a regression pipeline.

    Phase 4: Multi-form delivery (ongoing)
    Reuse the same scheduling and governance layer across both the open-weights and private-deployment lines, swapping only the inference backend, to avoid maintaining a separate logic per delivery form.

    9. A Checklist of Common Pitfalls

    • Equating long-horizon capability with long context: context length only solves “seeing,” while checkpoints and correction loops are what solve “finishing.”
    • Piling on concurrency before doing the merge: parallelism goes up but results conflict, and the net gain is negative.
    • Retrying instead of applying backpressure: constantly retrying under resource scarcity amplifies local congestion into a global avalanche.
    • Discarding on budget exhaustion: design for suspension and recovery so long tasks can resume across windows.
    • Running generated code only in the host environment: the absence of a sandbox is the most typical high-risk gap in a long-horizon agent.
    • Evaluating only single-turn quality: single-turn scores correlate weakly with task success rate; you must use task-level acceptance.
    • Ignoring KV cache and queue wait time in capacity planning: in long-context, long-task scenarios, these two often become bottlenecks ahead of compute.

    Conclusion

    The common thread across long-horizon coding, hundred-agent coordination, and day-level autonomous operation is that their bottlenecks lie not in “the quality of a single model output” but in the traditional distributed-systems topics of state management, task scheduling, fault recovery, and capacity governance. Design an agent as a distributed job that must run for days—with a DAG, checkpoints, idempotency, quotas, backpressure, and observability—and its capability ceiling opens up naturally. On cloud platforms such as Baidu AI Cloud, general-purpose capabilities like the container platform, object storage, message queue, logging service, and monitoring and alerting form the standard foundation of this infrastructure, letting teams focus their energy on task decomposition and domain logic itself.