Scaling to 1,000 AI Agents: Formal Verification and System Engineering

A notable shift is taking place in the evolution of large language models. The competitive focus is no longer limited to a single model’s score on an individual benchmark. Increasingly, the question is whether models can be organized into large agent teams that continuously produce verifiable results over long-running, highly constrained tasks.

Two publicly discussed examples are especially representative. One involves fully formalizing a mathematical proof. The other starts with an empty directory and builds a bootable system. The first tests a strict verification loop; the second tests long-running engineering orchestration. Together, they show that agent collaboration is moving from demonstrations toward production engineering.

The core challenge is not making more agents talk at the same time. It is building a collaboration system that supports decomposition, verification, rollback, and measurement.

1. From Single-Model Capability to Multi-Agent Orchestration

Improving the capability of individual models remains important. A roughly 30% increase in output speed and a roughly 20% reduction in per-call cost can directly reduce the marginal cost of high-frequency workloads.

However, single models face limitations on complex tasks, including context-window constraints, long-range consistency problems, accumulated errors, and weak task decomposition. Multi-agent orchestration addresses these limitations by dividing a large goal into many parallelizable and verifiable subtasks, then creating a feedback loop through schedulers, verifiers, and scoring systems.

The main challenges of coordinating thousands of agents include:

1. Task decomposition quality: If tasks are too broad, they become difficult to verify. If they are too granular, coordination costs can grow rapidly.
2. Stable interface contracts: Agents need explicit input and output formats so that intermediate artifacts can be merged reliably.
3. A reliable verification loop: Systems cannot depend on model self-evaluation. They need deterministic checkers, test cases, or formal verification tools.
4. Long-running state management: Tasks may continue for hours or days, requiring persistent plans, artifacts, failure reasons, and budget records.

The following two cases represent the contrasting requirements of strong verification and long-running engineering.

2. Case Study: Formalizing a Mathematical Proof

Formalization translates a mathematical proof into a language that a computer can check step by step. Unlike a natural-language proof, a formal proof cannot rely on vague transitions such as “obviously” or “similarly.” Every step must be validated by a checker.

This makes formal mathematics an effective stress test for agent collaboration. Agents must not only generate proof attempts, but also revise them based on precise verifier feedback.

Publicly described experiments indicate that a team of thousands of agents completed the formalization of a well-known theorem in approximately 30 hours. By comparison, a weaker configuration completed only about one-quarter of the work in 18 hours before progress nearly stopped. The comparison suggests that model capability and orchestration strategy are equally important.

The process is also significant. The agents reportedly did not use mathematical reference materials. They had only a list of intermediate theorems for scoring, while the agents themselves determined how to divide the work, with very limited human intervention.

2.1 Breaking Down the Collaboration Workflow

A reusable formalization workflow generally includes the following stages:

• Goal decomposition: Divide the main theorem into lemmas, auxiliary definitions, and intermediate conclusions.
• Parallel proof search: Assign multiple agents to explore different lemmas and generate candidate proof fragments.
• Deterministic checking: Use a formal proof checker to determine whether each fragment is valid.
• Scoring and selection: Score candidates according to intermediate-theorem completion, checker pass rate, and the number of unresolved dependencies.
• Rescheduling: Further decompose failed fragments or allocate more resources to critical-path tasks.
• Merging and regression checking: Merge validated fragments and rerun the full verification process to detect interface inconsistencies.

A simplified workflow can be represented as follows:

goal = load_theorem()
plan = decompose(goal)
context = load_intermediate_theorem_list()
for round in range(max_rounds):
    candidates = agents.parallel_generate(plan, context)
    checked = verifier.check(candidates)
    score = scorer.evaluate(checked, context)
    plan = scheduler.update(plan, checked, score)
    persist(plan, checked, score)
    if verifier.all_passed(checked):
        break
final_proof = merge(checked)
assert verifier.check(final_proof)

2.2 Engineering Principles

Put the verifier before the generator. In formalization tasks, generation can be expensive while verification is comparatively manageable. The verifier must be authoritative. Every model output should pass through the checker; model confidence cannot be treated as proof of correctness.

Use intermediate theorem lists as scheduling inputs. Such lists serve both as progress metrics and as a basis for resource allocation. Without them, agents may repeatedly explore local problems without making global progress. With them, a scheduler can identify the critical path and direct more agents toward blocked areas.

Place boundaries around self-organized collaboration. Allowing agents to decide how to divide work can improve flexibility, but the system should limit file-write scope, dependency-change permissions, and maximum retry counts. Otherwise, duplicate work and state contamination become likely.

Store failure cases in a structured format. Failed proof fragments, verifier error messages, and retry counts should be recorded for future scheduling decisions. Without structured failure memory, thousands of agents can repeatedly make the same mistakes.

3. Case Study: Building a Bootable System from an Empty Directory

A more engineering-oriented demonstration starts with an empty folder and builds an operating system. Public descriptions indicate that engineers orchestrated more than 1,000 sub-agents over eight consecutive days. The final result could boot on a laptop and included drivers, file storage, windows, a login interface, an installer, and an embedded assistant.

The importance of this example is not the operating system itself. It is the way the project demonstrates long-running, multi-agent engineering organization.

3.1 Task Characteristics

Unlike formal proof, system development does not have one checker capable of determining global correctness. Instead, it relies on many local tests, build checks, and boot validations.

The task has several defining characteristics:

• Multiple layers: Boot processes, drivers, storage, processes, windows, and installers have complex dependencies.
• Slow feedback: A complete build and boot test may take several minutes or longer.
• Many interfaces: Modules require stable contracts to avoid extensive merge conflicts.
• Long execution cycles: Projects lasting several days need checkpoint recovery and automated retry support.

3.2 The Engineering Feedback Loop

A practical orchestration model can include:

1. Role-based agents: Architecture agents define module boundaries, implementation agents write code, testing agents create test cases, and review agents check interface consistency.
2. Contract-first development: Each module should produce interface definitions, dependency lists, and acceptance criteria before implementation begins.
3. Continuous builds: Every merge should trigger a build and a minimal boot test. Failures should cause an automatic rollback.
4. Persistent state: Plans, branches, build results, failure logs, and budget usage should be stored persistently.
5. Human approval gates: High-risk changes—such as boot flows, permission models, and installer operations that write to disk—should require human review.

An illustrative configuration is shown below:

project: minimal_system
agents:
  architect:
    outputs: [module_boundaries, interface_contracts]
  implementer:
    inputs: [interface_contracts]
    outputs: [source_code, unit_tests]
  tester:
    inputs: [source_code]
    outputs: [build_result, boot_test_result]
  reviewer:
    inputs: [source_code, interface_contracts]
    outputs: [review_report]
policies:
  max_retry: 3
  require_build_before_merge: true
  require_boot_test: true
  human_gate: [bootloader, permission_model, installer]

3.3 Key Challenges in Long-Running Tasks

Long-running tasks usually fail not because a model cannot write code, but because project state drifts over time. Across several days, agents may repeatedly modify the same module, causing interface mismatches, dependency conflicts, and reduced test coverage.

The system therefore needs:

• Versioned intermediate artifacts: Every build result should be traceable, and failed changes should be reversible.
• Observability: Record each agent’s task, execution time, token usage, failure reason, and artifact hash.
• Budget controls: Set usage limits for different modules so that one blocked task cannot consume the entire budget.
• Incremental verification: Run unit tests and static checks before full boot tests to reduce feedback latency.

4. Performance, Cost, and Quota Policies

Publicly described information indicates that this class of model can provide approximately 30% higher output speed and approximately 20% lower per-call cost. For developers, this means shorter response times for frequent coding assistance and lower unit costs for batch agent workloads.

However, lower per-call cost does not necessarily mean lower total cost. Multi-agent orchestration can multiply the number of calls. Effective cost control requires tracking both model pricing and task retry rates.

Quota policies are also relevant. Public information describes a more flexible usage model in which certain subscription tiers can allocate part of their weekly quota to a new model during a designated activity window without an additional charge, sometimes with a quota reset included. Higher tiers may receive additional reset opportunities based on subscription duration. These benefits are plan-specific and may depend on availability, so users should verify the applicable terms in the official product documentation.

The implications for developers include:

• Plan capacity by model: Model-specific resets mean quotas may not be fully interchangeable. Confirm quota boundaries before starting large experiments.
• Use activity windows for load testing: If part of a weekly quota can be allocated to a new model, reserve the remaining capacity for production workloads.
• Treat reset credits as limited resources: Allocate them according to project priority, especially for tasks on the critical path.
• Monitor costs at the agent level: Track calls, retries, and output quality for each subtask to avoid a situation in which lower unit prices lead to higher total spending.

5. Practical Recommendations for Developers

5.1 Build the Verification Loop Before Scaling the Agent Count

Do not begin by targeting thousands of agents. First use 10 to 20 agents to validate task decomposition, interface contracts, and verifier reliability. Then increase the scale gradually.

Scaling without a verification loop only amplifies errors.

5.2 Use Contracts to Constrain Collaboration

Every subtask should define its inputs, outputs, acceptance criteria, and maximum retry count. Interface contracts should be versioned, and automated checks should run before merging.

For coding tasks, static analysis, unit tests, and minimal builds should serve as merge gates.

5.3 Treat Long-Running Work as a Workflow System

Long-running tasks require persistence, retry handling, rollback, logging, and budget controls. A practical architecture should separate the agent runtime from business logic, use a message queue for task distribution, store intermediate artifacts in object storage, and use logging and monitoring services for observability.

5.4 Address Security and Compliance Early

Agents may execute code, write files, or call external services. Systems should enforce least-privilege access, credential isolation, network allowlists, and human approval gates.

High-risk operations—such as writing to disk through an installer, modifying the permission model, or loading drivers—should require mandatory human review.

5.5 Start with Well-Bounded Programming Tasks

Programming remains one of the most valuable application areas for these models. Developers should begin with clearly bounded tasks such as code completion, unit-test generation, defect repair, and interface-documentation synchronization.

They can then expand to cross-module refactoring and system construction. Each scenario should have measurable acceptance criteria, such as build pass rate, changes in test coverage, rollback rate, and the amount of manual rework required.

6. Using Baidu AI Cloud for General Infrastructure

When deploying an agent orchestration system, stable infrastructure is essential. Baidu AI Cloud can provide general-purpose infrastructure capabilities for workloads of this type: container platforms can run agent runtimes and verifiers; object storage can retain intermediate artifacts and build results; message queues can decouple task distribution; logging and monitoring services can provide end-to-end observability; and key management can protect credentials.

These services can also be integrated with continuous integration and continuous delivery workflows to enforce merge gates. Actual service selection should be evaluated against compliance requirements, budget, and the team’s technology stack. An agent platform should not become an unobservable black box.

7. Conclusion

The real barrier to coordinating thousands of agents is not increasing the number of model calls. It is building an engineering loop that supports decomposition, generation, verification, scoring, scheduling, and persistence.

The formal proof case shows how deterministic verifiers and intermediate progress lists can convert model capability into checkable output. The system-building case shows that long-running tasks require contracts, rollback, observability, and budget controls.

For developers, the most practical path is to validate the complete workflow on small tasks before gradually increasing the number of agents. Lower per-call costs and more flexible quota policies can support experimentation, but verification, security, and cost monitoring must come before scale.