Verification-First Workflow for Long-Running AI Agent Development

1. The Premise Has Changed: Coding Is No Longer the Primary Bottleneck

When AI coding assistants first became widely available, teams focused almost entirely on making models generate code faster and in larger volume. In high-intensity, long-running practice, however, a counterintuitive pattern emerges: when a model can produce hundreds of lines of code in tens of seconds, the overall delivery cycle does not shrink proportionally.

The reason is simple: the time structure of a single iteration has changed.

1. Generation: seconds to minutes, and highly parallelizable.
2. Verification: compilation, static checks, unit and integration tests, build artifacts, deployment; minutes to tens of minutes, and strongly serial.
3. Review: humans confirm tradeoffs and risks; minutes to hours, and effectively incompressible.

In other words, end-to-end throughput is constrained by the “verification + review” segment, while model improvements mainly compress the generation segment.

Using a rough Amdahl’s Law-style intuition: if verification accounts for 70% of the time in a single iteration, halving generation time improves overall throughput by about 17%, while halving verification time can improve it by about 35%. This is why, in long-running tasks, the marginal benefit of “spending tokens to optimize tests and validation pipelines” is often much higher than the marginal benefit of “buying a faster generation mode.”

A typical waste pattern looks like this: the agent submits a change for testing, the tests take five minutes, the change is rejected by the quality gate, the agent fixes it, resubmits, and waits another five minutes. If generation becomes 1.5x faster, but every loop still pays a fixed validation cost, only the smallest part of the loop has been shortened. At that point, the benefit of a “fast mode” is diluted to the point of being barely noticeable.

Conclusion: the first priority in workflow redesign is not generation speed. It is shortening the feedback loop and increasing the amount of useful information produced per loop.

---

2. Spec First: Keeping Long-Running Tasks from Drifting

Three common failure modes appear in long-running agent tasks:

• Mid-course solution changes: the agent independently redesigns an interface that was already agreed upon.
• Semantic drift: requirement A is implemented as requirement B.
• Infinite loops: the agent repeatedly tries and rolls back changes in the same place.

These problems usually share the same root cause: intent exists only in the conversation context, and that context can be compressed, truncated, or polluted by later failure logs.

The engineering solution is to externalize intent into versioned documents inside the repository, forming a three-layer structure.

2.1 Three-Layer Document Structure

Layer	File Naming Convention (Illustrative)	Contents	Stability
Requirements	docs/spec/<feature>.md	Goals, non-goals, user stories, acceptance criteria	Changes require review
Design	docs/design/<feature>.md	Module breakdown, data flow, interface contracts, error codes, data migration and rollback strategy	Changes require review
Execution	docs/tasks/<feature>.md	Atomic task list, Definition of Done (DoD), current status	Frequently read and written by the agent

The value of the requirements layer lies in writing down non-goals. “What not to do in this change” constrains the agent’s behavior more effectively than “what to do,” and it is one of the strongest single defenses against mid-course solution changes.

2.2 Contractization: Writing Interfaces Directly into the Documents

The most important artifact to make explicit in the design layer is the contract, because it serves as the agent’s anchor for action. At minimum, specify:

• Function or interface signatures and parameter validation rules;
• Return structures and error-code enumerations;
• Boundary conditions such as empty collections, zero values, concurrency, idempotency, and duplicate requests;
• Persistence impact, including whether new fields are added, whether migrations are required, and whether changes can be rolled back.

{
  "interface": "OrderService.refund(orderId, amount, requestId)",
  "preconditions": ["orderId exists", "amount > 0 and <= refundable amount"],
  "returns": "{ ok: bool, refunded: int, reason?: string }",
  "error_codes": ["ORDER_NOT_FOUND", "AMOUNT_EXCEEDED", "ALREADY_REFUNDED"],
  "invariants": [
    "Repeated calls with the same requestId produce the same result (idempotency)",
    "refunded_total <= order.amount"
  ],
  "out_of_scope": [
    "No split partial-refund support",
    "No external payment gateway integration"
  ]
}

2.3 Task Granularity: One Verifiable Conversation at a Time

Each task in the execution layer should satisfy three conditions:

• It can be completed within a single conversation or session.
• Its completion state can be verified automatically.
• It can be independently rolled back if it fails.

If tasks are too large, context is crushed by history. If tasks are too small, orchestration overhead exceeds the benefit. A practical rule of thumb is that a task should usually touch 1 to 5 files and be directly judged by one test or a small set of tests.

The Definition of Done (DoD) should be a fixed template so that the agent’s claim of “I am done” has objective meaning:

• All related unit and integration tests pass;
• All new or modified public interfaces have corresponding test coverage;
• Static checks and type checks introduce no new warnings;
• The change summary, impact scope, and rollback method are written into the commit message and PR description;
• If data migration is involved, both forward migration and rollback paths have been verified.

---

3. Task Layering and Context Hygiene

3.1 Separate the Orchestrator from the Executor

A single long conversation that handles “understanding requirements + coding + debugging + verification” is one of the most unstable patterns. A more stable structure is layered:

• Orchestration layer: maintains specification documents and the task list, decides what to do next, and judges whether acceptance criteria are met. It does not directly write business code.
• Execution layer: each subtask runs in an independent conversation or subagent, receiving only the task description, relevant contracts, and acceptance commands. It produces code changes and test evidence.
• Validation layer: automated pipelines perform verification and do not rely on the model’s self-judgment.

This turns the uncontrollable problem of “context length” into the controllable problem of “task partitioning.” Each execution session is short-lived, its context remains clean, and failures only require rerunning that specific subtask.

3.2 Three Hard Rules for Context Hygiene

1. Externalize state: write all progress into docs/tasks/*.md. After a session restarts, the first step is to read the document, not the conversation history.
2. Compress failure history: carry only the “currently unresolved failure signature + list of attempted solutions” into the next round, rather than dumping full failure logs back into the context.
3. Solve one type of problem at a time: do not change business logic and debug test infrastructure in the same session. Otherwise, failure causes become impossible to attribute.

---

4. Test-Driven Development: Turning Acceptance Criteria into Executable Assertions

If verification is the bottleneck, then “what to verify” must be explicitly designed, not added after the fact.

4.1 Change the Order: Write Tests First

In agent-collaboration mode, “tests first” means something different from traditional manual development. Tests are not only a quality mechanism; they are also the measuring instrument that gives the agent closed-loop feedback.

If humans or the orchestration layer write assertions first and then ask the execution layer to implement them, teams can avoid the self-certifying loop where the model writes its own tests and then declares itself successful.

# Define executable acceptance criteria first, then let the execution layer implement them
def test_partial_refund_keeps_order_consistent():
    order = Order(order_id="A1", amount=1000, currency="CNY")
    result = order.refund(300)
    assert result.ok
    assert result.refunded == 300
    assert order.refunded_total == 300
    assert order.status == "PARTIALLY_REFUNDED"
def test_refund_is_idempotent():
    order = Order(order_id="A2", amount=500, currency="CNY")
    order.refund(500)
    again = order.refund(500, request_id="same-request")
    assert not again.ok
    assert again.reason == "ALREADY_REFUNDED"
    assert order.refunded_total == 500   # Idempotency: a repeated request does not change state

4.2 Layer Tests and Locate Problems Correctly

• Unit layer: pure functions and business rules; millisecond-level; largest in number; no external dependencies.
• Integration layer: boundary behavior involving databases, message queues, and external interfaces, including idempotency and concurrency scenarios.
• Contract layer: whether request/response shapes and error codes match the design document.
• End-to-end layer: critical user paths; smallest in number and slowest to run.

The rule is clear: if something can be asserted at the unit layer, do not push it to the integration layer. This is one of the most direct ways to shorten the feedback loop.

4.3 Use Coverage Correctly

Coverage should be treated as a blind-spot detection signal, not as a performance target. Several practical guidelines:

• Focus on branch coverage and diff coverage for critical modules, rather than obsessing over the global line-coverage number.
• Global numbers can be inflated by tests with empty assertions or snapshot tests with no meaningful assertions. Therefore, quality gates should also enforce a minimum ratio of assertions to tests.
• Introduce mutation testing as a periodic health check. It injects artificial defects to verify whether the tests can actually catch them, helping identify cases that “look covered but are ineffective.”
• Define explicit exclusions. Generated code, stub implementations for third-party contracts, and pure configuration declarations should not count toward the denominator. Otherwise, teams may distort design decisions just to improve a number.

Three common anti-patterns deserve special attention:

• Writing assertion-free tests merely to increase coverage;
• Testing only the happy path and ignoring boundary conditions;
• Replacing test doubles with many real dependencies, causing the unit-test layer to degrade into a slow integration-test layer.

---

5. Validation Infrastructure: Self-Hosted Runners and Test Routing

When commit frequency reaches dozens per day, managed shared build resources often become a new queueing bottleneck. A cost-effective approach is to rent a pay-as-you-go cloud host from a cloud provider and self-host a CI execution node.

The motivation is not primarily cost savings; it is control:

• Eliminate queueing: the execution node is dedicated and not preempted by other tasks.
• Persist dependency and build caches: cache directories remain on disk, avoiding cold-start dependency reinstalls on every run.
• Customize the environment: container images, system dependencies, and data volumes can be fixed according to the project’s needs.
• Control costs: the node can be shut down during off-peak hours or started by event-driven scheduling policies.

One caveat: self-hosting shifts the security boundary from the platform to your own team. Therefore, it must be accompanied by strict controls:

• Network isolation between execution environments and development environments;
• Credentials injected through secret management rather than written to disk in plaintext;
• Build artifacts signed and verified;
• Runner images updated regularly.

5.1 Test Routing: Make the Feedback Loop as Short as Possible

Running the full regression suite on every change is one of the biggest killers of feedback speed in large projects. A more practical approach is to route changes to the smallest sufficient test set based on what changed.

{
  "routes": [
    { "match": ["src/api/**"],         "suite": ["unit", "contract", "integration-api"] },
    { "match": ["src/worker/**"],      "suite": ["unit", "integration-worker"] },
    { "match": ["db/migrations/**"],   "suite": ["unit", "migration-verify"] },
    { "match": ["docs/**", "**/*.md"], "suite": ["lint"] }
  ],
  "fallback": "full-regression",
  "policy": {
    "cross_cutting": ["schema", "auth", "config"],
    "on_cross_cutting_change": "full-regression"
  }
}

The key is the cross_cutting whitelist. If data models, authentication, or configuration structures are modified, the system should trigger full regression regardless of where the changed files are located. This rule ensures that routing-based acceleration does not come at the cost of missed coverage.

5.2 Tiered Quality Gates

Stage	Trigger	Contents	Target Feedback Time
Fast feedback	Every push	Static checks, type checks, unit tests	1–3 minutes
Pre-merge	Pull request review request	Integration tests, contract tests, routed relevant suites	5–10 minutes
Post-merge	Changes to the main branch	End-to-end tests, smoke tests, performance baselines	Asynchronous execution with alerts on failure

Useful acceleration mechanisms include:

• Sharding unit tests by test case and running them in parallel;
• Reusing dependency and compilation caches;
• Rerunning failed tests first, combining fail-fast and fail-rerun strategies;
• Setting explicit timeouts for long-running tests;
• Isolating flaky tests instead of repeatedly retrying them.

---

6. Automated Pull Requests and Review Gates

When agents automatically create branches, commit changes, and open pull requests, human attention should shift from “reviewing code” to “reviewing solutions.” This means the review process must be structured; otherwise, a large volume of pull requests will overwhelm reviewers.

Pull request template (populated by automation, with fields the agent cannot skip):

• Change summary and corresponding task ID;
• Impact list, including interfaces, data tables, configuration items, and downstream callers;
• Test evidence: executed test suites, number of passing tests, coverage diff, and mutation-testing results if available;
• Risks and rollback plan: whether data migration is involved, whether gradual rollout is possible, and the rollback command;
• Explicit statement of “behavior not included in this change,” aligned with the non-goals in the requirements document.

Human review should focus on four categories of problems that machines still find difficult to judge:

1. Whether the abstraction is reasonable, and whether temporary patches were introduced only to make tests pass;
2. Concurrency, idempotency, transaction boundaries, and consistency semantics;
3. Error-handling paths and observability, including whether logs, metrics, and tracing are complete;
4. Security boundaries, including input validation, permission checks, secret handling, and unauthorized access.

If the template forces the inclusion of “test evidence,” the agent cannot pass the gate with vague claims such as “done.” That alone is a low-cost quality improvement.

---

7. Guardrails for Preventing Agent Runaway

Long-running unattended operation will inevitably encounter loss of control. Guardrails must be put in place in advance.

• Hard limits: maximum tool-call rounds per task, maximum wall-clock duration, and maximum consecutive failures.
• Loop detection: identify repeated failure signatures, such as the same error, same file, and same modification pattern. If the same failure occurs twice in a row, escalate to a human instead of attempting a third retry.
• Failure classification and policy:
  • Syntax errors, formatting issues, missing imports, or a single assertion failure: allow automatic repair and limited retries.
  • Design-level failures, such as ambiguous requirements, contract conflicts, or the need to change the data model: stop immediately, output the problem list, and wait for human decision-making.
• Least privilege: prohibit direct access to production environments and production data; restrict command execution to a whitelist; limit network access; require human confirmation for destructive operations such as force-push, branch deletion, or bulk history rewriting.
• Observability: session logs, token consumption, phase durations, and failure-cause distributions must be aggregatable. Otherwise, it is impossible to answer the question: “Where did the time actually go?”
• Prioritize rollback capability: land all changes as independent commits. Do not mix multiple tasks into a single commit.

---

8. Fragmented Time and Remote Collaboration

High-intensity collaboration often happens outside the traditional workstation. Therefore, the development environment itself needs to be cloud-based and session-persistent.

• Dev containers and remote hosts should remain in the cloud. The local machine is used mainly for editing and viewing, so the environment is not affected by switching devices.
• Mobile devices should only be used for three things: checking progress, making small decisions such as whether to merge or escalate to a human, and starting the next task. Avoid writing code on devices with low interaction efficiency.
• Commits should be small enough that work can be interrupted at any time and restored to a known state from the documentation and recent commits.
• Scheduled jobs should run long regression suites and end-to-end tests during unattended periods, so that results are ready when people return to their desks.

One warning: treating online time as a goal is unsustainable and can easily be misread as an efficiency metric. What matters more is tracking feedback-cycle length, rework rate, and change failure rate. These metrics better reflect whether the process has actually become faster.

---

9. ROI Priorities and Rollout Path

With limited engineering effort and token budget, invest in the following order:

1. Test infrastructure and fast-feedback loops: highest return, and reusable over the long term.
2. Static checks, type checks, and formatting gates: remove entire classes of problems from human review.
3. Observability: logs, metrics, and tracing enable agents to perform self-diagnosis.
4. Specification and task document systems: determine whether long-running tasks drift.
5. Self-hosted execution nodes and test routing: benefits become visible when commit frequency is high enough.
6. Generation-side speed optimization: lowest priority, because its marginal benefit is most easily diluted by validation.

A phased rollout plan:

• Week 1: establish the core quality gate trio — formatting, static checks, and unit tests — and require all commits to include tests. Do not chase coverage numbers at this stage.
• Weeks 2–3: introduce the three-layer document system for specs, design, and tasks, plus the DoD template; set up self-hosted execution nodes and tiered quality gates; integrate the automated pull request template.
• Week 4 and beyond: enable test routing and the cross-cutting change whitelist; introduce periodic mutation-testing health checks; build a guardrail metrics dashboard covering failure-cause distribution, loop-interception counts, and feedback-cycle percentiles; isolate and govern flaky tests.

---

Conclusion

The productivity gains brought by AI agents must ultimately be realized through stable delivery. When writing code is no longer expensive, the center of engineering shifts from “implementation” to “defining correctness” and “proving correctness.”

Write intent as specifications. Write acceptance criteria as executable assertions. Compress feedback loops to the minute level. Use guardrails to stop failures before they become runaway incidents.

After doing these four things, the human role also changes: from a person who writes code to a person who designs systems, defines standards, and owns judgment. Over the long term, that is exactly the position that becomes harder to replace.