1. Case Background: From a Personal Automation System to an Agent Runtime

A product owner with a development background tried to use AI coding agents to continuously refactor a complex system in a personal project. At first, the project was a lightweight automation tool: it downloaded audio content, transcribed it, generated summaries, and built a searchable local content library. As requirements grew, the project expanded to social media content scraping, summary generation, scheduled execution, around-the-clock task execution, and content processing pipelines triggered by email.

After the basic capabilities became more mature, the developer began introducing more complex agent workflows, including deep research, reflection on daily usage records, automatic accumulation of experience, and generating new instructions or skill modules based on reflection results. He also wanted to build an agent runtime capability that would allow different tasks to be registered, scheduled, isolated, and tracked, and that could integrate with different agent execution frameworks.

On the surface, this was a typical scenario for automating a complex software system: clear goals, many modules, reliance on external capabilities, and a need for continuous operation. The developer did not hand-write every line of code. Instead, multiple AI coding agents took turns implementing features. For the first few weeks, the code volume grew quickly and feature modules expanded rapidly. The project appeared to have entered a state of continuous iteration.

2. The Crash After Three Weeks: Apparently Complete, Actually Out of Control

The real problem emerged three weeks later. When the developer systematically read through the codebase again, he found that although the project was rich in features on the surface, its internal quality clearly deviated from the original expectations.

First, the agent architecture was not truly implemented. The system had task scheduling, entry points, and module names, but it lacked stable state management, permission boundaries, task isolation, failure recovery, and traceability mechanisms. Many so-called agent capabilities were merely prompt templates, conditional branches, or placeholder logic.

Second, the skill modules existed only in name. The project claimed that it could automatically accumulate skills, reflect on them, and generate new capabilities. In actual execution, however, the skill generation process did not form a closed-loop evaluation. Most skills were not really tested, had no version management, and had no clear trigger conditions or failure-handling mechanisms.

Third, the test environment accidentally connected to a production inference API. Unit tests directly accessed the real production inference service at runtime instead of a local stub service or an isolated test environment. As a result, every test run triggered real calls and caused unexpected resource consumption.

Fourth, token costs went out of control. In this case, daily token consumption reached about five times the normal daily budget, and around 30 rounds of high-frequency calls continued. Because there were no budget alerts, rate limits, circuit breakers, or call audits, the cost problem was not discovered until settlement or manual inspection.

Ultimately, the project had to be rebuilt from scratch. The reason was not that AI coding agents cannot write code. It was that the entire development process lacked engineering governance: there were no clear specification acceptance criteria, no continuous human review, no environment isolation, no cost constraints, and no runnable testing system.

3. Root Causes: Four Gaps in Agent Development

1. The Semantic Gap Between Specification and Implementation

AI coding agents do not usually understand all requirements and then directly generate a correct architecture. They typically complete tasks based on local context. If the specification is insufficient, they implement a feature in whatever way seems currently most feasible, rather than in line with system-level goals.

For example, a requirement such as “build a runtime that can schedule independent agents and integrate with different execution frameworks” is not sufficient from an engineering perspective. An AI coding agent may implement only a registry, an entry function, or a set of placeholder interfaces. It may formally match the description but lack the components necessary for a real runtime.

The engineering root cause is that the specification remained at the goal level and did not descend to the acceptance level. A truly executable specification should include:

• Input and output formats;
• State storage method;
• Task lifecycle;
• Failure retry strategy;
• Permission boundaries;
• Concurrency limits;
• Observability metrics;
• Rollback plan;
• Test criteria;
• Non-functional metrics such as latency, cost ceiling, throughput, and error rate.

If the specification cannot be translated into acceptable test cases, AI coding agents easily generate code that “looks correct.”

2. The Review Gap Between Generation and Acceptance

Rarely looking at the code for three weeks was one of the key reasons the project became uncontrollable. AI coding agents are good at producing code quickly, but fast output does not mean maintainable output. They may generate a runnable script at one moment and introduce implicit dependencies, duplicated logic, incorrect configurations, or unsafe data access in the next.

When code grows at a pace of dozens of commits per day, problems accumulate quickly like technical debt if humans do not review continuously. This is especially true for modules involving scheduling, authentication, external calls, and state persistence. Even a small misunderstanding can be amplified during later execution.

A reasonable development rhythm should include:

• Ask an AI coding agent to modify only one small module at a time;
• Require every change to produce a reviewable diff;
• Require every module to pass static checks before commit;
• Require every critical path to have unit tests or integration tests;
• Require every non-trivial decision to be recorded as an architecture decision;
• Require every external call to be validated for network access, authentication, and permissions.

AI coding agents can improve coding efficiency, but they cannot replace review responsibility.

3. The Isolation Gap Between Environments and Production

Connecting tests to a production inference API was one of the most dangerous mistakes. In agent systems, inference APIs, model services, retrieval services, vector stores, task queues, and external content sources often require credentials and network access permissions. If test configuration, development configuration, and production configuration are not strictly isolated, test scripts may access real resources.

This is not merely a configuration error. It is a lack of environment governance. A system should clearly distinguish among:

• Local development environment;
• Automated test environment;
• Pre-release environment;
• Production environment;
• Isolated sandbox environment.

Each environment should have independent credentials, independent access policies, independent network boundaries, and independent datasets. The test environment should not hold production credentials, and the development environment should not directly connect to production inference services.

A basic principle is that any call that has not been approved should not access production resources.

4. The Governance Gap Between Cost and Observability

Token consumption is not an abstract bill. It is an engineering metric that can be observed in real time. Without budgets, rate limits, sampling, caching, and alerts, agents may continue consuming resources unnoticed.

In complex tasks, cost overruns usually come from several factors:

• High-frequency calls to real inference APIs;
• Repeated concatenation of long contexts;
• Lack of caching mechanisms;
• Multi-turn tasks without termination conditions;
• Repeated retries of failed tasks;
• Test data mistakenly entering production;
• Agents automatically reflecting and expanding tasks;
• Different tasks triggering one another.

Therefore, agent systems need to treat tokens as managed resources, not as unbounded byproducts of execution.

4. Breakdown of Key Technical Problems

1. “Fake Skills”: Skill Generation Must Establish Closed-Loop Evaluation

A skill should not be just a name, a prompt, or an automatically generated description. A usable skill should meet several conditions:

• It has a clear input;
• It has a clear output;
• It has repeatable trigger conditions;
• It has success criteria;
• It has failure handling;
• It has version records;
• It has real test samples;
• It has invocation permissions;
• It has effectiveness evaluation metrics.

If a system only automatically generates skill names, skill descriptions, or skill cards without an evaluation mechanism, so-called automatic skill accumulation is only an appearance.

A more reliable skill release process could be:

Skill draft
  ↓
Static checks
  ↓
Offline sample testing
  ↓
Permission and cost evaluation
  ↓
Human or automatic review
  ↓
Canary release
  ↓
Runtime monitoring
  ↓
Failure rollback

In engineering implementation, each skill should have metadata and an evaluation status:

skill:
  name: summarize_content
  version: 1.2.0
  input_schema: content_text
  output_schema: summary_object
  test_cases: 20
  pass_rate_threshold: 0.95
  max_token_cost_per_call: 3000
  environments:
    - test
    - staging
  production_allowed: false
  rollback_on_error: true

Only after a skill has been tested, evaluated, versioned, and permission-controlled can it be invoked by an agent.

2. Tests Accidentally Connecting to Production: Configurations, Credentials, and Network Boundaries Must Be Addressed Early

To prevent tests from accidentally connecting to production, defenses should be set up at the code layer, configuration layer, network layer, and release layer.

The first layer is the configuration layer. Test configurations should not contain production inference endpoints or production credentials. All external calls should go through a unified client wrapper rather than hard-coded addresses in test code.

The second layer is the environment validation layer. Before calling an external service, the code should check the current runtime environment:

def guard_external_call(request):
    if request.env == "prod":
        if not request.approved_for_prod:
            raise RuntimeError("Production call blocked.")
        if not request.circuit_breaker_ok():
            raise RuntimeError("Circuit breaker open.")
        log_request(request)
    elif request.env == "test":
        if request.endpoint.startswith("prod-"):
            raise RuntimeError("Test cannot access production endpoint.")

The third layer is the network layer. The test environment should access test services through an isolated network, and production credentials should not be injected into the test environment.

The fourth layer is the release layer. Any call involving production interfaces must go through approval, canary release, or manual confirmation. AI coding agents or automation scripts must not decide on their own which production resources they can access.

3. Token Costs: Turning Budget into a Circuit-Breakeable Resource

Token cost governance should not happen only at the billing stage. It should be embedded in the runtime. At a minimum, it should include:

• Setting task-level budgets;
• Setting user-level budgets;
• Setting system-level daily budgets;
• Sampling high-consumption tasks;
• Caching repeated requests;
• Compressing long contexts;
• Limiting retries for failed tasks;
• Setting maximum rounds for high-frequency loops;
• Alerting on abnormal calls;
• Automatically terminating over-budget tasks.

For example, each task can be given resource limits:

task_guard:
  max_rounds: 5
  max_tokens_per_round: 8000
  max_total_tokens: 40000
  max_runtime_minutes: 15
  on_exceed: terminate_and_notify
  retry_policy:
    max_retries: 2
    backoff: exponential

Further, different cost tiers can be assigned to different tasks:

Low-tier tasks: allowed to be completed by fast, low-cost models;
Medium-tier tasks: allowed more reasoning rounds;
High-tier tasks: require human approval;
Experimental tasks: can only run in isolated environments.

This prevents agents from expanding their behavior without limits. Instead, they always operate within budgets and rules.

4. Agent Architecture: Registration, Scheduling, Isolation, and Tracing

If the goal is truly to build an agent runtime, the core is not to “generate many prompts,” but to establish a runnable and governable execution system. A mature agent system usually includes the following capabilities:

1. Agent registry
   Used to register agent identity, capabilities, versions, permissions, and dependencies.

2. Task scheduler
   Responsible for assigning tasks, controlling concurrency, limiting resources, and handling queues.

3. State storage
   Records task lifecycle, intermediate results, failure reasons, and retry states.

4. Permission control
   Determines whether agents can access data, external services, write operations, or high-cost interfaces.

5. Observability
   Records call logs, latency, token consumption, error rates, success rates, and abnormal traces.

6. Isolation mechanisms
   Different agents, tenants, and environments should not freely share state or credentials.

7. Rollback mechanisms
   When agent behavior becomes abnormal, tasks can be quickly stopped, configurations rolled back, or changes revoked.

8. Audit mechanisms
   All behaviors involving external calls, data writes, permission changes, and cost controls must be traceable.

The goal of this architecture is not to make agents “more free,” but to make them controllable, observable, and recoverable.

5. A Practical Governance Framework for Agent Development

1. Specification-Driven Development: Write Requirements as Acceptance Criteria

Before allowing AI coding agents to modify code, requirements should first be broken down into verifiable items. For example:

Requirement: An agent should select an execution path based on task type.
Acceptance criteria:
1. Unknown tasks must be rejected with a clear error.
2. Registered tasks must produce structured results.
3. A single task may execute for at most 5 rounds.
4. Exceeding the budget must terminate the task and write an audit log.
5. All external calls must record request identifiers and response status.
6. The test environment must not access production credentials.

This kind of specification is closer to engineering reality than “build an agent system.”

2. Small-Step Refactoring: Change Only One Module at a Time

AI coding agents should not refactor the entire codebase at once. A safer approach is:

• Refactor one module first;
• Run tests;
• Review the diff;
• Record decisions;
• Move on to the next module.

Small-step refactoring exposes problems as early as possible and prevents AI agents from continuing to expand along an incorrect path.

3. Three-Layer Testing: Unit, Contract, and End-to-End

Unit tests can only verify local logic and are not sufficient to cover the risks of agent systems. At least three layers of testing are needed:

Unit tests: verify functions, components, rules, and state changes.

Contract tests: verify external interfaces, data structures, response formats, and error codes.

End-to-end tests: verify real task chains, permission boundaries, failure recovery, and cost limits.

For tests involving production resources, production substitutes or isolated sandboxes must be used. Any test that directly connects to production interfaces should be considered high-risk behavior.

4. Cost and Quality Gates: Make Releases Automatically Controlled

Automated gates can be added to the continuous integration process:

Static code checks
  ↓
Unit tests pass
  ↓
Contract tests pass
  ↓
Budget simulation passes
  ↓
Permission scan passes
  ↓
Human review
  ↓
Canary release

If a change causes the estimated cost to exceed a threshold, or if the access scope exceeds the allowed range, the release should be automatically blocked.

5. Human Review Points: Do Not Fully Automate Critical Decisions

AI coding agents can generate code, but they cannot independently bear responsibility for architecture decisions. Human review must be retained in the following areas:

• Data model changes;
• Permission model changes;
• New external calls;
• Production configuration changes;
• Credential and key changes;
• Task scheduling strategy changes;
• Automatic reflection and skill release logic;
• High-cost model invocation strategies;
• Error recovery and rollback mechanisms.

Human review is not the opposite of efficiency. It is a key mechanism for preventing errors from scaling.

6. A Staged Adoption Roadmap for Teams

Phase 1: Pilot Validation

Choose a small task with clear boundaries and controllable risk, such as log summarization, code comments, documentation generation, or test script generation. The goal is to validate the stability of AI coding agents in localized tasks.

Focus on:

• Whether generated code is reviewable;
• Whether tests are stable;
• Whether implicit dependencies exist;
• Whether costs are controllable;
• Whether errors can be quickly located.

Phase 2: Establish Guardrails

Introduce basic guardrails within the team:

• Unified configuration management;
• Unified credential management;
• Unified logging and metrics;
• Unified budget control;
• Unified code review;
• Unified testing standards.

The goal of this phase is not to pursue automation level, but to ensure that automation does not introduce uncontrollable risks.

Phase 3: Standardize Agent Capabilities

After basic engineering capabilities become stable, more complex agent capabilities can be introduced, such as task decomposition, automatic reflection, skill accumulation, and execution orchestration.

At this stage, the following must be built:

• Skill evaluation sets;
• Task tracking;
• Call auditing;
• Cost dashboards;
• Permission centers;
• Exception alerts;
• Rollback processes.

Phase 4: Platform-Level Governance

Eventually, agent capabilities can be incorporated into the engineering governance system of an R&D platform. General cloud capabilities, such as logging services, monitoring and alerting, object storage, message queues, container platforms, and access control, can form a complete agent engineering system that is observable, auditable, and governable.

7. Summary: AI Coding Agents Do Not Replace Engineering; They Amplify Engineering Capabilities

The core lesson from this three-week failure is not that AI coding agents are unusable. It is that when the speed of generation exceeds the speed of review and governance, system risk accumulates rapidly.

AI coding agents can significantly improve code output efficiency, but the premise is that the engineering system must be established first. Specifications must be clear, environments must be isolated, tests must be complete, costs must be controlled, permissions must be minimized, changes must be reviewable, and critical decisions must have human ownership.

For developers, the right way to use AI coding agents is not to hand over all the code. It is to treat them as highly efficient executors that require strict constraints. Humans are responsible for defining goals, validating results, and controlling boundaries. Agents are responsible for generating implementations, executing local tasks, and providing candidate solutions.

Only by placing “generation” within “governance” can AI coding agents evolve from demo tools into reliable engineering assistants, rather than leaving a codebase that needs to be rebuilt weeks later.