Autonomous Coding Agents: Capability Boundaries and Safety Governance Practices

1. The real issue behind the behavior: interaction paradigm, not “who is smarter”

Many developers explain differences in feel as “which model is stronger.” From an engineering perspective, the more important difference is the interaction paradigm:

• Single-turn request-response paradigm: The model only answers what is asked and does not take extra actions. This offers strong controllability, but humans must prepare the full context.
• Goal-planning-execution-verification paradigm (Agent paradigm): After receiving a goal, the model decomposes tasks, calls tools, reads the environment, verifies results, and iterates. Humans only need to provide intent and constraints.

Reported behaviors such as “it did a few more steps than expected,” “it monitored processes on its own,” and “it initiated cleanup itself” are the expected result when the second paradigm is granted read/write access to the runtime environment. This is not mysterious; it is the product effect of permissions × planning capability × context density.

Understanding this helps move the discussion from emotional “useful or not useful” judgments to engineering questions that can be designed, evaluated, and governed.

2. Three technical sources of proactive behavior

2.1 Explicit planning and reflection loops

A typical implementation is the loop: plan → execute → observe → reflect → correct. The agent breaks a large goal into a subtask list, executes each step, reads the tool return value, and then decides the next step. This explains the impression that the agent “plans several steps ahead”: the plan is an explicitly generated intermediate artifact, not a hidden guess.

An important engineering point is that the visibility of the plan directly determines controllability. If the plan exists only inside the model, humans cannot intervene before a destructive action occurs. If the plan is output in a structured form—such as a task list, a list of commands to execute, or a list of affected files—it naturally becomes an approval interface.

2.2 Tool access and runtime permissions

The reason an agent can “discover malware” or “track processes” is simply that it has permission to execute system commands, enumerate processes, and read network connections. Technically, this is very ordinary:

# These are generic capabilities provided by the operating system; the agent is only authorized to call them
ps aux                      # Enumerate processes
ss -tunap || netstat -tunap # View network connections and outbound endpoints
ls -la /proc/<pid>/fd       # View files opened by a process

The more important implication is that any capability callable by the agent can be triggered by an injected piece of text. The larger the permission surface, the larger the attack surface, regardless of whether the model is “smart.”

2.3 Context accumulation and preference persistence

The feeling that “it understands my habits without being told” usually comes from three types of context:

1. Project-level conventions: technology stack, directory structure, naming conventions, and testing requirements;
2. Decision records: why option A was chosen over option B, and which options have already been rejected;
3. Historical modification style: commit granularity, comment habits, and error-handling patterns.

This also explains the observation that “the more I prepare, the stronger it becomes”: context density determines first-attempt success. Therefore, context engineering should be treated as a first-class engineering asset: structured, versioned, and reviewable, rather than scattered across chat logs.

A recommended practice is to encode conventions in the repository so that they evolve with the code:

# Project conventions file (example)
## Technology stack and versions
- Runtime version, package manager, lockfile policy
## Directory and layering constraints
- Cross-layer direct references are prohibited; shared capabilities must go through designated modules
## Coding standards
- Error-handling methods, logging field standards, forbidden APIs
## Verification requirements
- Minimum test suite and static checks that must run after every change
## Forbidden actions (hard constraints)
- Do not modify CI configuration or dependency lockfiles
- Do not execute deletion, data exfiltration, or installation commands

3. The overlooked part: calibrating output certainty

Complaints that “no matter how I ask, it does not speak plainly and always insists on balance” are technically an issue of calibrating uncertainty disclosure against actionability, not a language-style preference.

An effective approach is to separate the two into layers rather than mixing them into one narrative:

• Layer 1: Actionable conclusion (give the answer, command, or judgment first);
• Layer 2: Conditions and boundaries (when the conclusion does not hold);
• Layer 3: Verification method (how the user can confirm it).

Conclusion: Suspect connection pool exhaustion first, not slow queries.
Conditions: If QPS has not increased but P99 rises sharply, connection leakage is more likely.
Verification: Compare time-series curves of active connections and waiting queue length.
Uncertainties: Runtime metrics are missing; GC pauses cannot be ruled out; sampling is required to confirm.

Prompt constraints for the agent can be more direct:

Answer rules:
1. Give the conclusion and actionable steps first, then the reasoning;
2. Disclose uncertainty only when it affects a decision, and identify which decision it affects;
3. Do not repeat the same risk warning unless there is new evidence;
4. If information is insufficient, list the minimum information required instead of providing generic filler.

These constraints can significantly reduce information-density loss and reduce rework caused by answers that cannot be used.

4. Four risk categories of proactive behavior (governance focus)

Risk category	Specific examples	Impact level
Unauthorized and irreversible actions	Deleting files, bulk cleanup, uninstalling, force pushing, resetting configuration	High (cannot be rolled back)
Damage driven by false positives	Treating a suspected anomaly as a conclusion and remediating automatically	High (legitimate workloads may be killed)
Prompt injection and supply-chain risk	Instructions embedded in web pages, dependency documentation, or log content	High (can be triggered remotely)
Credential and data leakage	Reading .env files, secret files, or internal addresses into context or logs	High (compliance risk)

Among these, damage driven by false positives is the most easily underestimated. If the “automatically cleaned seven malware samples” scenario occurred in production, an equivalent situation could be: the agent judges a process to be abnormal, then terminates a business process, clears a cache directory, and rolls back configuration. If the judgment logic is wrong once, the loss is real.

Therefore, there is one non-negotiable principle: by default, reject remediation actions in production; detection and remediation must be decoupled, and remediation requires a human gate.

5. Autonomy level model: making the degree of hands-off configurable

Do not choose only between fully automatic and fully manual. Authorize according to action type.

Level	Permission scope	Typical scenarios
L0	Suggest only, no execution	Architecture review, code explanation
L1	Read-only investigation	Read code, run tests, view logs and metrics
L2	Controlled writes inside the workspace	Edit code, generate tests, commit local branches
L3	External side effects requiring human confirmation each time	Install dependencies, change configuration, access external APIs
L4	Unattended operation (allowlisted tasks only)	Scheduled dependency upgrades, batch documentation fixes

For implementation, separate action classification from authorization policy. Manage the policy with a declarative file and include it in code review:

# agent-policy.yaml (example; field names may vary by tool)
version: 1
defaults:
  autonomy: L2
  require_plan_for: [destructive, network_write, install]
allow:
  - read: ["src/**", "tests/**", "docs/**"]
  - write: ["src/**", "tests/**"]
  - exec: ["<build_tool>", "<test_tool>", "<static_check_tool>"]
deny:
  - exec: ["rm -rf *", "dd *", "chmod -R 777 *"]
  - path: ["**/.env*", "**/secrets/**", "**/*credential*"]
  - network: ["*"]          # Deny network by default; allow exceptions only when needed
approval:
  install_dependency: human
  config_change: human
  deploy: human

The gate logic can be implemented with a small amount of code. The key is to place the decision outside the agent:

# Action risk classification and human gate (example logic)
DESTRUCTIVE = {"delete", "purge", "overwrite_config", "kill_process", "push_force"}
NETWORK_WRITE = {"http_post", "upload", "deploy"}
def authorize(action, policy, env):
    if action.kind in DESTRUCTIVE and env != "dev":
        return "REJECT"                      # Production rejects destructive actions by default
    if action.kind in NETWORK_WRITE and not policy.allows(action.target):
        return "REJECT"
    if action.needs_admin() and not action.approved_by_human:
        return "HUMAN_REQUIRED"              # Privilege escalation requires human approval
    if policy.allows(action):
        return "ALLOW"
    return "HUMAN_REQUIRED"

6. Isolation, credentials, and audit: three infrastructure blocks

Isolation: Run the agent in a disposable container or an independent working copy. Mount the code directory read-only and keep the workspace writable. Route outbound network traffic through an allowlist proxy. Do not mount host credentials. Destroy the environment when the task ends. This way, even if an instruction is injected, the blast radius is limited to a disposable environment.

Credentials: Do not inject long-lived secrets into the agent. Use short-lived, least-privilege temporary tokens issued and revoked by a credential broker outside the agent. Redact sensitive values uniformly in logs and context.

Audit: All tool calls must be recorded. Recommended fields include action, target, parameter summary, decision, approver, duration, and result status:

{
  "trace_id": "b7f2...",
  "session_id": "s-2026-...",
  "action": "process_terminate",
  "target": "pid:18422",
  "args_digest": "sha256:...",
  "risk_level": "destructive",
  "decision": "HUMAN_REQUIRED",
  "approved_by": "user:zhang",
  "env": "prod",
  "result": "denied",
  "timestamp": "2026-08-26T21:30:11Z"
}

Send audit data to a centralized logging service. Configure separate monitoring and alerts for high-risk actions: deletion actions exceeding a threshold within a single session, access to credential paths, and privilege-escalation requests should trigger immediate alerts and pause the session.

If a team is already using Baidu AI Cloud, these capabilities can map to general product categories: a container platform for isolated execution environments, object storage for workspace snapshots, a log service for aggregating audit events, monitoring and alerting for high-risk action rules, and identity and access management for least privilege.

7. Rollback capability matters more than “can it do it”

Every write operation performed by an autonomous agent should produce a rollback unit:

• Code changes: Create a branch or worktree first; do not write directly to the main branch;
• Configuration changes: Save a diff patch before and after the change;
• File operations: Change deletion to moving into an isolated directory with a retention window;
• Data operations: Back up a snapshot before executing.

One core principle: rewrite destructive operations as reversible operations. Changing “delete” to “move into a recycle directory and mark it retained for 24 hours” immediately lowers the risk level by an order of magnitude.

8. Evaluation and acceptance metrics before launch

Do not judge usability by subjective feel. Use metrics:

1. First-pass rate: The proportion of tasks that pass without manual rework;
2. Human intervention rate: The number of human interventions required per 100 actions;
3. Privilege-escalation interception rate: The proportion of injection or privilege-escalation test cases correctly blocked by policy (must be close to 100%);
4. False-positive rate: The proportion of anomalies judged by the agent that turn out not to be real;
5. Rollback rate and rollback time;
6. Context hit rate: The proportion of first attempts that correctly used project conventions;
7. Invalid action rate: The proportion of tool calls that do not contribute to the result (measuring cost and noise).

Maintain a fixed regression test suite. It must include security cases: embed malicious instructions into dependency documentation, log text, and web page content to verify whether the agent executes them; place credential files in the workspace to verify whether the agent reads and exfiltrates them. These cases should enter CI just like unit tests.

9. Why users keep investing: switching costs and asset accumulation

From an engineering perspective, “long-term renewal” is often not brand loyalty, but sunk assets:

• Project conventions stored in context and rule files;
• Prompt and workflow habits formed by the team;
• Historical evaluation sets and tuning experience;
• Adaptation costs for surrounding toolchains.

This leads to two conclusions. For individuals and teams, context assets should be as portable as possible: use plain-text convention files, decouple rules from tools, and maintain evaluation sets independently, avoiding lock-in to a single tool’s proprietary format. For vendors, retention depends on continuously reducing “switching friction” rather than creating lock-in. This is also a useful question to ask when evaluating any development tool.

10. Implementation checklist

• Define the agent’s autonomy level and write it into team norms;
• Use a declarative policy file to manage allow/deny lists and include it in code review;
• Decouple detection from remediation; remediation actions require human confirmation;
• Production environments deny deletion, cleanup, and privilege-escalation actions by default;
• Isolate the execution environment; deny network access by default and allow outbound access only through an allowlist;
• Use short-lived, least-privilege credentials and redact logs uniformly;
• Audit all tool calls and configure real-time alerts for high-risk actions;
• Make all write operations rollbackable; change deletion into reversible operations;
• Build a regression evaluation set that includes security cases and connect it to CI;
• Periodically review permission policies and narrow them according to the principle of least privilege.

Conclusion: The value of autonomous coding agents comes from permissions and context, and the risks come from the same two sources. By making the degree of hands-off a configurable, auditable, and rollbackable engineering parameter, efficiency gains can be achieved without sacrificing system safety.