Remotely Controlled AI Coding Agents: Control Plane Architecture and Engineering Practices

1. Where the Problem Starts: The Control Plane Is Moving, Not the Model

Over the past year, the way people interact with coding agents has gone through three clear stages. First was inline assistance in the local terminal and editor: the human sits at the machine, and the agent acts as an “autocompleter.” Second was autonomous execution from a resident desktop process: the agent holds filesystem and command-execution rights, while the human steps back into the role of reviewer. The third stage is happening now: the control plane is separating from the execution machine. Instructions are issued from a personal device, execution happens on another always-on workstation, and results are returned to the phone as structured summaries.

One judgment here is easy to misread, so it is worth stating plainly: the technical core of this change is not a leap in model capability but a relocation of the control plane. Any autonomous execution system has to answer three questions:

1. Who issues the instructions, and in what form are they expressed;
2. Inside which isolation boundary does execution happen, and which resources can it touch;
3. How can a human reliably confirm and approve the execution results.

In the past, all three questions were crammed into the same machine and the same window, so the boundary between the “shell” and the “kernel” was blurred. Once the control plane goes cross-device, that boundary is forced open, and the architectural problems surface. This article is about how to design that pipeline.

2. The Platform Logic: Why the Moat Is Not in the “Shell”

There is a structural fact the industry quietly understands: a resident agent framework that uses a chat window as its control plane is, at its core, an orchestration shell. What it contributes is session maintenance, scheduled triggering, external service connections, and message templates. The real reasoning ability comes from the underlying large model, and the real ability to read and write code and run commands comes from the agent runtime.

When the foundation provider fills in the control plane’s own capabilities—cross-device dispatch, persistent sessions, structured approval, and artifact return—the differentiation space of the middle shell gets squeezed from three sides:

• Distribution: Whoever controls the resident client controls the entry point. A chat window is someone else’s distribution channel; a native client is your own.
• Experience: A chat box can express “what to do,” but it struggles to express “show the diff, approve line by line, roll back locally.” Review-type interactions need a rich UI.
• Cost: Calling the same model and adding a markup layer offers no price advantage. The only place to save money is in context and tool-call strategy—and that belongs on the foundation side.

The conclusion is not complicated: if the middle layer wants to survive long term, it has to move up toward industry workflows, private knowledge, and compliance auditing—the places the foundation is unwilling or too slow to enter—rather than keep fighting over the general-purpose control plane.

3. Trade-offs Among Four Control Plane Forms

Form	Context construction	Result presentation	Approval granularity	Main weakness
Chat message	Weak; relies on user narration and pasted history	Plain text; long diffs are unreadable	Coarse; basically only “continue/stop”	Cannot carry file-level review
Terminal/CLI	Strong; can reference paths and command output directly	Text stream plus patches	Medium; interactive confirmation	Requires the person to be at the machine
Editor inline	Strong; naturally aware of the workspace	Inline diff; best review experience	Fine; can accept block by block	Bound to a single workspace
Cross-device remote	Medium; needs summaries and indexes to compensate	Structured cards with on-demand expansion	Fine; supports tiered approval on mobile	Long pipeline; high cost for reliability and security design

The essence of the cross-device remote form is to use one reliable link to connect “the execution power of a terminal/editor” with “the dispatch capability you carry with you.” Its engineering difficulty is not in model calls, but in the link and governance.

4. Reference Architecture Layers

Mobile client     issue commands / tiered approval / browse results / notifications
      |  auth + long-lived connection + push
Access layer      session routing, command validation, idempotent deduplication
      |  event bus
Orchestration     task state machine, context compression, tool scheduling, retry policy
      |  managed calls
Execution layer   desktop runtime: filesystem, terminal, dependencies, build and test
      |
Governance layer  permission declarations, approval hooks, sandbox, audit trail
      |
Data layer        session event stream, task artifacts, metrics, cost ledger

4.1 Sessions Must Be Externalized: Local-First Does Not Mean State Lives in Memory

The most common production failure in this kind of system is this: after a process restart or a network drop, the context of a long task is entirely lost. The reliable approach is event sourcing plus checkpoints: append user commands, tool calls, tool results, and approval decisions to disk as immutable events, and derive task state by replaying the events. The execution side holds only a cache, never the single source of truth.

// Minimal skeleton of event sourcing (illustrative)
type Ev =
  | { t: 'user_msg';  at: number; text: string }
  | { t: 'plan';      at: number; steps: string[] }
  | { t: 'tool_call'; at: number; id: string; name: string; args: unknown }
  | { t: 'tool_res';  at: number; id: string; ok: boolean; summary: string }
  | { t: 'approval';  at: number; id: string; decision: 'allow' | 'deny' }
  | { t: 'artifact';  at: number; uri: string; kind: 'diff' | 'log' | 'report' };
// Recovery: replay events to rebuild state; tool calls without a result are
// retried by idempotency key or skipped.
function restore(events: Ev[]) {
  const pending = new Set<string>();
  for (const e of events) {
    if (e.t === 'tool_call') pending.add(e.id);
    if (e.t === 'tool_res' || e.t === 'approval') pending.delete((e as any).id);
  }
  return { pending }; // after reconnect, handle pending first instead of re-planning
}

4.2 Link Reliability: At-Least-Once Delivery plus Idempotent Execution

Between the mobile client and the desktop, you typically pair a long-lived connection with a push channel. The design points are: heartbeat and exponential-backoff reconnection; messages carry a client-generated idempotency key, so the server executes duplicate dispatches of the same key only once; and progress reporting must be throttled and merged (for example, aggregated by a token budget or a time interval). Otherwise the mobile client gets flooded by high-frequency events, while needlessly burning through the quota.

4.3 Context Management for Long Tasks

An autonomous task can easily exhaust the context window after half an hour. A reusable three-part strategy:

• Rolling summaries: compress completed steps into “decision plus conclusion,” discarding raw noisy output;
• Fetch evidence on demand: keep only references to raw tool output (file path, line range, artifact address) and retrieve it when needed;
• Task graph and sub-agent isolation: hand parallelizable subtasks to sub-agents with independent context, return only structured conclusions, and avoid polluting the main context with intermediate process.

4.4 Approving Dangerous Operations: This Is the Main Battlefield of Security Design

Remote control turns “the human is not at the machine” into the default state, so approval must be driven by a tiered policy rather than “ask once every time.” A suggested four-level model:

1. Read-only: reading files, searching, the observation part of running tests—auto-allow;
2. Local reversible write: file modifications inside the workspace—execute automatically, but must produce a rollback-able artifact;
3. External impact: installing dependencies, accessing the network, writing to external services—requires explicit authorization, with a timeout;
4. Irreversible: bulk deletion, force-pushing to a repository, production changes, permission and credential changes—requires mandatory human confirmation, and “remember this choice” is forbidden.

The mobile approval card must carry enough information to make a judgment: the target path and blast radius, the raw command, whether it is reversible, and how to roll it back. The default behavior should be deny on timeout, not allow on timeout.

type Risk = 'readonly' | 'local-write' | 'external' | 'irreversible';
function guard(call: { name: string; args: any }): { risk: Risk; needApproval: boolean } {
  const { name, args } = call;
  if (name === 'read_file' || name === 'search' || name === 'run_tests') {
    return { risk: 'readonly', needApproval: false };
  }
  if (name === 'write_file' && inWorkspace(args.path)) {
    return { risk: 'local-write', needApproval: false }; // but a diff artifact must be written
  }
  if (name === 'install_dependency' || name === 'http_request') {
    return { risk: 'external', needApproval: true };      // bound by a timeout
  }
  if (name === 'run_shell') {
    const irreversible = /\brm\s+-rf\b|drop\s+table|push\s+--force/.test(args.command ?? '');
    return { risk: irreversible ? 'irreversible' : 'local-write', needApproval: irreversible };
  }
  return { risk: 'external', needApproval: true };        // unknown tools default to strict
}

4.5 Sandbox and Credentials

The execution layer must assume that “the agent will be misled or injected with malicious content.” Minimum requirements include: container or VM isolation, a least-privilege system account, a whitelist for file access outside the workspace, an allowlist for outbound network targets, capacity limits for build artifacts and cache directories, and a wall-clock time limit per task. For credentials, use short-lived tokens injected at the gateway, so the agent process itself never holds long-term keys; all logs and reported summaries must be redacted to avoid keys being returned to the mobile client or written into the session event stream.

4.6 Observability and the Cost Ledger

Metrics worth collecting: distribution of end-to-end task duration, the human approval rejection rate (which measures how often the agent behaves dangerously), tool-call failure and retry counts, per-task token usage and cost, and the artifact first-pass rate (whether it triggers human rework). The session ID should run through logs, metrics, and artifacts, so that “one tap on the phone” can be correlated with “one hour of consumption on the desktop.”

5. Build or Integrate: A Middle Path

Most teams do not need to build from the model up. The pragmatic middle path is: keep the mature model and execution foundation, and build the control plane and security governance layer in-house. There are three reasons: approval policy is tightly coupled to organizational structure; audit trails often have to satisfy internal compliance; and the reliability of the cross-device link directly affects the user experience and reputation.

A rollout checklist:

1. Define the risk-tiering table first, then write feature code;
2. Persist the session event stream to disk, supporting resumption from checkpoints;
3. Force every write operation to produce a rollback-able artifact;
4. Default unknown tools to strict, using an allowlist rather than a denylist;
5. Default mobile approval to deny on timeout;
6. Put outbound network and file access behind allowlists;
7. Make credentials short-lived and centrally managed, and redact all logs uniformly;
8. Set both a time limit and a cost limit per task;
9. Cover audit trails for “who approved what, what was executed, and which files were changed”;
10. Before launch, run prompt-injection drills, treating malicious instruction files in the repository as a real attack surface.

When hosted in the cloud, these capabilities map to general categories: store session events and artifacts in object storage, use a message queue to decouple tasks, use a container platform for execution isolation and elastic scaling, delegate auditing and alerting to log and monitoring/alerting services, and hand keys to a dedicated key management service. Baidu AI Cloud, with general capabilities across object storage, message queues, container platforms, log services, and monitoring/alerting, can cover the hosting needs of this kind of system; the specific selection still needs to be evaluated against your own compliance and capacity planning.

6. Conclusion

The real difficulty of a remotely controlled coding agent is holding both “the efficiency of autonomous execution” and “controllability when the human is absent” at the same time. The model decides what the agent can do; the control plane decides what it dares to do and what it is allowed to do. The former can be purchased; the latter must be designed yourself. Getting tiered approval, sandbox boundaries, session persistence, and the cost ledger solid pays off more in the long run than chasing the novelty of interaction forms.