Cross-Device AI Agents: Architecture Practices for Remotely Controlling Computers

1. Paradigm Shift: From Human-Supervised Assistance to Autonomous Task Execution

Over the past two years, most AI assistants have remained at a stage where users need to stay nearby and supervise the work. Users enter instructions one by one, the AI returns results, and users then verify and correct them. This workflow relies heavily on the user being present. In this mode, the AI behaves more like a faster input method, while real decision-making and task coordination are still handled by the user.

The emergence of cross-device AI agents marks a shift in interaction paradigm. Once a task is clearly defined, the AI can continue to execute it autonomously after the user leaves the computer. The user can then check progress or adjust the direction remotely from another device, typically a phone. Three essential changes define this shift:

• Separation of control and execution: The device used to issue commands, such as a phone, is decoupled from the device that performs the task, such as a desktop computer. The user is no longer tied to the computer.
• Task continuity: The agent can maintain an ongoing session and orchestrate multi-step, cross-application tasks as a unified workflow, rather than handling isolated question-and-answer interactions.
• Autonomous execution loop: The AI shifts from being a recommender to being an executor, directly operating files, applications, and interfaces.

To understand the engineering value of such a system, it is important to first look at its overall architecture.

2. Overall Architecture: Phone as Remote Control, Desktop as Executor

A typical cross-device remote agent can be abstracted as a design that separates the control plane from the execution plane.

The control plane, usually running on a phone, is responsible for initiating tasks, adjusting task direction, and checking progress. It does not directly operate the desktop. Instead, it acts as a lightweight remote control that sends the user’s intent to the execution side in the form of messages.

The execution plane, running on the desktop, is where the actual work happens. It runs continuously on the user’s local device and can access the local file system, launch applications, and read or manipulate interface elements. To keep it controllable and prevent it from acting unpredictably, the execution side usually includes a resident agent runtime. This runtime receives instructions, invokes system capabilities, and reports status back to the user.

Pairing and authentication are key to connecting the two planes. A common industry practice is for the desktop to generate a one-time pairing credential, such as a QR code. After the phone scans the code, an encrypted channel is established, and both sides complete mutual identity verification. This design ensures that only the user’s own phone can control the user’s own computer, while avoiding dependence on fixed network addresses or long-term open ports.

After successful pairing, the user can maintain a continuous conversation channel between the phone and the execution side. This allows the user to add new tasks or correct the direction without repeatedly re-establishing a connection.

3. Local vs. Cloud Execution: Choosing Where Tasks Run

The most important architectural decision in a cross-device agent system is where the task actually executes. There are two typical approaches, each with clear strengths and limitations.

Local Execution Mode

In local execution mode, tasks run directly on the user’s device. The agent can natively access local files and directly invoke desktop applications. The advantages are significant:

• No file or data transfer: Sensitive files do not need to be uploaded, making privacy boundaries clearer.
• Full system access: The agent can directly operate the GUI and access local application state.
• Low latency: The interaction path is short, so feedback is immediate.

However, local execution also has notable limitations:

• The device must remain online and uninterrupted: If the user closes the laptop lid, disconnects from the network, or shuts down the machine, the task is interrupted.
• Dependence on the current session state: Background tasks are difficult to run truly unattended for long periods.

Cloud Execution Mode

Moving cowork-style tasks to the cloud is intended to address the fundamental limitation of local execution: the task cannot be separated from the device. Cloud execution provides several advantages:

• Continuity and stability: Tasks can run persistently in the cloud without depending on whether the user’s device is awake.
• Elastic scaling: Compute resources can be allocated on demand using container platforms.
• Unified operations: Task status can be centrally observed through logging, monitoring, and alerting services.

However, cloud execution introduces new challenges. In particular, access to local files and desktop applications cannot be inherited directly. An agent running in the cloud does not naturally have access to file handles on the user’s machine, nor can it necessarily see the user’s desktop. Therefore, moving to the cloud usually requires redesigning how data is accessed, such as through controlled temporary file uploads or object storage as an intermediary, and redefining security boundaries accordingly.

The Two Modes Are Not Mutually Exclusive

A mature solution is often a hybrid architecture. High-frequency, privacy-sensitive operations that depend on the local interface remain on the local device, while tasks that can run independently of the device, require long-running execution, or need elastic resources are moved to the cloud. Using a message queue to schedule tasks between the two sides is a common engineering approach for decoupling.

It is also worth noting that these capabilities often come with clear user eligibility requirements and lifecycle management. In early stages, access may be limited to higher-tier subscribers, and when capabilities are still immature, the product team may proactively restrict entry or stop accepting new users. This is a common product rhythm for early exploratory capabilities. Developers building similar systems should therefore reserve feature flags and rapid rollback mechanisms.

4. Safety Guardrails: Permission Controls Before Accessing New Applications

When an AI is allowed to operate an entire computer autonomously, the biggest risk is not that it works slowly, but that it performs the wrong action. As a result, safety guardrails are not optional; they are foundational to this type of system.

The widely recognized core principle is human-in-the-loop plus least privilege:

• Explicit consent before accessing new applications: When the agent first attempts to access an application or directory that has not previously been authorized, it should pause and request explicit user permission rather than silently exceeding its scope.
• Least-privilege authorization: By default, the agent should receive only the minimum permissions required to complete the current task, and those permissions should be revoked when the task ends.
• Auditable operation logs: Key actions should be recorded so that they can be traced and corrected after the fact.
• Additional confirmation for high-risk actions: Irreversible operations such as deletion, overwriting, or sending data externally should require an extra confirmation step.

The purpose of these mechanisms is to limit damage even if the AI makes a judgment error during execution. In other words, the security design should assume that the AI will eventually make mistakes and build safeguards around that assumption.

5. GUI Automation Limits: What It Can and Cannot Do

To understand the real capability level of this type of system, it is important to examine how far it can actually go in GUI automation. Based on substantial real-world testing feedback, a clear boundary emerges between what works reliably and what still tends to fail.

Tasks that are relatively reliable usually have the following characteristics: clear objectives, stable interface structure, and closure within a single application. Examples include:

• Searching for and organizing local files
• Reading from or writing to specific office or collaboration applications
• Summarizing or extracting insights from existing content such as emails or documents

Tasks that still tend to fail often cross multiple technical difficulties, such as:

• Opening new applications: This involves cross-process launching and window focus management, where timing uncertainty is high.
• Transferring data between applications: The agent must extract content from one application, switch to another, and write it in, making it extremely difficult to preserve intermediate state.
• Operating browser web pages: Dynamic DOM changes, login state, and anti-automation mechanisms make element location highly unstable.

Overall, the current success rate of cross-device agents for such tasks is roughly at a “half usable” level. They are sufficient for structured daily tasks but are not yet ready to handle open-ended, multi-step, or strongly time-dependent workflows without conditions. When integrating these capabilities, developers should design task granularity accordingly: decompose complex tasks into a series of high-success-rate atomic operations and validate the result of each step.

6. Engineering Recommendations for Developers

If you plan to build your own cross-device remote agent, the following dimensions are a practical starting point:

1. Prioritize control and execution separation: First design the encrypted pairing and message channel, then expand capabilities. The lighter the control plane, the more reusable the architecture.
2. Design the permission model first: Before writing the first line of execution logic, establish the closed loop of authorization, execution, revocation, and auditing.
3. Route execution location based on task profile: Keep privacy-sensitive tasks that depend on the local interface local; move long-running or elasticity-demanding tasks to the cloud, and connect both sides with a message queue.
4. Use general-purpose cloud services for foundational capabilities: Task state, file transfer, logging, and monitoring can be built using common cloud service categories such as object storage, message queues, container platforms, logging services, and monitoring and alerting. Avoid reinventing the wheel.
5. Balance task decomposition and validation: Break large tasks into high-success-rate atomic steps, validate results step by step, and promptly report or roll back when failure points occur.
6. Use gradual rollout and observability: Control the scope of feature availability with switches, and use complete logs and alerts to understand the health of each execution.

Cross-device AI agents represent an important step in the evolution of AI from assistance to autonomous execution. Their technical challenges are concentrated in the decoupling of control and execution, the trade-off between execution locations, fine-grained permission design, and the stability of GUI automation. For developers, rather than pursuing full automation all at once, it is better to first build a solid foundation: a secure pairing channel, permission guardrails, and an observability system. Then expand the agent’s capability boundary gradually. Safety and controllability will always matter more than flashy capabilities.