Building Computer Agents with Recorded Skills: A Technical Approach to Cross-Device Automation

I. Breaking the Information Silo: Why Personal Automation Is Hard

Modern internet services are highly fragmented. A user’s daily tasks are often scattered across multiple applications: receipts might be stored in a standalone client, an email inbox, and a billing page; local service queries, parking payments, and food delivery each exist as separate apps. This state of “sealed doors” is a classic manifestation of information silos.

More critically, with the rise of Personal Agents, this silo problem is beginning to backlash against AI capabilities. A clear trend has emerged in the industry recently: e-commerce platforms, banks, hotels, and online document services have started actively blocking automated access from AI assistants, frequently popping up warnings like “Unauthorized agent access violates the Terms of Service.” In other words, Agents attempting to complete tasks by calling platform APIs or simulating requests are facing increasingly strict blockades.

This reflects a fundamental contradiction: platforms have no obligation to open their data interfaces to third-party agents. The path for API-level integration is becoming narrower.

II. A New Perspective: Why Computer Use Is Key

Since the application layer is a dead end, a natural technical approach is to shift the perspective down to the operating system layer. All applications, whether closed or open, ultimately run on the OS, render their interfaces on the screen, and receive input via mouse and keyboard. The “universal interaction capability” at the OS level inherently bypasses API authorization issues.

This is the value of Computer Use technology. Its core principle is a perception-decision-execution loop:

loop:
    screenshot = capture_screen()          # Perception: Capture screen state
    action = LLM_plan(screenshot, goal)    # Decision: Model analyzes and plans next step
    execute(action)                        # Execution: Simulate click/type/scroll

Compared to traditional RPA (Robotic Process Automation), Computer Use powered by large language models brings several qualitative leaps:

1. Understanding Natural Language Goals: There is no need to pre-define rigid operational steps; you simply describe the task, such as “help me collect this month’s invoices.”
2. Interface Adaptability: The model can dynamically adjust its behavior based on screen content to handle UI revamps, pop-up interferences, and loading delays.
3. Generalization: Faced with application interfaces it has never seen before, the model can still infer how to operate based on common sense.

However, general models also have shortcomings. For highly personalized workflows (e.g., “invoices must be registered into my specific ledger file, categorized by default as subscription fees”), relying solely on model inference is unstable. This leads to the core mechanism discussed in this article: recording operations to generate skills.

III. The Metaphor of Mentoring: Technical Essence of Record-to-Generate Skills

Imagine mentoring an intern: simple tasks can be communicated verbally, but complex tasks require a live demonstration. The demonstration process conveys not just “what to do,” but also the tacit knowledge of “how to do it”—where to click first, what to watch out for, and what default values to enter.

“Recording operations to generate skills” productizes this metaphor:

• You operate, AI observes: The user manually completes a task on the computer while explaining key points in natural language.
• AI synthesizes and solidifies: The system structurally organizes the screen operations, input content, and explanations to generate a reusable “skill” description.
• One-command reuse: Later, simply stating the task intent triggers the agent to execute according to the solidified workflow. It directly reuses the recorded flow when encountering identical scenarios, and flexibly applies its general capabilities when deviations occur.

From an engineering perspective, this process is essentially a compilation from demonstration trajectories to task scripts:

(Screen event stream + User voice explanation)
        ↓ Model induction
{
  "skill": "collect_invoices",
  "steps": ["Open App A invoice page", "Query email attachments", "Visit billing page to download"],
  "params": {
    "email": "<Default value observed during recording>",
    "ledger_path": "<User-specified ledger path>",
    "category": "Subscription fee"
  }
}

The fact that parameters are recorded as default values is crucial, as it gives the skill “specialization”: the general model handles fallback and generalization, while recorded experience ensures precise customization. This concept aligns philosophically with industry directions like imitation learning and few-shot workflow grounding.

IV. A Complete Practice: Cross-Entry Invoice Auto-Collection

Let’s take an almost universal manual chore as an example: collecting reimbursement invoices for AI services. These invoices are typically scattered across at least three places—an application’s invoice management page, billing emails in a personal inbox, and an overseas service’s billing download page. After retrieval, they often need to be logged into a summary spreadsheet.

The complete implementation steps are as follows:

Step 1: Record the demonstration. Open the three entries sequentially, walk through the query and download process, and casually note “use this email,” “log it into that spreadsheet on the desktop,” and “categorize as software subscription by default.” After recording, the system automatically generates a skill. Information demonstrated only once—like the email address, ledger path, and invoice category—all become default parameters.

Step 2: Trigger execution. Later, simply say “help me collect this month’s invoices.” The agent will access the three entries sequentially according to the skill, download the files, and log them into the ledger.

Step 3: Observe an interesting detail. During execution, a status prompt indicating “Agent is operating” appears at the top of the screen. You can even see it opening the very application where the recording originated to perform operations—the AI can manipulate its own operating environment. This precisely illustrates the fundamental difference between an OS-level solution and an API-level solution: it does not depend on any party “opening the door.”

V. Connecting Mobile Apps: Cross-Device Loop via Screen Mirroring

If computer workflows can be recorded, what about mobile apps? The key technology here is system-level screen mirroring and input return mapping.

Modern desktop operating systems generally support mobile mirroring capabilities: the phone’s screen is projected in real-time to the computer display, while mouse and keyboard inputs can be reverse-mapped to the phone. This means the mobile screen technically becomes a “window” on the desktop screen, allowing:

1. Mobile app interfaces to be captured via screenshots and simulated with clicks;
2. The same Computer Use agent to incorporate mobile operations into task orchestration;
3. Demonstration trajectories during skill recording to naturally cross boundaries between “desktop app -> email -> mobile app.”

A typical hybrid task could be: “Check my account balance, pay the parking fee, and finally log the result into a memo document on the computer.” For the user, this is a single command; for the system, it is a cross-device workflow orchestration where the screen mirroring layer masks the difference between “this is a phone” and “this is a computer.”

VI. Boundaries and Considerations

This solution is not without limitations. Several points must be recognized before engineering implementation:

• Stability depends on the interface: Screen operations are inherently affected by network latency, pop-ups, and CAPTCHAs. Complex tasks still require human fallback.
• Permissions and security: The ability to operate everything on your behalf means that screen recording and accessibility permissions must be granted cautiously. Sensitive operations (like payment confirmations) should ideally retain a manual checkpoint.
• Compliance boundaries: Although the OS-level solution doesn’t rely on API authorization, the constraints imposed by Terms of Service on automated behavior still deserve respect. Account security and rate control are fundamental practices.

Conclusion: API blockades have caused personal Agents to hit a wall, but shifting the perspective to the operating system reveals a new solution: the screen is the interface, and operations are the protocol. Combined with the customization mechanism of “recording demonstrations to generate skills” and the cross-device channel of mobile screen mirroring, the application silos scattered across computers and phones are being reconnected in a more universal way. For developers, this might be one of the most worthwhile directions to invest in when building personal automation systems.