On-Device AI Agents in Practice: Deploying and Selecting Models for Phone Automation
A hands-on engineering guide to building a phone automation agent that understands screens, plans steps, and executes actions. This article covers the perception–decision–execution architecture, environment setup, the trade-offs between local quantized models and cloud multimodal LLMs, structured action output, prompt design, failure recovery, and troubleshooting tips.
Introduction: From Fixed Scripts to Agents That “See the Screen”
Traditional UI automation relies on fixed coordinates, resource IDs, or widget paths, all of which tend to break completely when an app is redesigned or the screen layout changes. With large language models, automation logic can become a closed loop of “understand the screen, plan the steps, execute the action” — describe the goal in natural language and complete cross-app operations such as “play a specific song” or “find a nearby restaurant and place an order.”
It’s worth noting that this capability is not the monopoly of any particular product, but a common industry approach for on-device agents: the model handles understanding, planning, and error correction, while the operating system’s accessibility capabilities or input injection mechanisms handle execution. Setting specific products aside, this article walks through the full process of building, debugging, and tuning such an agent from an engineering perspective.
1. Overall Architecture: A Three-Layer Perception, Decision, and Execution Loop
A usable phone agent typically consists of four layers:
- Perception layer: Captures the current screen state, including screenshots, the widget tree (text, type, clickable state, and coordinates), and the name of the current foreground app;
- Decision layer: The LLM receives the task description, screen representation, and action history, then outputs the next action;
- Execution layer: Uses the system’s accessibility service or input event interfaces to perform taps, swipes, text input, back navigation, and so on;
- Orchestration layer: Handles the task loop, history compression, failure retries, and logging, deciding when the task is complete.
Of these four layers, the decision layer determines the upper bound of performance, while the perception layer determines engineering usability — both are indispensable. A typical single-step decision flow looks like this:
[screenshot + widget tree + action history]│▼[prompt assembly: task goal / action space / history]│▼[LLM inference: output structured action]│▼[action validation → accessibility service execution → state readback]│└──── task incomplete → next iteration
2. Preparation: Devices and Environment
2.1 Hardware Selection
Use a spare phone that has been wiped of personal data, with all payment and social accounts signed out. This is both a privacy precaution and a practical safety measure — automated actions carry a risk of accidental touches and shouldn’t run on your primary device. On the computer side, you need a standard environment capable of running the runtime and debugging tools (the experiments in this article were based on an ARM laptop with 24GB of RAM).
2.2 Device Authorization and Debugging Channel
Three things need to be done: enable developer options and the debugging channel; authorize the automation service in the accessibility settings; and grant necessary permissions such as “display over other apps.”
Wireless debugging is recommended to avoid keeping the device tethered. If possible, prefer the platform’s built-in debugging protocol over vendor-specific drivers, since this yields better compatibility across device brands.
2.3 Runtime Environment
Using the Python ecosystem as an example, set up an isolated virtual environment with pinned dependency versions to avoid conflicts with system packages:
python3 -m venv agent-envsource agent-env/bin/activatepip install -r requirements.txt
Pin dependencies to exact versions (e.g., pkg==1.2.3) rather than just major version numbers, so the environment can be reliably reproduced when switching machines or rebuilding.
2.4 The Trade-offs of “One-Click Install” Approaches
The popular “lazy install” approach in the community essentially lets an AI coding tool read the project instructions and automatically run the installation scripts. This significantly lowers the barrier to entry, but from an engineering standpoint it must be said: automatically executed scripts run with the same permissions as the current user — they download dependencies, modify configurations, and start background services.
Always do three things: test-run in an isolated environment first; read through the key scripts before installing; and inject sensitive information via environment variables rather than writing it into config files. The lesson here is to weigh “convenience” and “control” separately — speed is fine during exploration, but before long-term use, return to manual installation and version pinning.
3. Choosing the Decision Model: Local Small Models vs. Cloud LLMs
This is the most critical fork in the road. Here’s a practical comparison based on hands-on experience with a 9B-class quantized model versus a cloud multimodal LLM:
| Dimension | Local Quantized Model | Cloud Multimodal LLM |
|---|---|---|
| Screen understanding | Mostly relies on the widget tree; mediocre on icon-based apps | Reads screenshots directly, with more accurate semantic understanding |
| Response speed | Several seconds per step; noticeably sluggish on long tasks | Typically sub-second to one second |
| Hardware requirements | Requires a capable CPU/GPU and ample memory | Only a network connection and a lightweight client |
| Privacy | Data never leaves the device | Screen content is uploaded |
| Cost | One-time hardware investment | Pay-per-call |
3.1 The Compute Bill for Local Deployment
The real cost of a local model isn’t just the weight files. With 4-bit quantization, a 9-billion-parameter model’s weights take up about 5GB. Add KV cache, runtime overhead, and OS usage, and a 24GB machine runs tight on headroom when the agent shares memory with a browser and IDE — leading to inference queuing or even processes being killed by the OS.
On speed, CPU inference is heavily constrained by memory bandwidth. End-to-end single-step latency is typically in the range of several seconds, and a full task often takes a dozen to dozens of steps — the cumulative experience is “it works, but it isn’t pleasant to use.”
3.2 Common Pitfalls on the Local Route
- Quantization format mismatched with the inference backend: Different backends support different quantization schemes. On load errors, check the format first rather than repeatedly reinstalling;
- Context and output format: Small models easily lose constraints over long contexts. Compress history, limit the number of steps, explicitly require valid JSON output, and add automatic retries on parse failures;
- Missing multimodal capability: Most small models accept text only, so screen information must first be converted into a widget-tree summary. Information loss on icon-heavy UIs directly leads to wrong decisions.
3.3 Using the Cloud Route Correctly
If you use a multimodal LLM service from a cloud platform such as Baidu AI Cloud, feed in “screenshot + widget-tree summary + action history” together and require the model to output structured actions. Pay attention to three engineering details: manage API keys properly and never hard-code them in clients; set timeouts and rate-limit retries to avoid tasks hanging; and strictly validate responses, falling back to widget-tree strategies on parse failures instead of failing outright.
4. Action Execution and Structured Output
The execution layer should abstract away low-level differences into a small set of atomic actions, making it easier for the model to learn and recover from errors:
ACTIONS = {"tap", "long_press", "swipe", "input", "back", "launch", "wait", "finish"}
Force the model to output JSON, with examples to constrain the format:
{"action": "tap","target": "search box","reason": "need to enter the search page before querying content"}
The key engineering detail is action validation: coordinates produced by the model must fall within screen bounds; if element location fails, retry by re-fetching the widget tree first, and after two consecutive failures trigger a screenshot review. This step often improves overall success rates more than swapping in a stronger model.
5. GUI-Based Orchestration: Why Add a Graphical Layer
Once things run from the command line, adding a GUI delivers immediate benefits: editable task goals, observable execution, and replayable failure steps.
For developers, the core value of a GUI is turning a “black box” into a “debuggable white box” — only by seeing the screenshot, model output, and execution result for every step can you tell whether perception, decision, or execution failed.
6. Prompts and Failure Recovery
Split the prompt into four sections: system role and capability boundaries, available actions, the current task goal, and a summary of past steps. Use a rolling summary for history instead of concatenating everything, to prevent context bloat.
Handle failure recovery in tiers: retry perception when element location fails; step back one action after several steps with no progress; end the task and emit diagnostic information when the step limit is reached.
7. Troubleshooting Checklist
- Device won’t connect: confirm the debugging authorization was accepted and drivers are working; try restarting the debug service;
- Accessibility service keeps turning off: some systems revoke permissions under power-saving policies — add the app to the battery optimization whitelist;
- Tap position is offset: usually caused by resolution or status bar height differences; use the actual bounds returned by the widget for coordinate conversion rather than fixed ratios;
- Text input is empty: first check whether the input field truly has focus, then check for IME interference;
- Task stalls midway: usually an unmet wait condition — add timeouts and screenshot readback to every action.
8. Safety Boundaries and Practical Advice
Automation means the model makes decisions on your behalf. Stick to three ground rules: keep no sensitive accounts or payment credentials on the device; require secondary confirmation for irreversible operations such as transfers, deletions, and publishing; and keep complete execution logs for important tasks to enable auditing.
Overall, if you prioritize stability and performance, start with a cloud multimodal model; if you have a hard requirement that data never leaves the device, accept the local model’s speed compromise and invest your effort in improving perception-layer information. Phone automation is still at the stage of “handling simple repetitive tasks” — it’s more realistic to treat it as a productivity tool rather than a fully autonomous agent.
