On-Device AI Agents: How GUI Agents Evolve from Screen Reading to Task Execution
This article traces the technical evolution of GUI agents on mobile and other intelligent terminals, from bolt-on frameworks that convert the interface into text, to vision-native models that read screens directly from pixels, and finally to reinforcement-learning-driven agents capable of multi-step task completion. It also examines the engineering problems that determine whether such agents can actually run on-device, including compute and latency constraints, privacy boundaries, action safety and permission control, and task memory across apps and pages. Finally, it discusses cloud-device collaboration as the practical architecture and outlines the likely directions of the next stage: cross-app coordination, proactive assistance, multi-module design, and intent understanding.
1. From “Phone Assistant” to On-Device Agent
Competition among smart devices is shifting. The focus is moving away from hardware specs and isolated features toward how well a device understands user intent and carries it out.
Traditional voice assistants can only handle fixed commands: open an app, set a reminder, look up some information. A more general on-device agent has to go further. It needs to interpret the current screen state, infer the user’s goal, call system capabilities, and complete a sequence of actions across multiple applications.
The core of this capability is not simply “being able to read the screen.” It is forming a complete closed loop:
- Perceive the current interface and device state;
- Understand the goal expressed in the user’s natural language;
- Decompose that goal into executable steps;
- Choose an appropriate way to act;
- Execute taps, swipes, text input, back navigation, and similar actions;
- Verify the outcome and correct errors when needed.
So the essence of on-device intelligence is not simply putting a large model inside a phone. It is making the model the task planner and interaction agent on the device. GUI agents are one of the main technical directions for reaching that goal.
2. The Three-Stage Evolution of GUI Agents
2.1 Bolt-On Frameworks: Converting the Interface into Model-Ready Information
Early GUI agent approaches generally did not modify the base model. Instead, they wrapped it in an information conversion and tool-calling layer. The system first extracts interface information through page structure, the accessibility tree, text recognition, or object detection, then formats that content into text or function parameters the model can consume.
A page might be converted into something like this:
Page: Shopping listElement 1: Search box, located at the topElement 2: Product card, located in the middleElement 3: Confirm button, located at the bottom
The model reasons over these structured descriptions and outputs actions such as “tap element 3” or “type a keyword into the search box.”
The advantage of this approach is that it is relatively straightforward to engineer. Developers can reuse existing large language models, constrain the output format through prompting, and rely on external tools for coordinate lookup, text recognition, and action execution. For apps with well-structured pages and clear control semantics, it can produce usable results quickly.
Its limitations, however, are equally clear.
First, the model does not see the full interface, only a “summary” produced by tooling. If the recognition module misses text or misclassifies a control, all downstream reasoning is built on bad information. Second, different apps expose their page structure to different degrees. Some custom-drawn controls, dynamic canvases, and complex animations are very hard to describe through traditional structured interfaces. Third, system capability depends heavily on prompt design and tool quality. The model itself never develops a stable visual interaction ability.
In technical terms, this stage is closer to “a text agent operating a graphical interface” than to a model that natively understands GUIs.
2.2 Vision-Native Models: Reading the Interface Directly from Pixels
As vision-language models matured, the research focus shifted toward feeding screen captures directly into the model and having it output action coordinates or action sequences. The system no longer depends on page structure. Instead, the model identifies text, icons, buttons, layout relationships, and visual hierarchy from pixels.
This change brought three important improvements.
First, models can handle interfaces with opaque structure. Whether a page is built from standard controls or from custom-drawn graphics, image buttons, and complex layouts, anything a person can understand visually is a candidate for the model to recognize.
Second, vision models can reason with spatial relationships. Tasks like “tap the settings entry in the top-right corner” or “select the action button on the second card” require combining position, text, and context, not merely matching a control name.
Third, the input modality is closer to how people actually use devices. Humans do not read a page tree and then analyze control attributes. They look at the screen and make a judgment. Vision-native approaches therefore have stronger generality.
Visual understanding, however, does not mean the problem is solved. A screenshot carries a large amount of redundant information, and the model has to localize targets accurately within limited compute. Mobile screen resolutions and aspect ratios vary considerably, so coordinate mapping is prone to drift. Pop-ups, the on-screen keyboard, scrollable regions, and dynamic content all interfere with judgment. And a single screenshot reflects only an instantaneous state; it cannot naturally express task history or the results of earlier actions.
For these reasons, vision-native GUI agents usually need multi-frame observation, post-action verification, and mechanisms such as zooming into local regions before they can interact reliably.
2.3 Reinforcement-Learning-Driven Agents: From “Seeing” to “Finishing the Task”
Imitation learning alone lets a model learn common UI interaction patterns, but with unfamiliar apps, error states, or multi-step tasks, decisions can remain unstable. Reinforcement learning is introduced to address long-horizon planning and trial-and-error.
A typical GUI task does not end after one tap. The agent has to string together multiple actions, each of which changes the screen state, and every later decision depends on the outcome of the previous one. Reinforcement learning can optimize the overall policy using feedback such as task success, step validity, erroneous actions, and safety risk.
The task process can be abstracted as:
State s_t: current screen, action history, task contextAction a_t: tap, swipe, type, back, or waitTransition: executing the action yields new state s_(t+1)Reward r_t: task progress, completion outcome, penalties forerrors, safety constraints
Compared with pure behavior cloning, reinforcement learning cares more about whether the end goal is reached. A model may take a different tap path to complete the same task; as long as the path is safe and the result is correct, it still receives positive feedback. This helps the model adapt to complex flows, page changes, and partial failures.
Reinforcement learning also brings new engineering challenges. Exploration on real devices is expensive, and a wrong tap may trigger irreversible operations such as deletion, payment, or privacy leakage. Training data has to cover enough apps and states. And if the reward is poorly designed, the model may chase surface-level progress while ignoring the real objective. Practical systems therefore typically combine offline trajectories, simulated environments, action filters, and human review.
3. Key Problems in On-Device Deployment
3.1 Compute Resources and Response Latency
On-device hardware has limited compute, memory, and battery. If a vision-language model is too large, latency and power consumption rise, which hurts the user experience. Model compression methods such as quantization, pruning, distillation, and low-rank adaptation are therefore needed, along with inference pipelines tuned for the target on-device chips.
A workable architecture is layered processing: lightweight models handle simple tasks locally, and stronger models take over complex ones. Frequently produced screen recognition results can be cached to avoid recomputation. Whole-screen images can be analyzed at low resolution first, with detailed recognition applied only to candidate regions.
3.2 Privacy and Data Security
One advantage of on-device intelligence is that data can stay inside the device. Screens often contain contact details, verification codes, payment information, and work documents. Uploading them to a remote service on every action creates obvious privacy risk.
Systems should define data boundaries at the architecture level: sensitive captures should be processed locally by default, logs should be redacted by default, model inputs should avoid carrying unnecessary personal information, and actions involving accounts, payments, or identity verification should require extra confirmation. For tasks that must use remote inference, the principle should be minimum-necessary upload, sending only the required local information.
3.3 Action Safety and Permission Control
Once a GUI agent can act, an error is no longer just an inaccurate answer; it can cause real damage. A tiered permission model is therefore necessary.
Routine operations, such as opening a page or scrolling through content, can run automatically. Operations that send messages, submit orders, delete files, transfer money, or change system settings should clearly show the target, the object, and the expected result before execution, and require user confirmation. For multi-step tasks, the system also needs an interruption entry point, timeouts, and rollback strategies.
3.4 Task Memory and State Management
Complex tasks often span multiple pages or even multiple apps. The agent has to keep track of the task goal, completed steps, current state, and reasons for failure, while avoiding the mistake of treating stale interface information as current fact.
A sound approach is to split memory into three layers: short-term visual memory holding the most recent frames; task memory recording the current flow and key variables; and long-term preference memory storing only stable information the user has authorized. Each layer should have its own lifecycle and access rules.
4. Cloud-Device Collaboration Is the Realistic Path
Being “on-device” does not mean every capability must run fully locally. For simple tasks with strong real-time or privacy requirements, a local model is the better fit. For complex planning, infrequent use, or tasks that need broad knowledge, a remote model can offer stronger capability.
A sensible cloud-device architecture has three parts: task routing, data classification, and result verification. The system first chooses where to execute based on task sensitivity, complexity, and latency requirements. It then decides whether to upload the full screen, a local region, or an abstracted structural representation. Finally, the device runs a safety check on the returned result.
This approach balances capability, cost, privacy, and experience, and it matches the gradual, incremental way on-device intelligence is actually evolving.
5. The Road Ahead
GUI agents are likely to develop along the following lines.
First, from single-screen operation to cross-app task coordination. Models will need to understand system-level context rather than focusing on one page at a time.
Second, from passive response to proactive assistance. Proactive behavior, however, must rest on clear authorization and controllable boundaries; it cannot come at the cost of predictability for the user.
Third, from a single model to multi-module collaboration, with separate components for visual perception, task planning, action execution, safety review, and result verification.
Fourth, from “completing actions” to “understanding intent.” Users typically state goals, not steps. Agents need to ask for clarification when information is uncertain and request confirmation at critical points.
Overall, the competitive edge of on-device intelligence does not lie in model parameter count alone. It lies in whether visual understanding, continuous decision-making, device control, and safety mechanisms can be integrated into a reliable closed loop. GUI agents have moved from peripheral tooling to vision-native models and now to reinforcement-learning-driven systems. The focus of the next stage will be lowering on-device operating cost, raising success rates on complex tasks, and defining execution boundaries that users genuinely trust. Only when an agent can see correctly, reason clearly, act accurately, and remain traceable will on-device intelligence grow from a showcase capability into a routine way of interacting with everyday devices.
