On-Device GUI Agents: From External Frameworks to Model-Native Intelligence
This article traces the evolution of GUI agents for on-device intelligence along a timeline that spans three key stages: external prompt-engineering frameworks (2023–2024), imitation-learning model-native agents (2024), and reinforcement-learning model-native agents (2024–2025). It explains the core ideas, capability boundaries, and engineering tradeoffs of each stage, compares the three approaches, and then examines the constraints and design considerations of deploying GUI agents on-device, including compute and latency limits, privacy and security, power consumption, and cloud-edge collaboration. It closes with a progressive roadmap for taking GUI agents from usable to production-ready, together with guidance on building an evaluation system that measures end-to-end task completion rather than single-step accuracy alone.
The core vision of on-device intelligence is to let AI truly “live” inside devices—perceiving, understanding, and operating the graphical user interface (GUI) locally—so that phones, tablets, PCs, in-car systems, and a wide range of smart terminals can “see the screen, understand intent, and get things done” like human users, without depending on a constant round trip to the cloud.
Over the past two to three years, the key technology behind this vision—GUI agents—has undergone a fundamental paradigm shift, moving from “external frameworks” to “model-native agents.” Understanding this evolution not only helps clarify the technical direction but also informs engineering decisions for on-device deployment. Using a timeline as the backbone, this article breaks down three key stages, their core ideas, capability boundaries, and engineering tradeoffs, and then extends the discussion to the constraints and roadmap for on-device deployment.
Stage 1: External Frameworks—Prompt-Engineering-Driven Interface “Translation” (2023–2024)
The earliest form of GUI agents did not modify the underlying large model. Instead, they built a layer of “translation + tools” around the model. The core idea was to convert a human-readable interface into text or function interfaces that the model could process. This route had two main implementations:
- Structured-text route: Parse the page into an HTML/DOM tree or an accessibility tree, abstract interactive elements (buttons, input fields, links) into structured node descriptions, and have the model reason over this “tree” to choose operation targets.
- Visual-marking route (Set-of-Mark): Overlay numbered markers on a screenshot, labeling each clickable region as 1, 2, 3, and so on. The model only needs to output a decision such as “click element #5,” and an external tool then maps that decision to coordinates.
The capability ceiling of this stage depended heavily on two external factors: the quality of the prompt design, and the accuracy of external tools—especially OCR recognition, object detection, and coordinate localization. Any error in these stages would cause later decisions to go wrong step by step. The more fundamental limitation was that the model did not truly “see” the interface; it received a text-based intermediate representation and was essentially performing text-based logical reasoning rather than visual understanding. As a result, these agents tended to be fragile when facing highly unstructured interfaces, interfaces lacking semantic annotations, or dynamically rendered pages.
Stage 2: Imitation-Learning Model-Native—Teaching the Model to “See” (2024)
As the capabilities of vision-language models (VLMs) improved overall, the technical path began to shift toward being model-internal. The representative idea of this stage was to abandon reliance on low-level structured data such as HTML/DOM and instead take screen pixels directly as input, letting the model understand the interface layout and output executable coordinates.
The key breakthrough was the “nativization” of perception: the model no longer depended on external OCR or detection models to “translate” the interface. Instead, like a human, it understood where buttons were, what menus existed, and what the current state was simply by “looking” at the screen. This mode significantly improved adaptability to unstructured GUIs—whether the interface used native widgets or was rendered from a web page, and whether or not it carried accessibility annotations—as long as it could be presented in pixels, the model could understand it.
For training, this stage commonly adopted an imitation-learning approach: collect large amounts of trajectory data mapping “interface states → human actions,” and use supervised fine-tuning to teach the model human operating patterns. The advantage was that data was relatively easy to obtain and training was stable. The limitation was that imitation learning could only learn behavior within the data distribution; when encountering complex decision scenarios not covered in training, its generalization ability was limited.
Stage 3: Reinforcement-Learning Model-Native—Advancing Toward Complex Decisions (2024–2025)
Imitation learning solved the problem of “can it do it,” but struggled with “does it do it well, and can it generalize.” Reinforcement learning (RL) was therefore introduced and has become the mainstream direction. Its core breakthrough is that, rather than simply imitating human trajectories, the agent interacts with the environment and receives reward signals based on task completion, optimizing its decision strategy through repeated trial and error.
Research in the field has first validated the feasibility of “building GUI agents with reinforcement learning,” demonstrating that reward-driven policy optimization can handle multi-step, long-sequence complex interface tasks. Building on this, further work has validated the effectiveness of reinforcement learning on large-scale GUI tasks, showing stronger decision robustness and higher task-completion rates in real, diverse interface environments.
The value of reinforcement learning is that it can go beyond the boundary of human demonstration data. By using reward shaping to guide the agent toward better operation sequences, it performs better in complex scenarios such as multi-step reasoning, error recovery and backtracking, and cross-application coordination.
Comparing the Three Approaches and the Evolution Logic
Placing these three stages side by side reveals a clear evolutionary throughline: the center of capability has progressively shifted from “external tools” to “the model itself.”
- External frameworks: The model performs text reasoning, with capability limited by tool accuracy and prompt design. Advantages: low implementation barrier and strong explainability.
- Imitation-learning model-native: The model learns visual understanding, with perception nativized. Advantage: broad adaptability. Disadvantage: constrained by the training-data distribution.
- Reinforcement-learning model-native: The model learns autonomous decision-making, with optimized decisions. Advantage: stronger generalization and robustness. Disadvantage: high training cost and low sample efficiency.
A schematic code structure can summarize the core interaction loop of a model-native agent: the model receives a screenshot and task description, outputs a structured action instruction, executes it, then re-observes, repeating until the task is complete.
# Schematic: the interaction loop of a model-native GUI Agentdef run_gui_agent(task: str, max_steps: int = 20):observation = capture_screen() # 1. Capture the current screen pixelshistory = []for step in range(max_steps):# 2. The model outputs a structured action based on pixels + history + taskaction = model.predict(screen=observation,instruction=task,history=history,)# 3. Parse the action: tap / type / swipe / wait / doneif action.type == "tap":execute_tap(action.x, action.y) # Coordinates are given directly by the modelelif action.type == "type":input_text(action.text)elif action.type == "swipe":swipe(action.start, action.end)elif action.type == "done":return action.result# 4. Re-observe and proceed to the next stepobservation = capture_screen()history.append(action)
On-Device Intelligence: Deployment Constraints and Engineering Tradeoffs
Deploying GUI agents on-device faces constraints that differ significantly from those in the cloud, and these constraints directly determine the technology choices.
- Compute and latency constraints: On-device chip compute and memory bandwidth are limited, and large-model inference is latency-sensitive. Viable engineering techniques include model quantization (such as INT8/INT4), knowledge distillation, operator fusion, and deciding whether to trigger cloud collaboration based on task complexity.
- Privacy and security: Screen content is highly sensitive and involves scenarios such as accounts, chats, and payments. On-device inference naturally offers the privacy advantage of “data never leaving the device,” which is one of the core values of on-device intelligence compared with pure cloud solutions.
- Power consumption and heat: Continuously running vision models and inference loops consumes significant power, requiring optimization through task scheduling, on-demand wakeup, and result caching.
- Cloud-edge collaboration: When on-device capability is limited, a layered architecture of “lightweight on-device perception + heavy cloud inference” can be adopted. By leveraging general cloud capabilities (such as model hosting services, containerized inference platforms, message queues and logging services, and monitoring and alerting), teams can build an elastic, observable inference pipeline while keeping sensitive data on the device as much as possible.
A Roadmap: From Usable to Production-Ready
For engineering implementation, the following progressive roadmap can be followed:
- Step 1, close the loop: First use an external framework with structured parsing to validate the feasibility of the “perceive–decide–act” loop, quickly getting typical tasks running end to end.
- Step 2, nativize perception: Introduce vision-language models to understand the interface directly from pixels, improving adaptability to unstructured interfaces.
- Step 3, reinforce decision-making: Introduce reinforcement learning to optimize the strategy, improving the completion rate and robustness of multi-step complex tasks.
- Step 4, optimize for on-device: Through techniques such as quantization, distillation, and cloud-edge collaboration, consolidate capabilities on the device while balancing privacy, latency, and power consumption.
At the same time, the evaluation system needs to evolve in parallel: beyond single-step action accuracy, more attention should be paid to end-to-end task completion rate, average number of steps, error-correction ability, and generalization performance on unfamiliar interfaces. Only by upgrading the question from “does it work” to “does it work stably and broadly” can the technology truly hold value for large-scale deployment.
Conclusion
The evolution of GUI agents is essentially a history of technological progress in which “perceptual capabilities are progressively internalized and decision-making capabilities are progressively strengthened”: from text reasoning dependent on external tools, to model-native visual understanding, and then to reinforcement-learning-driven autonomous decision-making. The vision of on-device intelligence is to bring this evolutionary path under the real-world constraints of compute, privacy, and power consumption. For developers, understanding the capability boundaries and engineering tradeoffs of each stage is what makes it possible to make sound choices between “the agility of external frameworks” and “the ceiling of model-native agents,” and to truly build on-device agents that understand both the interface and the user’s intent.
