## The Evolution of On-Device Intelligence: A Technology Roadmap for GUI Agents
GUI agents let AI models read the screen, plan actions, and operate an interface on the user's behalf. This article traces the three-stage evolution that brings this capability onto phones and tablets. It begins with prompt-engineering frameworks that translate interfaces into text or annotated screenshots. It then moves to vision-language models trained with imitation learning, and finally to reinforcement-learning-driven models that can correct mistakes, plan across multiple steps, and handle unseen apps and layouts. It also covers key engineering challenges for on-device deployment: compute and latency constraints, perception accuracy, safety boundaries, and observability. It closes with a short-, mid-, and long-term roadmap for on-device intelligence.
1. Introduction: Why the Ability to Operate Interfaces Is the Key Leap for On-Device Intelligence
Over the past two years, the focus of large-model competition has gradually shifted from whether a model can answer questions to whether it can complete tasks. Yet most real-world tasks, such as booking tickets, comparing prices, filling in forms, and moving information between apps, ultimately have to be completed through the graphical user interface (GUI). This has given rise to a core research direction: the GUI Agent. A GUI Agent is an AI system that lets a model directly understand the screen, plan operations, and execute actions within an interface.
On-device intelligence refers to bringing this capability down to terminal devices such as phones and tablets, so that an on-device agent can take over tedious interaction flows on the user’s behalf. It differs fundamentally from a traditional voice assistant or a shortcuts app. Rather than relying on fixed scripts written in advance by developers, the model makes decisions by “looking at the screen,” much as a person would. To understand both the vision and the roadmap for on-device intelligence, we first need to clarify the three-stage technical evolution of the GUI Agent.
2. Stage 1 (2023–2024): External Frameworks Driven by Prompt Engineering
The earliest mainstream approach left the model untouched and focused on transforming the interface input instead. Its core assumption was that a complex graphical interface could be translated into text or a structured format the model was already familiar with, and that the model could then reason its way to the next action. Two representative paradigms emerged in this stage:
- Structured text approach: The interface is extracted into a hierarchical structure such as an HTML/DOM tree or XML. Interactive elements such as buttons, input fields, and links are then serialized into text descriptions, and the model reasons over this text to choose an action target. Early academic web-task benchmarks and some app-level agent frameworks proposed by internet companies followed this route.
- Visual marking approach (Set-of-Mark): An interface screenshot is run through detection and recognition, and numeric markers such as [1][2][3] are overlaid on the key clickable elements. The model then only needs to output an instruction such as “click the third marker,” without having to understand pixel coordinates. An early mobile agent approach proposed by a cloud vendor belongs to this category.
The value of this stage was that it quickly proved the approach was feasible. However, its ceiling was equally clear:
- The capability ceiling was limited by prompt quality and the accuracy of external tools. Once OCR or element detection models made a mistake, everything that followed was compromised.
- The model never actually “saw” the interface. It was always processing translated text and performing text-level logical reasoning rather than visual understanding. As a result, it failed easily in scenarios such as canvas-rendered interfaces, game UIs, and non-standard controls.
- The pipeline was long, and errors accumulated along it. Detection, serialization, inference, and action execution formed a long chain. Every link added latency and uncertainty.
3. Stage 2 (2024): Imitation-Learning Vision-Language Models — Perception Goes Native
As the capabilities of vision-language models (VLMs) advanced by leaps and bounds, the technical route began to shift from external add-ons toward models with built-in capabilities. Representative work from this stage includes a GUI-native understanding model proposed by a research team, a GUI curriculum learning approach proposed by researchers, and an interface foundation model proposed by a joint research team. They share two key characteristics:
- No reliance on low-level structured data. HTML/DOM/XML are no longer needed, and pixel screenshots are used directly as input.
- Direct output of executable actions. These actions include click coordinates, swipe directions, text input, and more.
This marks the “nativization” of perception. The model begins to understand layout, control semantics, and spatial relationships by “seeing” the screen, much as a human does. On the training side, this stage relied heavily on imitation learning. Human operation trajectories, meaning screenshots paired with action sequences, were collected as the supervision signal. This taught the model: given the current screen, what would a human do next?
Imitation learning offers intuitive data and stable training. Its limitation, however, is that the model can only approximate the distribution of the demonstration data. When the agent deviates from the human demonstration path and enters a state not covered by the training set, it often lacks the ability to correct itself. In long-horizon tasks, it tends to drift further and further off course.
4. Stage 3 (2024–2025): Reinforcement-Learning Vision-Language Models — Autonomous Decision-Making Goes Mainstream
Breaking through the “imitation only” bottleneck depends on letting the agent learn through trial and error and optimize its policy through interaction with the environment. Reinforcement learning (RL) provides exactly this mechanism, which is why it has become the mainstream approach. Its evolution followed a clear progression:
- Proof of feasibility. A university team was the first to verify that a GUI Agent could be built with reinforcement learning. Their work showed that the closed loop of “interaction, reward, and policy update” could run on interface tasks.
- Validation at scale. Research teams subsequently validated the effectiveness of reinforcement learning on large-scale GUI tasks. This took the method from a proof of concept to something usable in engineering practice.
- Online reinforcement learning and long-horizon capability. The work of one vendor introduced large-scale online reinforcement learning. This allows agents to optimize their policies through continuous interaction with the operating system environment. Over time, agents gradually acquire three capabilities that were previously difficult to obtain:
- Error correction: After detecting a mis-click or an unexpected page navigation, the agent can backtrack and re-plan.
- Long-horizon planning: Across multi-step tasks, the agent maintains a consistent goal and is not distracted by intermediate pages.
- Generalization: When faced with new apps or new layouts not seen during training, the agent can still produce reasonable actions.
Only at this point did GUI Agents truly gain the ability to autonomously execute tasks in dynamic, open environments. The on-device intelligence assistant is precisely the integration and real-world deployment of this technical route.
The following illustrative pseudocode describes the core interaction loop of a Stage 3 agent. It shows how the “perceive–decide–execute–reward” cycle closes:
def run_gui_agent(task, env, policy, max_steps=30):observation = env.reset() # Current screen screenshothistory = []for step in range(max_steps):# 1. Perception: the model reads pixels directly and understands the interfaceprompt = build_multimodal_prompt(task, observation, history)# 2. Decision: output a structured action (click coordinates/swipe/text input, etc.)action = policy.infer(prompt)# 3. Execution: execute the action in the environment (OS/simulator)next_obs, reward, done, info = env.step(action)# 4. Reinforcement learning signal: reward comes from task completion or process feedbackhistory.append((observation, action, reward))if done:breakobservation = next_obsreturn history, info.get("success", False)
During training, the reward typically comes from signals such as whether the task was accomplished and whether the interface state is moving toward the goal. The agent uses these signals to update its policy, gradually evolving from superficial mimicry to autonomous self-correction.
5. Engineering Essentials for On-Device Deployment
Moving from cloud-based training to on-device deployment still requires bridging several engineering gaps. These gaps are also critical to whether on-device intelligence can scale:
- Compute and latency constraints: On-device resources are limited. Model size, quantization and compression, and inference latency must be balanced against one another. A common practice is to run heavy inference in the cloud. This can involve elastic scheduling through container platforms, message queues to smooth out traffic peaks, object storage to manage trajectory data, and logging, monitoring, and alerting to ensure stability. The device itself handles sensing and action execution. Together, this forms “cloud–edge–device collaboration.”
- Perception accuracy: Screenshot resolution, control density, and scroll state all affect comprehension quality. This requires dedicated data cleaning and annotation workflows.
- Safety boundaries: The agent has real operational permissions. It therefore needs an action whitelist, secondary confirmation for high-risk operations, and rollback mechanisms to prevent losses caused by mistaken actions.
- Observability: Complete action trajectories, reward curves, and failure attribution must be captured and retained. Only then can they support the continuous iteration of online reinforcement learning.
On the cloud side, general-purpose platforms such as Baidu AI Cloud offer compute scheduling, model serving, and data governance capabilities. They can be considered as one option for the underlying infrastructure that supports the training described above.
6. Roadmap Outlook
Drawing on the three-stage evolution, the roadmap for on-device intelligence can be summarized as follows:
- Short term: Center on vision-native models combined with reinforcement learning, and improve the success rate and robustness of single-app, short-to-medium-horizon tasks.
- Mid term: Strengthen cross-app collaboration and long-horizon planning, and introduce more powerful memory and state management mechanisms.
- Long term: Move toward a general-purpose agent built on cloud–edge–device collaboration. While safeguarding privacy and security, it should enable natural interaction in which “one sentence completes a multi-step complex task.”
At its core, the vision of on-device intelligence is to let AI evolve from a “conversational partner” into an “executor.” Understanding how GUI Agents evolved from prompt engineering to reinforcement learning means grasping the technical main thread of this transformation.
