The Evolution of GUI Agents: From External Frameworks to Native Agents

    A technical overview of how GUI agent architectures evolved from 2023 to 2025 across three paradigms: external frameworks built on prompt engineering, native visual perception driven by imitation learning, and autonomous decision-making driven by online reinforcement learning. The article explains why each stage emerged, where the capability ceilings of each one sit, and includes a side-by-side comparison table, a decision-making heuristic for architecture selection, pseudocode for the perception-decision-execution loop, and the engineering factors that matter when deploying these agents on-device.

    Introduction: Why GUI Agents Are Becoming the Key Entry Point for On-Device Intelligence

    The graphical user interface (GUI) is the primary medium through which humans interact with digital devices. Making an agent “read the screen and operate it” the way a person does — across arbitrary applications — is widely seen as one of the most important breakthroughs on the path to general-purpose on-device intelligence. Unlike traditional automation that depends on fixed API calls, the core value of a GUI agent is this: it does not require the application side to expose any interface. Relying only on its ability to understand and operate the interface, it can complete composite tasks across applications and across scenarios.

    Between 2023 and 2025, this field did not improve in a straight line. Instead, it went through three distinct paradigm shifts. Understanding the logic of these three stages is useful not only for making architectural choices, but also for revealing the underlying drivers that took on-device agents from “usable” to “genuinely good.”

    Stage 1: External Frameworks (2023–2024)

    Core idea: don’t touch the model, reshape the input

    Early approaches shared a common premise: leave the large model itself alone. Because general-purpose models at the time had no direct ability to understand graphical interfaces, engineering effort went into prompt engineering to “translate” the interface into a form the model could process. The model behaved closer to a text-based logical reasoning engine than to an agent with genuine visual perception.

    Two parallel technical paths

    Input reshaping in this stage followed two main paths:

    • Structured text interface path: convert the interface into a structured description the model can read — for example, the page DOM tree, accessibility information, or cleaned-up HTML fragments. Text-based grounding then encodes clickable elements as function calls or coordinate instructions. This kind of scheme parses accurately in structurally well-formed scenarios, but depends heavily on whether the underlying data is obtainable.
    • Visually marked screenshot path: process a screen capture directly, detect interactive elements with an auxiliary model, and overlay numeric markers on the image (Set-of-Mark), so the model can output a decision such as “click element number 3.” This kind of scheme is less dependent on interface structure, but introduces an extra detection stage.

    Capability ceiling: the model never really “saw” anything

    The fundamental limitation of this stage was the absence of native perception. The agent’s capability ceiling was pinned down by three factors: the quality of prompt design, the accuracy of external tools (such as OCR and object detection models), and whether the underlying structural data was available at all. The model itself never understood screen pixels directly — it was reasoning at the level of text. Once the interface structure became complex or non-standard, or element detection drifted, the task chain would destabilize quickly.

    Stage 2: Native Visual Perception Driven by Imitation Learning (2024)

    From “translating text” to “looking directly at pixels”

    As the foundational capabilities of vision-language models (VLMs) improved, the center of gravity shifted toward what is intrinsic to the model. The representative direction in this stage dropped the strong dependence on structured data such as DOM and XML, and instead let the model take pixels directly as input to understand interface layout and output executable action coordinates.

    The capability jump that came from native perception

    The significance of this shift is that the model began to “see” the screen the way a person does, rather than reading the interface’s structure. The gains show up on three levels:

    • Decoupling from structured data: applications are no longer required to expose a DOM or accessibility tree, and adaptability to non-standard, unstructured interfaces (game screens, graphical canvases, mixed-rendered pages) improves markedly.
    • Native expression of action semantics: the model regresses click coordinates directly in visual space, so perception and action close the loop within the same representational system, reducing the information loss caused by intermediate conversions.
    • The imitation learning driver: by training on large-scale “interface-to-action” trajectory data, the model learns the basic patterns of how humans operate an interface, laying the groundwork for more complex decisions later.

    The limitation of this stage is that imitation learning can only reproduce the behavior distribution present in the training data. Faced with error accumulation over long-horizon tasks and with unfamiliar scenarios, its generalization ability remained insufficient.

    Stage 3: Autonomous Decision-Making Driven by Reinforcement Learning (2024–2025)

    Why complex decisions require reinforcement learning

    Real tasks usually involve multi-step, long-chain operation and are full of uncertainty: a click fails, a page transition misbehaves, a dialog unexpectedly covers the screen… Imitation learning struggles with this dynamic process of trial, error, and correction. This is exactly where reinforcement learning breaks through: it models the task as a sequential decision problem and lets the agent optimize its policy through reward signals in continuous interaction with the environment, rather than merely replicating a fixed trajectory.

    The online interaction loop: agents learn to self-correct

    The key advance of the reinforcement learning stage was the move from offline learning to online interaction. Through constant trial and error in real interaction with the operating system or applications, the agent gradually acquires three higher-order capabilities:

    • Error correction: recognizing when an action did not achieve the expected result (the click had no effect, the page did not change) and proactively falling back or switching strategy.
    • Long-horizon planning: maintaining goal tracking across multi-step tasks, decomposing sub-goals sensibly, and evaluating intermediate states.
    • Generalized response: transferring learned strategies to interfaces and tasks never seen during training, reducing brittleness on out-of-distribution scenarios.

    The following pseudocode illustrates the typical perception-decision-execution loop of such an agent:

    1. def gui_agent_loop(goal, env, policy, max_steps):
    2. # Initialize task context and action history
    3. state = env.reset()
    4. history = []
    5. for step in range(max_steps):
    6. # 1) Native visual perception: rely on the screen capture only
    7. observation = env.screenshot() # Pixel input, no DOM dependency
    8. # 2) Policy decision: the model outputs an action (click coordinates / type / swipe, etc.)
    9. action = policy.decide(goal, observation, history)
    10. # 3) Execute the action and collect feedback from the environment
    11. next_state, done = env.step(action)
    12. # 4) Result check: did this advance the goal? Decide whether to retry or fall back
    13. if not env.progress(goal, observation, next_state):
    14. history.append({"action": action, "result": "ineffective"})
    15. continue # Triggers error correction
    16. history.append({"action": action, "result": "ok"})
    17. if done or env.goal_reached(goal):
    18. break
    19. return history

    Only at this stage did GUI agents truly gain the ability to execute tasks autonomously in dynamic, open environments — marking the transition of on-device agents from “demonstrably usable” to “production-ready.”

    Comparing the Three Stages and Choosing a Technical Approach

    Dimension External framework Imitation learning + native perception Reinforcement learning + autonomous decisions
    Input form Text interfaces / marked screenshots Native pixels Native pixels
    Depends on DOM Yes, strongly (structured path) No No
    Native perception No Yes Yes
    Decision mechanism Text-based logical reasoning Imitation learning Online reinforcement learning
    Long-horizon error correction Weak Medium Strong
    Adaptability to unstructured interfaces Weak Strong Strong
    Typical fit Simple, well-structured flows Routine cross-application tasks Complex multi-step, open-ended tasks

    The core principle for choosing is this: the longer the task chain and the more open the environment, the more you should favor native visual approaches with online reinforcement capability. In closed scenarios with stable structure and fixed flows, a lightweight external framework can still be the best answer on cost and controllability.

    Engineering Considerations for On-Device Deployment

    Getting an agent onto an actual device requires more than the algorithmic paradigm above. These engineering factors also have to be weighed systematically:

    • Latency and responsiveness: on-device compute is limited, so perception and inference must complete within an acceptable latency budget. Model quantization, pruning, and inference acceleration are the key levers.
    • Privacy and data boundaries: screen captures contain highly sensitive information, so perception and decision-making should run locally wherever possible, and only desensitized structured results should be uploaded when necessary.
    • Model compression and memory footprint: on-device deployment is sensitive to memory and power, so a balance has to be struck between preserving capability and staying within resource constraints.
    • Stability and rollback: for long-horizon tasks, key actions should have confirmation mechanisms and fallback paths designed in, to avoid irreversible mistakes.
    • Evaluation and regression: maintain an evaluation set covering typical tasks, and continuously track how well the policy generalizes across different interface versions.

    Roadmap Outlook

    From external frameworks to imitation learning, and then to reinforcement learning, the evolution of GUI agents is essentially a history of “native perception plus autonomous decision-making.” The next stop for on-device intelligence is to deeply fuse autonomous decision-making driven by reinforcement learning with native visual perception, under the three simultaneous constraints of latency, privacy, and capability — so that agents can reliably and safely complete real tasks in open environments across multiple devices and multiple applications. This roadmap points to where the technical work needs to go, and also provides a clear coordinate system for shaping on-device intelligence products.