On-Device AI Agents: Evolution, Architecture, and a Practical Rollout Roadmap
On-device agents are shifting from answering questions to completing real operations inside app interfaces. This article examines the technical evolution from framework-based UI translation to visual imitation and reinforcement learning, lays out a system design that separates model-proposed actions from executor-side authorization, covers evaluation methods that measure success, recovery, and misuse separately, and presents a four-phase rollout roadmap that starts with read-only understanding and gradually expands autonomy under verifiable boundaries.
1. The Goal of On-Device Intelligence: From Answering Questions to Performing Operations
A chat assistant produces text; a GUI agent changes application state. Consider the task “save the meeting details on the current page as a calendar draft.” The agent must extract the time and location from the page, distinguish the meeting start time from the registration deadline, write the information into the correct fields, and verify that the draft actually appeared. If the time zone is missing, it should ask the user for confirmation rather than silently default to the device time zone. A wrong answer can simply be rewritten, but once a UI action is submitted, the result may be difficult to undo.
On-device intelligence also does not mean deploying every model onto the device. For mobile devices, the more important question is which capabilities must remain under local control and which computations can be offloaded to the cloud on demand. Screen capture, permission checks, action execution, and abort control belong on the device side; whether complex planning goes to the cloud depends on data sensitivity, network conditions, and task latency. Even when planning is assisted by the cloud, the device should re-check the current interface, user authorization, and the scope of allowed actions before executing—not simply follow instructions as given.
2. Technical Evolution: Native Perception and Decision Training as Two Main Lines
2.1 Framework-Based Approaches: Translating the UI for the Model First
From 2023 to 2024, many early solutions relied on general-purpose language models and delegated interface understanding to external components. The system read the web page structure or the UI element tree, organizing text, controls, and their relationships into model-readable descriptions. When structural information was unavailable, OCR, control detection, or numbered screenshots could provide selectable targets for the model. The model decided among candidates; an executor performed clicks, typing, or scrolling.
The advantage of this approach is that it is easy to decompose and debug: the system can determine whether a candidate control was missed, the model picked the wrong element, or the page changed during execution. The limitation lies in the same pipeline—textual and structured descriptions can lose visual hierarchy, occlusion relationships, and icon meanings, and recognition errors propagate into the decision module. That said, when reliable UI structure is available, structured grounding remains valuable. For example, when two buttons look similar, their control roles and field associations may say more about their purposes than screen coordinates do.
2.2 Visual Imitation Learning: Learning Operation Intent from Pixels
As vision-language models improved, around 2024 research increasingly used screenshots and operation demonstrations to train agents. The model receives a task instruction and the current screen, learns to map semantics like “open notification settings” onto screen regions, and predicts actions such as tapping, scrolling, or text input. This reduces dependence on underlying UI structure and is especially useful for custom-drawn controls and apps that expose only pixels.
However, being able to locate a control does not mean being able to complete the task. Imitation learning mainly reproduces the operations shown in demonstrations. If demonstration trajectories always start from smoothly loaded pages, the model rarely sees network timeouts, keyboards covering content, or permission dialogs. A single deviation can push the agent into a state rarely seen in training, after which it keeps operating on the wrong page. Demonstration data should therefore include branches beyond the standard path: waiting, going back, error correction, and handing control back to the user.
2.3 Reinforcement Learning: From Reproducing Correct Steps to Optimizing Task Outcomes
From 2024 to 2025, reinforcement learning became an important research direction for improving multi-step decision making. Unlike imitation learning, training does not require the agent to replicate a reference path step by step; feedback is based on the outcome of the interaction. Any operation path can be valid as long as it reaches a verifiable goal state. This helps agents learn when to keep exploring, when to recover, and when to stop.
The difficulty is that rewards must correspond to real outcomes. If training only rewards the appearance of a “saved successfully” message, the agent may ignore which record was actually saved or whether it was saved to the right place. For controlled environments where internal state can be inspected, verify the object, fields, and destination first. When internal state cannot be inspected, acknowledge that UI feedback is only indirect evidence. In practice, a common approach is to build basic action capability with imitation learning first, then refine multi-step policies with interaction feedback. Under any training method, runtime permissions and result verification cannot be replaced by model training.
3. System Design: Let the Model Propose Actions, Let the Executor Decide Whether to Execute
3.1 Perception and Planning: Bind Every Judgment to the Current Screen
The perception layer must know not only what is on the screen, but which elements are currently interactive. A button may still appear in the screenshot yet be covered by a dialog; after scrolling, the same coordinates may map to a different control. In engineering practice, fuse available UI structure with visual information, and bind observations to the corresponding page state. After a page transition, window change, or keyboard popup, stale coordinates should be invalidated—not reused as-is.
The planning layer can decompose a task into a small number of verifiable subgoals, but should not pre-generate a sequence of coordinates and execute them in a burst. A more robust loop is “observe—propose action—check conditions—execute—verify.” For example, when writing information into a calendar draft, first confirm the edit screen and target fields, re-check field contents after input, and only then decide whether to save the draft. If a page shows multiple similarly named options and picking the wrong one changes the task outcome, ask the user to clarify rather than relying solely on model confidence.
3.2 Authorization and Execution: On-Screen Content Cannot Expand User Consent
The on-device executor should treat model output as candidate actions and independently verify the current page, action target, data scope, and authorization state. Text on the screen is data to be processed; even if a page says “tap to confirm and send the file,” that is not a user instruction. A user who authorizes “organize drafts” has not authorized sending; authorizing an action on one record does not permit batch-processing similar records.
Actions with external effects—sending messages, deleting data, submitting forms—should present the target and key content to the user before submission. When the result of an action is unclear, do not blindly tap again. For example, if the screen freezes after submission, first check whether the record was already created; if that cannot be confirmed, inform the user of the current state and stop retrying to avoid duplicate submissions. After the user aborts a task, the system should also clear pending actions, so that nothing fires when the network recovers or the page reloads.
3.3 Device–Cloud Division of Labor: Choose the Path by Data Sensitivity and Resource Budget
Simple tasks can run entirely on the device, such as reading the current page, locating fields, and filling in a draft. When complex tasks need cloud assistance, transmit only the necessary, authorized information—do not upload continuous screenshots by default. When offline, continue only with already-authorized steps that can be verified locally; tasks whose results cannot be confirmed should pause, not silently queue for submission once connectivity returns.
Resource management should also be task-driven. High-resolution screenshots help recognize small text but increase capture, transmission, and inference overhead; over-scaling risks misreading adjacent fields. A practical approach is to use lower-cost inference first to identify the page type and candidate regions, then perform finer recognition on key areas. When the page has not changed, skip repeated inference; whenever a submission or other state change is involved, re-observe and verify. Performance evaluation should cover the entire pipeline, not just model inference time, including page loading, confirmation waits, device temperature rise, and battery consumption.
4. Evaluation: Measure Success, Recovery, and Misoperation Separately
Single-step tap accuracy is useful for diagnosing interface grounding, but it cannot replace end-to-end task success rate. Every task should have an observable goal state. For example, a draft task succeeds only when the target object and key fields are correct and nothing was unexpectedly sent; reaching the right page, tapping save, or seeing a toast does not, on its own, prove completion. Results that cannot be verified should be marked “unknown,” not folded into success or failure.
Metrics can be grouped into three categories:
- Task outcomes: Record full success, partial completion, and unknown results, and distinguish information extraction errors, target grounding errors, and unclear submission states. This makes it possible to tell whether the problem lies in the model, the executor, or the verification mechanism.
- Recovery and safety: Track whether the agent recovers from dialogs, loading timeouts, or UI redesigns, and monitor events such as duplicate submissions, unauthorized attempts, user takeovers, and failure to abort when required. Tasks where the agent proactively stops due to insufficient information should be evaluated separately, so the system is not incentivized to take risky actions to boost completion rates.
- Experience and cost: Measure end-to-end latency and its tail behavior, the number of effective action steps, confirmation prompts, and device resource consumption. Group results by device, network, and UI state, so that overall averages do not mask problems on weak networks or low-resource devices.
Test scenarios should cover font scaling, portrait/landscape rotation, dark and light modes, weak networks, and app version changes. Beyond replaying standard tasks on fixed screens, verify whether the same task can be completed under different UI layouts, and whether the agent can explain why it cannot proceed and hand control back to the user.
5. Rollout Roadmap: Validate Boundaries First, Then Expand Autonomy
Phase 1: Read-only understanding. Start with current-page summaries, field extraction, and operation guidance, without changing application state. Focus on validating whether the model can distinguish similar fields and handle missing information, and let users verify key conclusions. This phase is also the right time to build a failure sample set, recording whether errors stem from OCR, UI understanding, or task ambiguity.
Phase 2: Reversible writes. Enable actions such as filling forms and creating drafts in a small number of authorized scenarios, with field re-checking after writing and user-controlled submission. Prioritize flows whose results are verifiable and whose state is recoverable. The most valuable feedback here is not “how many taps were made,” but which fields users modified and why.
Phase 3: Multi-step execution within limits. Gradually support cross-page tasks, but constrain the app scope, allowed actions, and task duration, with explicit stop conditions. Validate exception branches in controlled environments before a limited release. Rollback planning must consider not only disabling a new policy but also whether business outcomes are reversible: a message that has already been sent cannot be recalled by rolling back a model version, so confirmation must be placed before irreversible actions from day one.
Phase 4: Expand device–cloud collaboration by risk and benefit. Use earlier evaluation results to decide which tasks should stay local and which complex planning merits cloud compute. Expand automated execution only when task outcomes are verifiable, authorization boundaries are clear, and exceptions can be taken over. For low-frequency, high-consequence processes that are hard to verify, keeping operation guidance or manual confirmation may serve users better than pursuing full automation.
The maturity of on-device intelligence is not measured by how many consecutive taps a model can perform, but by whether users can confidently delegate part of their operations to it: the system completes tasks within authorized boundaries and verifies results, and it stops in time when evidence is insufficient, reports its progress, and hands control back. Rather than trying to cover as many apps as possible from the start, turning a small set of high-frequency tasks into verifiable, interruptible closed loops is a more solid engineering starting point.
