Mobile GUI Agents: Architecture Breakdown and Engineering Practices
This article breaks down a practical framework for mobile GUI agents—AI systems that let large language models move from generating text to executing real device operations, including in apps that lack open APIs. It walks through a two-layer design (a device-side execution skeleton and a ~9B-parameter decision model), then covers the engineering that makes it work: dual-channel perception with control-tree compression and coordinate alignment, a minimal seven-action space with strict JSON output, a perception-decision-execution loop, and long-horizon techniques such as hierarchical planning, memory compression, and cross-app context passing. It also explains deployment tradeoffs (on-device, cloud virtual devices, and hybrid), security and privacy guardrails, an end-to-end evaluation framework, and common failure modes with mitigation strategies.
1. Background: From “Talking” to “Doing”
Over the past two years, the capability boundary of large language models has largely played out within text: answering questions, generating code, and summarizing documents. But real-world productivity is heavily embedded in graphical interfaces—dozens of high-frequency apps, thousands of page states, and a large body of legacy systems with no open APIs. Historically, these scenarios could only be handled by RPA scripts with hard-coded control paths, which broke the moment an interface was redesigned.
Mobile GUI agents exist to fill this gap: instead of producing “a paragraph,” they produce “a sequence of executable device operations.” The technical evolution is fairly clear. Starting in 2023, the industry began exploring agent models with execution capability; in the second half of 2024, the first complete phone-use capability appeared and completed a fully automated interaction in a public demo; in 2025, cloud virtual device forms emerged, raising data-security levels through runtime isolation; and by the end of 2025, the core models, inference code, and toolchains were released under permissive open-source licenses (MIT / Apache-2.0 style), with demos covering more than 50 mainstream Chinese apps.
This trajectory points to one thing: the bottleneck for GUI agents has never been model intelligence alone—it is engineering-chain completeness.
2. Overall Architecture: A Two-Layer Collaboration Model
A usable mobile agent framework typically consists of two layers.
Layer 1: Device-side agent framework (the execution skeleton)
Responsible for interacting with the operating system, it provides four core capabilities:
- Interface capture: obtain the control tree via Accessibility Service, or grab screen frames via system APIs;
- Action injection: dispatch tap, swipe, text input, back, home, and other events to the system;
- State management: maintain task context, historical trajectories, and rollback anchors;
- Safety gate: intercept payment, transfer, deletion, and authorization actions for secondary confirmation.
Layer 2: Model-side decision core (around 9B parameters)
It takes a natural-language instruction and a structured description of the current interface as input, and outputs the next action. Choosing the ~9B range is not a compromise but a deliberate choice at the engineering sweet spot:
| Dimension | Large model (100B+ scale) | Lightweight model (~9B) |
|---|---|---|
| On-device feasibility | Requires cloud round-trips, hard to run offline | Can run on-device or at the edge after quantization |
| Single-step latency | Seconds | Hundreds of milliseconds |
| Long-horizon task cost | Steps × high unit price | Steps × low unit price |
| UI grounding accuracy | Strong general ability but needs heavy fine-tuning | Higher grounding accuracy after vertical fine-tuning |
The hard part of GUI tasks is visual grounding and action-sequence consistency, not open-domain reasoning. These capabilities are far less sensitive to parameter count than to the distribution of training data. So a small-to-medium model, fine-tuned on high-quality interface data, can approach or even exceed a general-purpose large model.
3. Perception Layer: Turning the Interface into Model-Readable Input
The quality of the perception layer directly sets the ceiling for the decision layer. The mainstream approach uses dual-channel input.
Channel 1: Structured control tree (primary channel)
The node tree returned by the accessibility service contains text, type, clickability flags, and bounding boxes. A raw tree often has hundreds to thousands of nodes and must be compressed, or it will blow up the context window:
def compress(nodes, max_nodes=120):"""Control-tree compression: pruning + semantic aggregation"""kept = []for n in nodes:if not n.is_visible or n.bounds.area() < 400:continue # drop invisible and too-small controlslabel = n.text or n.content_desc or n.widget_typeif label in {"", "ImageView", "View"}:continue # drop containers with no semanticskept.append({"id": len(kept),"label": label[:40],"type": n.widget_type,"clickable": n.is_clickable,"bbox": n.bounds.as_center(), # normalized coordinates})return rank_by_relevance(kept, max_nodes)
Channel 2: Visual frames (auxiliary channel)
For pure Canvas-rendered, game-like, or chart-based interfaces, the control tree fails, so you fall back to screenshot + visual encoding. Coordinates use a normalized representation to avoid bias from resolution differences.
Key engineering detail: coordinate alignment
The bounding-box coordinates in the control tree and the pixel coordinates of the screenshot must be mapped to a unified system. The common approach is to have the model output normalized coordinates (0–1), which the execution layer then multiplies by screen width and height to restore absolute pixels. If you feed absolute pixels directly to the model, everything collapses the moment you switch devices.
4. Decision Layer: Action Space and Prompt Protocol
Action space design principle: the minimal sufficient set
Too many action types make model selection harder; too few can’t cover real interactions. The commonly converged set is seven:
@dataclassclass Action:type: Literal["tap", # tap"swipe", # swipe (up/down/left/right)"input", # text input"key", # system key BACK / HOME / ENTER"launch", # launch app by package name"wait", # wait for page to settle"done", # terminate task and give conclusion]target: Optional[int] = None # control id (preferred)x: Optional[float] = None # normalized coordinate (fallback)y: Optional[float] = Nonetext: Optional[str] = Noneduration_ms: int = 300
Prefer the control id and use coordinates only as a fallback—this is the core design for cross-device stability.
Output protocol: enforce structure
The model output must be constrained to strict, parseable JSON; free text is forbidden, otherwise parse-failure rates become a primary source of failure:
{"thought": "The search box on the home page is visible; locate the search entry first, then enter the keyword","action": { "type": "tap", "target": 12 },"risk": "low","confidence": 0.86}
Corresponding prompt skeleton:
You are a phone-operation agent. User goal: {goal}Current interface (id|label|type|clickable|center coords):{compressed_ui}Previous action and result: {last_step}Rules:1. Output exactly one JSON object, with no explanatory text;2. Prefer the target control id; when it is not clickable, use x/y;3. When an action involves payment, deletion, sending, or authorization, risk must be "high";4. When the goal is achieved, output done and fill in summary;5. If the interface is unchanged for two consecutive steps, change strategy; do not repeat the same action.
5. Main Loop: The Perception–Decision–Execution Closed Loop
def run_task(goal, device, model, max_steps=30):mem = Memory(max_tokens=8192)for step in range(max_steps):raw = device.capture_tree() # 1. perceivestate = compress(raw) # 2. compressplan = model.plan(goal, state, mem.render()) # 3. decideif plan.action.type == "done":return verify(goal, device, plan.summary) # 4. verifyif plan.risk == "high" and not confirm(plan): # 5. safety gatereturn "ABORTED_BY_USER"result = device.execute(plan.action) # 6. executewait_for_stable(device, timeout=2.0) # 7. perceive only after the page settlesmem.append(state, plan, result)if detect_loop(mem, window=3): # 8. infinite-loop detectionmem.drop_last_n(2)mem.inject_hint("Duplicate action detected; try coordinate tapping or scrolling the page")return "MAX_STEPS_EXCEEDED"
A few easily overlooked points that make or break success:
- Page-stability wait: screenshotting before interface animations finish reads intermediate-state controls and causes coordinate drift;
- Infinite-loop detection: repeated action hashes within a sliding window are the most common failure mode for long-horizon tasks;
- Terminal-state verification:
donedoes not equal success. You must re-perceive the interface to confirm the target state was actually reached (e.g., the order page appeared, the message was sent); otherwise you get “the model claims it is done, but nothing actually happened”—a false success.
6. Long-Horizon Tasks: Planning, Memory, and Cross-App Context
A model strong at single-step decisions sees its success rate drop noticeably on tasks of more than ten steps, due to context drift. Three mitigations:
1. Hierarchical planning (Plan-Act-Reflect)
First have the model produce a high-level plan (e.g., “search product → add to cart → submit order”), then execute step by step, doing a review and plan revision after each sub-goal. With the plan made explicit, even an intermediate error won’t lose the original intent.
2. Memory compression
Raw interface history grows linearly. Use a “key frames + summary” strategy: keep only interfaces where the state changed materially, and replace the rest with a one-line summary.
3. Cross-app context passing
Cross-app scenarios are both the most valuable and the hardest for mobile agents. Use three passing mechanisms in priority order:
- Deep link / intent jump: prefer URL Scheme or implicit Intent to pass parameters directly, bypassing UI operations—highest success rate;
- Clipboard relay: copy the content, switch apps, then paste; this requires handling clipboard permissions and content contamination;
- UI transcription: when the target app has no jump entry, you can only type in character by character—slow and error-prone.
Engineering-wise, build “action routing” into the framework layer: before executing, check whether a shorter path exists (deep link > direct control access > coordinate tap). This meaningfully reduces step count.
7. Deployment Forms and Tradeoffs
Form 1: Local / on-device deployment
- Advantages: data never leaves the device, lowest latency, fits privacy-sensitive scenarios;
- Costs: requires model quantization (INT8/INT4), memory constraints, and background keep-alive strategies;
- Fits: personal assistants, internal office automation.
Form 2: Cloud virtual devices
- Advantages: strong isolation, batch concurrency, and fully auditable and replayable tasks;
- Costs: requires building a stable cloud-phone/virtualization environment and solving realism problems (some apps detect non-genuine-device environments and refuse to run);
- Fits: regression testing, batch data processing, and unattended tasks.
Form 3: Hybrid deployment
On-device handles high-frequency light tasks and privacy tasks; the cloud handles heavy tasks and batch processing, decoupled through a unified action protocol. This is currently the more common production choice.
For engineering, you can use general cloud capabilities to cut operating costs: use a container platform to handle elastic scaling of cloud virtual devices, use object storage to archive task screenshots and trajectories for replay, and use logging and monitoring/alerting to track core metrics such as success rate, average steps, and timeout rate.
8. Security, Privacy, and Compliance Red Lines
Because agents hold real operation permissions, the risk model must be designed up front:
- High-risk action whitelist gate: transfers, payment confirmation, data deletion, message sending, system-setting changes, and permission grants all require human confirmation or secondary verification and cannot be fully automated;
- Principle of least privilege: request only necessary permissions such as the accessibility service; do not ask for read access to contacts, SMS, or anything unrelated to the task;
- Sensitive-region masking: when entering passwords or verification codes, forbid screenshots and logging; that segment of the interface frame is masked and discarded at the capture layer;
- Intent–execution separation: the model can only propose action suggestions; final execution authority stays with the framework’s safety layer. This avoids prompt injection—text like “click here to claim” embedded in an ad popup could be misread by the model as a user instruction;
- Full-chain auditing: for each task, persist the original instruction, step-by-step actions, interface snapshots, and result states, supporting after-the-fact tracing and issue reproduction;
- Idempotency and retriability: before a write operation, check whether it already took effect, to avoid duplicate orders or duplicate submissions caused by retries.
Point 4 is the most underestimated. Any text appearing on the interface should be treated as untrusted data, not a trusted instruction. This is the fundamental security difference between GUI agents and ordinary LLM applications.
9. Evaluation: How to Prove It Actually Works
Don’t look only at model leaderboards—GUI agents must be evaluated end to end. Build four layers of metrics:
- Task success rate (core): the fully-achieved ratio on a fixed task set, grouped by single-app / cross-app / long-horizon (>15 steps);
- Efficiency: average steps, average time, and time ratio relative to a human baseline;
- Robustness: variance of the success rate for the same task across multiple resolutions, OS versions, and app versions;
- Safety: count of high-risk actions triggered by mistake and unauthorized-access events—must be 0 before launch.
Suggestions for building the evaluation set: cover the typical paths of 50+ real high-frequency Chinese apps, including at least 20% tasks that require cross-app jumps, 15% with popups or ad interference, and 10% with deliberately injected weak-network and page-load-failure conditions.
10. Known Difficulties and Mitigations
| Difficulty | Symptom | Mitigation |
|---|---|---|
| Dynamic content | Feeds and ad slots make control ids unstable | Prefer text semantic matching over id matching |
| CAPTCHA and login state | Requires human intervention | Design a “pause–human takeover–continue” hook |
| Long-list locating | The target control is off-screen | Step-by-step scrolling + re-perceive each step; no one-shot blind swiping |
| Permission popups | System-level dialogs interrupt the flow | Explicitly recognize system-package overlays at the perception layer and handle them first |
| Timing races | Clicking too fast drops events | Force a wait for interface stability after each action; set a retry limit |
11. Conclusion
Mobile GUI agents represent a clear paradigm shift: the large language model moves from being a consumer of information to an initiator of operations. Its feasibility doesn’t depend on whether the model can memorize more knowledge, but on three things—the fidelity of interface perception, the rigor of the action protocol, and the completeness of the safety guardrails.
From an engineering perspective, a truly deployable solution should have: a ~9B vertical fine-tuned model handling decisions, a refined seven-action space handling expression, a strict JSON protocol handling parsing, control-tree compression handling context control, and high-risk gates handling risk mitigation. When these close into a loop, the agent can reliably run a cross-app long-horizon task on a real device.
If you’re building your own agent system, start with single-app, short-task, low-risk scenarios. Get the three fundamentals solid first—perception compression, page-stability waiting, and terminal-state verification—then gradually expand to cross-app and higher-complexity flows. The quality of the fundamentals sets the ceiling for everything that comes after.
