Mobile GUI Agent Deployment: A Complete Guide to Cloud Inference and On-Device Debugging

    This guide walks through deploying an open-source GUI agent that operates a smartphone, end to end. It covers planning GPU memory, disk, and bandwidth for vision-language model inference; provisioning a cloud GPU instance with drivers and SSH hardening; building a containerized vLLM inference service that exposes an OpenAI-compatible Chat Completions API; verifying the endpoint and reaching it securely over an SSH tunnel; and building the on-device agent loop that captures screenshots, maps model coordinates to real taps, and terminates tasks safely. It also includes a performance-tuning checklist, security and compliance practices, a troubleshooting reference, and directions for improving reliability.

    1. Background: Giving LLMs “Hands”

    Over the past year, the screen-understanding capabilities of multimodal large models have matured rapidly. A model can now jointly interpret a UI screenshot, the on-screen control structure, and a natural-language instruction. For example, from a request like “order me a milk tea with less sugar,” the model can reason about which app to open, what to search for, which entry to tap, and how to fill in the notes. Agents that operate directly on the graphical interface and can autonomously plan and execute multi-step tasks are commonly known as GUI Agents.

    An ordinary chatbot stops at “giving suggestions,” whereas a GUI Agent can directly “take action.” This unlocks two typical kinds of value:

    • Cross-app task automation: Ordering food, comparing prices, posting updates, and batch-organizing information often require switching back and forth between multiple apps—tasks that are time-consuming and repetitive to do manually.
    • Accessibility assistance: It offers people who are unfamiliar with complex operations a “one-sentence” way to get things done.

    Cloud LLM APIs are convenient to call, but for privacy-sensitive scenarios, those needing deep customization, or high-frequency calls, self-hosting a “cloud inference + local execution” assistant gives you more control. This article’s main thread is as follows: on a cloud GPU instance, deploy an open-source phone-operation agent—built around a vision-language model and exposing an OpenAI Chat Completions–compatible API—using containers to achieve inference acceleration and a persistent service; then, on a local computer, use device-debugging tools to connect a phone and complete joint debugging, ultimately building your own AI phone-operation assistant.

    2. Overall Architecture: A Perceive–Decide–Execute Loop

    From an engineering perspective, a phone-operation agent consists of four stages:

    1. Perception: Capture the screen screenshot and tappable-element information, converting the “screen state” into a representation the model can understand.
    2. Decision: The multimodal model combines the task instruction, current screen, and operation history to output the next action.
    3. Execution: The phone carries out taps, swipes, text input, back navigation, and so on based on the action instructions, and the interface changes accordingly.
    4. Memory: Save the task goal, summaries of past screenshots, and the action trajectory for later steps to reference and to judge whether the task is complete.

    The corresponding deployment topology has two ends:

    Location Contents Responsibility
    Cloud GPU instance Containerized inference service Model loading, visual understanding, and action generation; exposes the API
    Local computer Client and agent loop Connects the phone, captures screenshots, sends actions, calls the inference API

    This split has three benefits:

    • The phone needs no extra inference components and relies only on the system’s built-in debugging and accessibility features;
    • Compute is concentrated in the cloud, so local device performance is no longer a bottleneck;
    • The inference service is deployed and upgraded independently, and switching models does not affect the phone-side code.

    The following sections proceed in the order: compute planning → environment initialization → container setup → model deployment → service verification → phone debugging → optimization and security.

    3. Compute and Cost Planning Before Deployment

    GUI Agent inference workloads have two distinct characteristics: first, every step carries a high-resolution screenshot, so the input is image-heavy; second, tasks usually run for many consecutive steps, and the context keeps accumulating. So compute planning cannot focus only on “can it run,” but must also consider “does it run fast and stay stable.”

    3.1 VRAM Estimation

    VRAM usage is roughly made up of three parts:

    • Model weights: At FP16/BF16 precision, each billion parameters takes about 2GB of VRAM. A quantized version reduces this proportionally.
    • KV cache: Grows linearly with context length and the number of concurrent requests; the longer the context and the higher the concurrency, the more it uses.
    • Vision encoding and activation overhead: The higher the screenshot resolution, the more visual tokens the image is split into, and the more significant this overhead becomes.

    Common open-source phone-operation models usually have billions of parameters. Loading the full model at FP16 needs roughly 20GB or more of VRAM for weights plus baseline overhead; once you add visual tokens and a certain length of dialogue history, around 40GB of VRAM is a safer starting point. GPUs with this capacity are available in the GPU cloud servers of mainstream cloud platforms (for example, Baidu AI Cloud); such a GPU can load the model on a single card while leaving headroom for longer contexts, offering a fairly balanced cost-performance ratio. If you use a community-provided quantized version, a more modest GPU can also run it, but this may affect localization accuracy, so it’s advisable to compare results on a small scale first.

    3.2 Disk Capacity

    Model weights are typically tens of GB, and the container image itself is several GB. Add logs and old images from multiple iterations, and the system disk should reserve at least 100GB; if the platform supports mounting a separate data disk, mount the data disk to /data first and put both models and container data there, which makes scaling and data retention easier.

    3.3 Bandwidth and Billing

    Model weights and container images are large, and bandwidth directly determines initialization time. By binary conversion, a 32Mbps connection downloads at about 3.9MB/s, so downloading a 40GB model takes nearly 3 hours; a 100Mbps connection theoretically needs only about 1 hour. So it’s recommended to:

    • Max out bandwidth: If the instance is billed by traffic, you can downgrade afterward once the download finishes and the service is stable; if the bandwidth rate can only be changed once per day, be sure to set it correctly the first time.
    • Pay as you go: Hourly or usage-based billing suits validation and short-term tasks; release the instance promptly when the task ends to avoid idle charges.
    • Prepare images in advance: Build a container image and push it to an image registry, or download the model files on your local computer first and then upload them to the instance, which can greatly shorten initialization time. If you relay files through object storage, prefer a bucket in the same region as the instance to reduce cross-network transfer.

    3.4 Security Groups and Port Mapping

    Cloud platforms’ port mechanisms usually have two layers: the security group (firewall rules) determines whether the public network can reach the instance; port mapping determines how a public port is forwarded to an instance port. Open only the ports that are truly necessary. The inference API is an internal service, so a safer approach is to not expose it publicly and instead access it locally through an SSH tunnel (see Section 7).

    4. Cloud Instance Initialization

    4.1 Choosing an OS and Creating the Instance

    Any mainstream long-term-support (LTS) Linux distribution such as 22.04 works; its newer kernel version has good compatibility with mainstream GPU drivers. After creating the instance, be sure to confirm its status, public access method, and port-mapping table in the console, and record the SSH login username and port.

    4.2 Installing GPU Drivers

    GPU cloud servers usually come with drivers preinstalled; otherwise, you need to install the official driver for your GPU. After connecting to the server, you can first check whether the accelerator is recognized, along with the driver and CUDA versions:

    1. # View the accelerator model, VRAM usage, and driver/CUDA versions
    2. nvidia-smi

    If the driver is missing, install the appropriate version following the official documentation for your OS and reboot; if VRAM is already occupied, first investigate leftover processes.

    4.3 Basic System and SSH Hardening

    After the basic configuration, it’s recommended to do the following hardening steps:

    • Key-based login: Write your local public key into the server’s ~/.ssh/authorized_keys and set 600 permissions; then disable password login.
    • Restrict access sources: In the security group, allow the SSH port only from IP addresses you commonly use, and change the default port.
    • Limit login attempts: Use the socket-activation mechanism provided by the system service manager (systemd) to limit the number of connections from the same address per unit of time, reducing brute-force risk.
    • Update promptly: Install system security updates and configure a faster software mirror.

    5. Container Environment Setup

    5.1 Why Use Containers

    Model inference is sensitive to the runtime environment, and different framework versions and different GPU-driver combinations can trigger dependency conflicts. Containers package runtime dependencies together, bringing three benefits: reproducible environments, no need to reconfigure when migrating to a new instance, and low cost to destroy and rebuild on failure.

    5.2 Installing a Container Engine

    Install the container engine from the official repository to make later upgrades easier:

    1. # 1. Remove old versions to avoid conflicts
    2. for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
    3. sudo apt-get remove -y $pkg
    4. done
    5. # 2. Add the official GPG key
    6. sudo apt-get update
    7. sudo apt-get install -y ca-certificates curl
    8. sudo install -m 0755 -d /etc/apt/keyrings
    9. sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    10. sudo chmod a+r /etc/apt/keyrings/docker.asc
    11. # 3. Add the repository (amd64 + Ubuntu 22.04 as an example)
    12. echo \
    13. "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
    14. https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    15. sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    16. # 4. Install and start
    17. sudo apt-get update
    18. sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    19. sudo systemctl enable --now docker
    20. # 5. Run without sudo as a non-root user (requires re-login to take effect)
    21. sudo usermod -aG docker $USER
    22. docker run --rm hello-world # verify the installation

    If the official repository is slow to reach on your network, you can switch to a mirror site, or configure a registry-mirror address in the daemon configuration file and then restart the service.

    5.3 Letting Containers Use the GPU

    By default, containers cannot access the GPU, so you need to install the container GPU runtime component (nvidia-container-toolkit) and restart the container engine. On Ubuntu, for example:

    1. # Add the repository and install the runtime component
    2. curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
    3. sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    4. curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    5. sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    6. sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    7. sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
    8. sudo nvidia-ctk runtime configure --runtime=docker
    9. sudo systemctl restart docker
    10. # Verify the GPU is visible inside the container
    11. docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

    Note: using the GPU inside a container requires a compatible driver already installed on the host; for the CUDA version correspondence, refer to the official documentation, and don’t confuse the CUDA version inside the container with the driver version.

    5.4 Adjusting the Storage Location

    Put model files in /data and explicitly mount them as a container data volume, so the image layers and model files reside on the same data disk, preventing the system disk from filling up quickly. If the platform provides an image registry, push the built inference image to it so a new instance can pull it directly after creation, avoiding repeated builds.

    6. Model Service Deployment and Inference Acceleration

    6.1 Choosing an Inference Framework

    Open-source phone-operation agent projects usually come with a vLLM-based launch script. When choosing an inference framework, it’s worth focusing on four things:

    • VRAM efficiency: Whether it supports paged KV-cache management techniques like PagedAttention to reduce VRAM fragmentation;
    • Scheduling: Whether it supports Continuous Batching to improve throughput under concurrency;
    • Model and quantization compatibility: Whether it supports the target model architecture and mainstream quantization formats such as AWQ, GPTQ, and FP8;
    • API compatibility: Whether it provides a Chat Completions–style API for seamless client switching.

    vLLM performs well across these dimensions and has a mature community ecosystem, making it a good first choice.

    6.2 Obtaining Model Weights

    Download the corresponding weights from the model hosting platform. Keep three points in mind when choosing weights:

    • Prefer officially released versions from trustworthy sources, and confirm file integrity with checksums;
    • Screenshot inputs are sensitive to resolution, so prefer versions with stronger vision capabilities;
    • When VRAM is tight, try community quantized versions, but compare operation success rates on a real task set—don’t just check whether loading succeeds.

    6.3 Starting the Inference Service

    Using the script provided by the project as an example (port 8000 is the container’s internal port; the actual public mapping takes precedence):

    1. cd AutoGLM
    2. # Point the model path to the mounted weights directory; adjust context length and VRAM utilization as needed
    3. bash scripts/launch_vllm.sh \
    4. --model-path /data/models/AutoGLM-9B \
    5. --port 8000

    The meaning of the key parameters and tuning suggestions:

    Parameter Meaning Tuning suggestion
    —model-path Model weights path Point to the mounted directory; avoid baking weights into the image
    —port Service port Match the internal port in the container port mapping
    —max-model-len Maximum context length Screenshots use a lot of tokens; too small truncates history, too large squeezes the KV cache
    —gpu-memory-utilization Upper bound on VRAM usage ratio 0.85–0.90 is fairly safe; too high easily causes OOM
    —tensor-parallel-size Number of cards for tensor parallelism Set to 1 if a single card suffices; for cross-card, it must divide the card count
    —dtype Compute precision Use half precision or a quantized version when VRAM is tight

    6.4 Running the Container Persistently

    During debugging, you can run the container in the foreground directly with the container engine. For production use, it’s recommended to start it in the background, fix the container name, and set automatic restart:

    1. # Build the image (first time)
    2. docker build -t auto-glm:latest .
    3. # Run in the background: map host port 8000 to container port 8000
    4. docker run -d --name auto-glm --gpus all --restart unless-stopped \
    5. -p 8000:8000 -v /data/models:/data/models \
    6. auto-glm:latest

    After the container starts, model loading takes some time, during which VRAM usage gradually rises; you can use nvidia-smi to observe loading progress. If you need it to start on boot, write the above command as a systemd unit; if one instance runs multiple containers, it’s recommended to manage them with an orchestration tool and configure resource limits.

    7. Service Verification and Client Debugging

    7.1 Health Check and API Verification

    Once model loading finishes, first confirm the service is alive and check the model list:

    1. # Check the service and model list locally on the server
    2. curl -s http://127.0.0.1:8000/v1/models

    If you’re using a local container environment like Docker Desktop, you can replace localhost with host.docker.internal to reach the host service.

    Then send a chat request with a screenshot:

    1. curl -s http://127.0.0.1:8000/v1/chat/completions \
    2. -H "Content-Type: application/json" \
    3. -d '{
    4. "model": "autoglm",
    5. "messages": [
    6. {"role": "system", "content": "You are a GUI agent. You are given a task and a screenshot of the screen."},
    7. {"role": "user", "content": [
    8. {"type": "text", "text": "Open Settings and check the current battery level"},
    9. {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_SCREENSHOT>"}}
    10. ]}
    11. ],
    12. "temperature": 0.1,
    13. "max_tokens": 1024
    14. }'

    The expected response contains fields such as Thought (reasoning) and Action (action), describing where to tap next or which operation to perform. The client, implemented against a Chat Completions–style API, only needs to change base_url and the model name to switch models.

    7.2 Tunneling and Access Security

    The inference API has no built-in authentication and should not be exposed directly to the public network. For local debugging, an SSH tunnel is recommended to map the remote port to a local one:

    1. # -N runs no remote command; -L local-port:target-address:target-port; -f runs in the background
    2. # Using the port mapping 8000->38888 above as an example
    3. ssh -N -L 8000:127.0.0.1:38888 user@<server-public-ip>

    After that, set base_url in your local code to http://127.0.0.1:8000. If you need to share the service among several people, add a reverse proxy in front of the inference service with key-based authentication and rate limiting, and don’t expose the port directly to the public network.

    8. On-Device Debugging: Bridging the Last Mile

    8.1 Device Preparation

    1. Turn on “Developer options” on the phone (usually unlocked by tapping “Build number” repeatedly in system settings) and enable “USB debugging”;
    2. Connect the phone to the computer with a data cable, and confirm “Allow” in the authorization prompt that appears on the phone;
    3. Confirm the device is connected, and set both the decoding and encoding device to the same phone (some models require selecting “Default USB configuration” or enabling “USB debugging (Security settings)” in developer options);
    4. If using wireless debugging for the first time, complete pairing with the pairing code on the same local network before connecting.

    If the device list is empty, check in order whether the cable supports data transfer, whether the USB interface mode has been switched to “charging only,” and whether the driver is installed. When a cloud instance needs a direct connection to the phone, you can forward a local USB device to the remote end via USB/IP and similar methods, but this approach is limited in stability, so the “cloud inference + local execution” split deployment is preferred.

    8.2 Common Debugging Operations

    Purpose Operation
    List devices The debugging tool’s devices subcommand
    Install an app The install subcommand, supporting -r for reinstall/overwrite and -g to automatically grant runtime permissions
    Run phone-side commands The shell subcommand, which prints the result directly to the terminal
    Forward a port The forward subcommand, mapping a phone port to a computer port
    View logs The logcat subcommand, best used with filter conditions

    When performing operations like screenshots and taps, it’s recommended to prepend -s to the debugging command to avoid misoperating the wrong device when several are connected.

    8.3 Screen Perception: Screenshots and the Control Tree

    The agent’s “eyes” have two information sources:

    • Screen screenshots: Use the debugging channel to invoke the system’s screenshot capability on the phone, then pull the image to the local machine. Too high a resolution increases visual tokens and inference time, so you can first scale it down proportionally or convert it to JPEG before uploading.
    • Control tree: Export the current screen’s XML control tree via the system’s automation framework; it contains each control’s coordinates, text, and clickable attributes. The control tree is precise but needs extra parsing, and some apps block its export.

    In real projects, a common approach is to combine the two: screenshots as the primary source and the control tree as a supplement, used to calibrate tap coordinates and judge element states.

    8.4 Coordinate Mapping and Tapping

    The coordinates output by the model are usually based on the pixels of the screenshot or input image, while the phone’s actual resolution may differ. You must convert before executing:

    1. scale_x = real_width / image_width
    2. scale_y = real_height / image_height
    3. tap_x, tap_y = int(x * scale_x), int(y * scale_y)

    If the phone is in landscape orientation or has a notch or status-bar offset, extra correction is needed. Taps are sent through the phone’s input-injection interface; for text input, prefer simulating an input method over injecting character by character, to correctly handle complex input such as CJK text.

    8.5 The Agent Loop and Task Termination

    A complete task loop can be abstracted as:

    1. def run_task(device, client, task, max_steps=20):
    2. for step in range(max_steps):
    3. screenshot = device.screenshot() # 1. Perceive
    4. response = client.chat(task, screenshot) # 2. Decide
    5. action = parse_action(response) # 3. Parse the action
    6. if action.type in ("finish", "call_user"):
    7. return response # Task done or human intervention needed
    8. device.execute(action) # 4. Execute
    9. # 5. Memory: add this step's screenshot and action summary to the context
    10. return "Reached the maximum number of steps; task not completed"

    Engineering-wise, it’s advisable to set three kinds of termination conditions:

    • The model explicitly judges the task complete;
    • The screen doesn’t change for several consecutive steps or the same action repeats, indicating a loop;
    • The maximum number of steps is reached, handing it over to a human.

    At the same time, write each step’s screenshot, model output, and action to a local log to make it easier to review failure causes.

    9. Performance Optimization and Security/Compliance

    9.1 Performance Optimization Checklist

    1. Control image size: Appropriately lower screenshot resolution and use JPEG compression; crop irrelevant regions when necessary;
    2. Compress history context: Keep screenshots only for the most recent few steps, and replace earlier steps with text summaries;
    3. Quantization and parallelism: Use high precision when VRAM is plentiful to improve quality; use a quantized version or tensor parallelism when VRAM is tight;
    4. Control concurrency: GPU VRAM is limited, and too-high inference concurrency causes queuing, so use request queues and rate limiting to guarantee time-to-first-token latency;
    5. Warm-up and persistence: Model loading takes a long time, so keep the service resident to avoid a cold start for every task;
    6. Monitor observability: Record request volume, time to first token, per-step latency, and GPU utilization; connect to logging and monitoring alerts to catch anomalies promptly.

    9.2 Security and Compliance Essentials

    A phone-operation agent inherently touches users’ accounts, messages, and payment information, so deployment and use must hold a few bottom lines:

    • Least privilege: Grant the assistant only the authorizations necessary to complete the task; operations involving payments, transfers, or deletion must have human confirmation;
    • Privacy protection: Screenshots and logs may contain sensitive information such as verification codes and chat records; store them encrypted, clean them up regularly, and don’t upload them to untrusted third-party services;
    • Respect platform rules: Do not use it for brushing orders, exploiting promotions, evading risk controls, or other uses that violate an app platform’s terms of service, to avoid account restrictions;
    • Access control: The inference service must be authenticated; keep image-registry and object-storage access credentials safe, and don’t hard-code them into code or images;
    • Personal information protection: When processing others’ information, follow applicable data-protection and privacy laws and regulations and obtain the necessary authorization.

    10. Troubleshooting

    Symptom Possible cause Recommendation
    Container fails to start, GPU not found Container GPU runtime not installed or not configured Install the runtime component and restart the container engine
    VRAM overflow (OOM) during model loading Context length too large or VRAM utilization too high Lower the context length and VRAM ratio, or switch to a quantized version
    Service unreachable from the public network Port mapping mismatch, security group not allowing it, or service not listening on 0.0.0.0 Verify the port-mapping table and security-group rules, and confirm the listening address
    Extremely slow model download Insufficient bandwidth or source-side throttling Increase bandwidth or use a mirror/local cache
    Tap position is offset Coordinates not converted or a status-bar offset Convert by the scale ratio and calibrate the offset
    Slow responses, stuck tasks Too-high concurrency or context too long Lower concurrency, compress history, check service logs
    Phone not recognized Not authorized, or a cable/USB-mode issue Re-authorize, change the cable, check the debugging mode
    Disk warning Accumulated images and logs Clean up unused images and logs, and migrate data to the data disk

    11. From “Working” to “Great”: Advanced Directions

    • Structured output constraints: Require the model to output actions in a fixed format to reduce parsing failures;
    • Reflection and retry: When an action is ineffective, have the model re-plan with the new screenshot instead of simply repeating;
    • Task templating: Solidify high-frequency tasks into parameterized flows, calling the model only at key decision points to reduce cost;
    • Device–cloud collaboration: Try small on-device models for simple tasks, and hand complex tasks to a cloud model;
    • Effect evaluation: Build your own task set and periodically track task completion rate, average step count, and human-intervention rate, using data to drive optimization.

    Conclusion

    Deploying an AI phone-operation assistant is essentially about connecting the model capability of “screen understanding” to a real device. The difficulty lies not in any single technique but in end-to-end engineering: compute planning sets the ceiling on the experience, containerization ensures a reproducible environment, inference acceleration determines response speed, coordinate mapping and the agent loop determine whether a task truly gets done, and security and compliance determine how far the system can go. After building it following this article’s process, it’s recommended to first validate repeatedly with low-risk tasks, then gradually expand to more complex scenarios, so that AI truly becomes an assistant that “takes action” for you.