Local Deployment of a Mobile-Use Agent: From Environment Setup to Service Launch

    This article explains how to deploy a mobile-use agent locally: assessing hardware, pinning software versions, downloading and verifying model files, reading model hyperparameters, writing a vLLM launch script, configuring multi-GPU tensor parallelism and multimodal image limits, resolving shared-memory issues, and completing API validation and performance testing. Placeholders and commands shown in the article, including `<MODEL_ID>`, must be replaced or adapted to the actual environment.

    1. Why Deploy a Mobile-Use Agent Locally

    These models can read mobile screenshots, identify buttons and input fields, plan operation steps, and output coordinates for taps or swipes. This capability is useful for automated testing, accessibility assistance, and workflow automation.

    During development, repeatedly calling public cloud APIs adds cost and network latency and slows iteration. Screenshots may also contain user data, so privacy and compliance requirements may apply. Local or private deployment can make calls faster and keep data on your own machines.

    The steps in this article are: assess hardware; pin software versions; download and verify the model; read the model configuration; write the launch command; configure multi-GPU parallelism and image parameters; address shared-memory issues; and complete API validation and performance testing. Replace placeholders such as <MODEL_ID> with your actual values.

    2. Pre-Deployment Assessment: VRAM, System Memory, and Shared Memory

    2.1 VRAM Estimation

    The VRAM used by model weights can be estimated roughly as parameter count × bytes per parameter. For a model with about 9 billion parameters in bf16 precision, each parameter takes 2 bytes, so the weights are about 18 GB.

    That 18 GB is only a lower bound. The vision encoder, activations, KV cache (the cache that stores historical attention state), and CUDA kernel workspace also consume VRAM. If a single 24 GB GPU holds the weights, very little headroom remains; long-context and multi-image requests can easily trigger out-of-memory errors.

    In practice, choose a single GPU with 40 GB or more of VRAM, or use two or more GPUs with at least 24 GB each for tensor parallelism. After deployment, you can control VRAM usage with --gpu-memory-utilization and --max-model-len; later sections describe specific values.

    2.2 System Memory

    When the model is loaded, weight files are first read into system memory and then transferred to VRAM. System memory should be at least 64 GB. If memory is insufficient, loading usually fails directly.

    2.3 Shared Memory

    Multi-process inference passes tensors between processes through shared memory (/dev/shm). Many containers allocate only 64 MB by default, which is insufficient for large tensor transfers.

    Check this on both the host and inside the container:

    1. df -h /dev/shm

    If the capacity is only tens of MB, handle it in Section 8.

    3. Software Environment: Pin Versions

    Strict version compatibility exists among the inference framework, PyTorch, and CUDA. If versions are not pinned, the same script may behave differently on different machines. Create a separate virtual environment for this project and record the version numbers in the deployment documentation.

    First, check the driver and GPUs:

    1. nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
    2. nvidia-smi

    The “CUDA Version” shown in the upper-right corner of nvidia-smi is the maximum CUDA version supported by the current driver. The CUDA version bundled with the PyTorch package cannot exceed that value.

    Create the environment and install:

    1. python3 --version
    2. python3 -m venv ~/venvs/mobile-agent
    3. source ~/venvs/mobile-agent/bin/activate
    4. pip install vllm
    5. python -c "import torch, vllm; print(torch.__version__, torch.version.cuda, torch.cuda.device_count(), vllm.__version__)"

    Installing vLLM directly is recommended, so that it can pull a matching PyTorch according to its own dependencies. If you need to specify a PyTorch version, first check the PyTorch and CUDA versions required in the vLLM release notes, then install from the corresponding CUDA source provided on the PyTorch official site. For example, in --index-url https://download.pytorch.org/whl/cu121, the number after cu must match the CUDA version supported by the driver.

    The output of the last command should show the PyTorch version, CUDA version, number of visible GPUs, and vLLM version. If the reported GPU count is less than the actual number of cards, it is usually a driver or container permission issue; resolve it before continuing.

    4. Download and Verify the Model

    4.1 Download Weights

    Create a dedicated directory first to avoid mixing it with other projects:

    1. mkdir -p /home/user/projects/mobile-agent-model
    2. pip install modelscope
    3. modelscope download --model <MODEL_ID> --local_dir /home/user/projects/mobile-agent-model

    <MODEL_ID> is usually in the format “organization/model-name”. Model files may be tens of GB, and if a download is interrupted, run the same command again to resume.

    4.2 Check the Directory Structure and Verify Files

    After download, the directory should contain at least the following:

    • Weight files, usually in .safetensors format, possibly split into multiple shards.
    • config.json, which records hyperparameters such as layer count and attention head count.
    • Tokenizer files, such as tokenizer.json or tokenizer.model.
    • Image preprocessing configuration, such as preprocessor_config.json.
    1. ls -lh /home/user/projects/mobile-agent-model

    If the repository provides a SHA256 manifest (a common file name is SHA256SUMS, or the model page may provide a hash list), run the following in the model directory:

    1. cd /home/user/projects/mobile-agent-model
    2. sha256sum -c SHA256SUMS

    Every line should show OK. If any line shows FAILED, redownload only the corresponding file. Do not start the service directly; otherwise, the loading stage may report errors that are difficult to diagnose. If the repository does not provide a manifest, at least confirm that all shards are complete and that file sizes match those stated on the model page.

    5. Read Key Hyperparameters

    Later configuration depends on the model’s maximum length, number of attention heads, and number of layers. Read these values from config.json with Python; no additional tools are required:

    1. python3 -c "import json; c=json.load(open('/home/user/projects/mobile-agent-model/config.json')); t=c.get('text_config', c); print('max_position_embeddings:', t.get('max_position_embeddings')); print('num_attention_heads:', t.get('num_attention_heads')); print('num_key_value_heads:', t.get('num_key_value_heads')); print('num_hidden_layers:', t.get('num_hidden_layers'))"

    Multimodal models often place language parameters under text_config, so the script tries to read that first and falls back to the top level if it is unavailable.

    The correspondence between these fields and launch parameters is as follows:

    • max_position_embeddings: the maximum context length supported by the model. --max-model-len should not exceed it.
    • num_attention_heads: must be divisible by --tensor-parallel-size. For example, with 32 heads, you can use 2, 4, or 8 GPUs in parallel; with 3 GPUs, the heads cannot be split evenly.
    • num_key_value_heads: affects KV cache size. The smaller this value is, the more context the same VRAM can hold.

    6. vLLM Launch Script and Multi-GPU Parallelism

    Save the launch command as a script (start_agent.sh) for reproducibility and version control:

    1. #!/usr/bin/env bash
    2. set -euo pipefail
    3. source ~/venvs/mobile-agent/bin/activate
    4. export CUDA_VISIBLE_DEVICES=0,1
    5. vllm serve /home/user/projects/mobile-agent-model \
    6. --served-model-name mobile-agent \
    7. --host 127.0.0.1 \
    8. --port 8000 \
    9. --tensor-parallel-size 2 \
    10. --dtype bfloat16 \
    11. --max-model-len 8192 \
    12. --gpu-memory-utilization 0.85 \
    13. --limit-mm-per-prompt '{"image": 2}' \
    14. --trust-remote-code

    Parameter meanings:

    • --tensor-parallel-size: splits the weights across multiple GPUs. The value must match the number of GPUs in CUDA_VISIBLE_DEVICES.
    • --max-model-len: maximum total length allowed for a single request, including input and output. Lowering it can free VRAM for the KV cache.
    • --gpu-memory-utilization: the fraction of VRAM that vLLM may occupy. If set too high while other processes also use VRAM, errors are more likely. Start with a value between 0.8 and 0.9.
    • --host 127.0.0.1: allows access only from the local machine. If LAN access is required, expose the service through a reverse proxy with authentication; do not listen directly on 0.0.0.0.
    • --trust-remote-code: allows execution of custom code in the model repository. Use it only when you trust the source.

    After startup, first check the logs for a message that model loading is complete and for the port listening message. If you encounter an out-of-memory error, lower --max-model-len, reduce the number of images in --limit-mm-per-prompt, or lower --gpu-memory-utilization.

    7. Multimodal Image Parameters

    Mobile screenshots often have high resolution and are split into many visual tokens, consuming context length. Therefore, control both the number of images per request and the pixel budget per image.

    --limit-mm-per-prompt '{"image": 2}' limits each request to at most two images. If your workload needs inference on only a single screenshot, setting it to 1 can significantly reduce VRAM pressure.

    If the model’s image processor supports a maximum-pixel parameter, set it through --mm-processor-kwargs. The exact field name varies by model; use preprocessor_config.json and the model page documentation as the reference. For example:

    1. --mm-processor-kwargs '{"max_pixels": 1003520}'

    When adjusting the pixel limit, first use a fixed set of screenshots to test whether coordinates are still recognized accurately; then decide whether to lower the resolution. Smaller images are faster, but buttons and small text may fail to be recognized.

    8. Shared Memory and Troubleshooting

    8.1 Shared Memory Inside Containers

    If the service runs in Docker, increase shared memory when starting the container. The following example uses 16 GB:

    1. docker run --gpus all --shm-size=16g -v /home/user/projects/mobile-agent-model:/models -p 127.0.0.1:8000:8000 <IMAGE_NAME> vllm serve /models --tensor-parallel-size 2

    Alternatively, you can use --ipc=host to let the container use the host’s shared memory directly, but this reduces container isolation. Use it only in trusted environments.

    After the change, enter the container and run df -h /dev/shm to confirm that the capacity has changed. A check on the host does not reflect the situation inside the container; both locations need to be verified.

    8.2 Common Issues

    • An error contains “bus error” or shared-memory-related information: check the /dev/shm capacity and increase --shm-size.
    • Out-of-memory during loading: lower --max-model-len, reduce the number of images, or lower --gpu-memory-utilization.
    • A request reports that the input is too long: the messages plus image tokens exceed --max-model-len. Reduce text length or the number of images, or increase the value appropriately, but reassess VRAM usage.
    • A tensor-parallelism splitting error occurs: check whether --tensor-parallel-size divides the number of attention heads evenly.
    • The model loads but inference is very slow: confirm that CUDA_VISIBLE_DEVICES points to free GPUs that are performing normally. Use nvidia-smi to check usage.

    Before modifying configuration or deleting old files, back up the model directory and launch script. Do not run recursive deletion on the model directory, especially on production machines. If you need to clean up old versions, move them to a backup directory first and delete them only after confirming the service works normally.

    9. API Validation and Performance Testing

    9.1 Health Check

    After the service starts, first confirm that it responds normally:

    1. curl http://127.0.0.1:8000/health
    2. curl http://127.0.0.1:8000/v1/models

    The first command returning HTTP 200 indicates that the service is alive. The second command should list mobile-agent, which confirms that the model name matches the launch parameter.

    9.2 Inference Test with an Image

    Prepare a screenshot named screen.png, encode it as base64, and send the request:

    1. IMG=$(base64 -w 0 screen.png)
    2. curl http://127.0.0.1:8000/v1/chat/completions \
    3. -H "Content-Type: application/json" \
    4. -d "{\"model\": \"mobile-agent\", \"messages\": [{\"role\": \"user\", \"content\": [{\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/png;base64,${IMG}\"}}, {\"type\": \"text\", \"text\": \"Find the login button and output the coordinates of its center.\"}]}], \"max_tokens\": 256}"

    The response should contain an action description and coordinates. To check whether the coordinates are correct, compare them item by item against manually labeled screenshots.

    9.3 Performance Testing Method

    During load testing, keep the following conditions fixed; otherwise, results cannot be compared:

    • Input: the same batch of screenshots and the same set of instructions, with image resolution and text length recorded.
    • Concurrency: test 1, 4, and 8 concurrent requests separately, and record results for each level.
    • Metrics: time to first token, output tokens per second, request success rate, and peak VRAM usage.
    • Duration: run each level for at least several minutes and discard data from the warm-up phase.

    Record the measured numbers in the deployment documentation, along with the machine model, driver version, vLLM version, and launch parameters. These numbers are valid only for the current environment; after changing machines or versions, retest.

    After completing the steps above, the service can be connected to automated tests or business workflows. Before going live, run a regression pass over a fixed screenshot set to confirm that recognition results match previous results.