Open-Source On-Device AI Agents: A New Approach to Mobile AI Privacy

    Phone AI agents need deep system access, but running the model locally is still constrained by mobile hardware. This article examines a phone-side agent framework open-sourced by Zhipu together with a roughly 9B-parameter model, in which the phone handles screen perception while inference runs on the user's own PC or server. It covers the data flow and trust boundaries, the practical benefits and the limits of the design, the likely three-stage evolution from cloud inference to on-device inference, a checklist of items developers should verify before integrating, and the role of cloud-side TEEs and federated computing.

    Why phone AI is hard to run fully locally

    Over the past two years, phone AI assistants have moved from voice Q&A to cross-app actions. That progression runs into a long-standing conflict: a model has to be capable enough, which means enough parameters, while phone SoCs offer limited compute and memory and struggle to run large models stably for extended periods. This is why most vendors still upload data for cloud inference.

    Chat logs, contact lists, and payment information are all sensitive. Even with encrypted transmission, many users are still uneasy. Phone agents raise the stakes further, because they need to read screen content, simulate taps, and infer what the user is trying to do. That requires deeper system permissions than an ordinary voice assistant, and the privacy exposure grows accordingly.

    On-device inference, meaning the model runs entirely on the device, is the intuitive fix: data never leaves the hardware, and the number of leakage paths shrinks. The practical constraints, however, are not loose. Most current phone SoCs run quantized models below 7B reasonably smoothly. For agent tasks that involve visual input, long context, and multi-turn reasoning, 9B-class models still struggle to run in real time on a phone. Industry views differ on when on-device agents at a higher capability level will become practical; some expect several more chip generations, with the timing depending on progress in process nodes, memory bandwidth, power consumption, and model compression.

    Zhipu’s open-source framework: a transitional approach

    Zhipu recently open-sourced a phone-side agent framework and released a purpose-built model of roughly 9B parameters. Rather than forcing the model to run on the phone, the design splits the task in two: the phone handles perception, while inference runs on a local PC or a self-hosted server.

    The data flow works roughly as follows.

    First, the on-device agent performs screen capture, UI element parsing, and user intent recognition on the phone itself. These steps are computationally light and well suited to mobile hardware.

    Second, the phone sends the processed data over a LAN or an encrypted channel to the user’s own computer or server. Once inference completes, the result returns to the phone in the form of operation commands.

    Third, the phone switches its connection target according to its environment. At home or in the office it connects directly to local hardware; when out, it connects to a self-hosted cloud service or an edge node.

    On this path, the phone, the LAN, and the compute device each form a trust boundary. Which end stores the screenshots, commands, and logs determines the scope of data exposure, and this has to be settled before deployment.

    For the record, this article has not verified the project’s repository URL, license, or specific version number; readers should rely on the official release page.

    What this approach buys you

    First, a more controllable data flow. Data stays on devices or servers the user owns and never passes through a public cloud platform, which removes third parties from the loop. This follows directly from the architecture, but it does not mean the absence of risk.

    Second, typically lower latency. Round-trip time inside a LAN is generally shorter than to a remote public cloud. Actual numbers depend on network conditions, model size, and hardware, so users need to measure for themselves.

    Third, compute that scales on demand. Users can run a single high-end PC, or assemble a small cluster from several consumer GPUs, and adjust model size and concurrency according to available VRAM.

    Fourth, more flexible model replacement. The design is not bound to any one cloud vendor’s hosted API, so developers can swap, fine-tune, or quantize models.

    The limits of this approach

    First, the technical barrier is high. Users must configure networking, deploy inference services, and manage keys and certificates. It targets developers rather than ordinary consumers.

    Second, it is not a zero-risk design. The phone-side agent depends on high-privilege capabilities such as accessibility services. If the framework or the app has vulnerabilities, screen content may be read. Local logs, screenshot caches, and temporary files can also retain sensitive information. The model may additionally misread the interface and carry out the wrong action.

    Third, the framework is closer to a development tool than an out-of-the-box application. A product aimed at ordinary users still needs an outer layer: a conversational interface, permission management, and task visualization.

    From a transitional setup to the long-term direction

    The evolution of privacy protection here can be split into three stages.

    Stage one is pure cloud inference. Models are deployed in data centers, and the phone only collects data and displays results. The advantage is strong model capability and fast iteration; the drawback is that data depends entirely on a third-party trust chain and on network conditions.

    Stage two is a local compute pool plus cloud model services. The phone does not connect directly to a public cloud but routes through the user’s own hardware, confining data movement to a range the user controls. Developers also get to validate how well models understand real interfaces and accumulate on-device optimization experience.

    Stage three is on-device inference. As process technology advances and quantization, distillation, and sparsification mature, 8B to 9B-class compact agent models may run smoothly on phones. At that point, the local compute pool may serve mainly as an optional accelerator for demanding scenarios. This stage is a long-term direction, not a settled outcome.

    Small models also have a capability ceiling. Parameter count and the breadth of training data limit what they can do. Complex tasks such as multi-step web operations and cross-app data consolidation may be beyond on-device small models. The more realistic architecture is therefore tiered, hybrid inference: simple intents answered on-device immediately; complex tasks handed to a home server or edge node the user trusts; and cloud access used only after explicit user authorization, when broad knowledge or stronger reasoning is required.

    What developers should verify before integrating

    The checklist below reflects general practice and does not represent official commands.

    1. Verify the source. Confirm the official repository, license, model version, and dependency environment, and record version numbers so results can be reproduced later.

    2. Choose a quantization bit width. Quantization lowers VRAM usage but costs some precision. Establish a quality baseline at higher precision first, then reduce the bit width step by step while comparing speed, VRAM usage, and output quality.

    3. Define measurement metrics. Pin down where latency starts and ends, for example from screenshot to command return. Record network, inference, and phone-side time separately, along with peak VRAM usage and power draw.

    4. Test Android permissions. Accessibility services, overlay windows, and background execution policies differ substantially across device brands, so test on several real devices.

    5. Distinguish transport encryption from end-to-end encryption. TLS 1.3 with mutual certificate authentication protects the communication link, but that is not the same as application-layer end-to-end encryption. If genuine E2E is required, design the encryption scheme and key management separately at the application layer.

    6. Reduce sensitive data exposure. Before taking a screenshot, mask regions such as the status bar, password fields, and payment information, and send only the interface content required to complete the task.

    7. Manage locally retained data. Set retention periods for logs and screenshot caches, store them encrypted, and clear them on a schedule.

    8. Design failure recovery. Use a task state machine to support resuming from a checkpoint and manual takeover, and keep an operation log so actions can be audited afterward.

    Cloud-side trusted execution environments and federated computing

    Two other routes exist while on-device compute matures.

    The first is a cloud-side trusted execution environment (TEE). It carves out a hardware-isolated, encrypted region inside the processor that limits how far cloud platform administrators can see runtime data. It depends on specific hardware and vendor implementations, remote attestation requires a trust chain, and research has demonstrated side-channel attacks, so the concrete security assumptions need to be evaluated.

    The second is federated computing. Devices train locally and upload only model parameter updates rather than raw data. Parameters themselves can still leak information, so measures such as differential privacy or secure aggregation are usually needed alongside.

    Both routes are complex to implement and carry considerable performance overhead. Near-term adoption on mobile remains difficult for now.

    Conclusion

    The local compute pool approach moves phone-agent inference from the public cloud onto hardware the user controls, reducing the paths through which data can flow out. It cannot eliminate every risk, but it gives users and developers an option they can inspect and control. For privacy-sensitive scenarios, validate permission scope, log retention, and network configuration in a test environment before deciding whether to adopt it in production.