How LLM Agents Control Mobile Apps and What the Token Surge Means

1. Introduction: Two Industry Signals, One Inflection Point

The industry has recently produced two notable technical signals. First, system-level agent capabilities are moving from conceptual demos toward practical usability. Large language models are being given the ability to directly control mobile apps, performing real operations such as tapping, swiping, and filling out forms on behalf of users. Second, public usage statistics show that token consumption for Chinese LLMs has climbed sharply in a short period, with agentic workloads driving much of the growth.

These two developments may appear independent, but they are actually two sides of the same technical curve. The former represents a shift in interaction paradigm, from generating content to executing tasks. The latter represents a shift in usage scale, from low-frequency Q&A to high-frequency agent loops. This article follows both threads, dissecting the complete technical pipeline behind LLM-driven app control and discussing the inference and cost engineering problems that accompany the token surge.

2. How LLMs Control Mobile Apps: Agent Architecture

Traditional mobile automation relies on script recording and playback. When the UI changes, the scripts break, and generalization is poor. LLM-driven agents are different because they replace fixed scripts with a closed loop of understanding, decision-making, execution, and verification. The overall architecture can be divided into four layers.

2.1 Perception Layer: Helping the Model See the Screen

• Vision-based approach: Screenshots are processed directly through multimodal understanding to identify buttons, input fields, list items, and their positions. The advantage is strong generalization without dependence on system interfaces. The downside is high image token consumption, and small controls or dense lists can be misidentified.
• Structured approach: The accessibility service or UI automation framework provided by the operating system reads the control tree to obtain text, control types, coordinates, and other structured information. The advantage is precision and low token overhead. The downside is that some custom-drawn interfaces expose no semantic information, and cross-app reading is subject to permission restrictions.
• Hybrid strategy: In engineering practice, structured information typically serves as the primary source, with visual understanding as a fallback. The system switches to screenshot mode when encountering custom-drawn controls or image-based verification pages, balancing accuracy and cost.

2.2 Decision Layer: Task Planning and State Management

• Task decomposition: A natural language goal such as “book me a flight for tomorrow morning” is broken down into executable subtasks: open the app, search for flights, apply filters, fill in passenger information, submit the order, and so on.
• State memory: The agent maintains context including completed steps, assumptions about the current page, and user preferences, preventing loops or repeated form filling.
• Reflection and retry: When execution results do not match expectations, such as a page not navigating, a popup blocking progress, or a network timeout, the model needs to recognize the anomaly and generate a recovery strategy. This is typically implemented through a think-act-observe loop: the model first outputs reasoning, then outputs an action, observes environmental feedback, and decides whether to continue or correct course.

2.3 Execution Layer: Action Space and System Permissions

Typical atomic actions include tap, long press, swipe, text input, back, and app launch. There are two main execution methods:

• System-level injection: Events are injected through the operating system’s accessibility interfaces. This is highly stable but requires explicit user authorization.
• Coordinate simulation: Touch events are simulated at recognized coordinates. This offers good compatibility but is susceptible to resolution differences and unusual popup layouts.

For security design, sensitive operations such as payments, message sending, and data deletion must have a whitelist and a second confirmation mechanism. When necessary, the agent should degrade to providing action recommendations for the user to execute manually, keeping the final decision with the user.

2.4 Closed-Loop Overview

A minimal agent main loop can be represented as follows:

while not task_done:
    screen = capture_screen()          # Capture screenshot or read control tree
    obs = perceive(screen)             # Extract structured UI elements
    thought = plan(obs, task_state)    # Plan the next action
    if thought.action == "finish":
        task_done = True
    else:
        execute(thought.action)        # Inject tap/swipe/input
        verify(thought.expectation)    # Verify the result matches expectations

All action recommendations should be described with a structured schema and include an expected result field for later verification and auditing:

{
  "action": "click",
  "target": {"type": "button", "text": "Submit Order"},
  "expectation": "Page navigates to payment confirmation"
}

3. Key Engineering Challenges

3.1 Latency and cost. Every step an agent takes requires a perception-inference-execution cycle. A single complete task may trigger dozens of model calls, and with multimodal input added on top, token consumption is far higher than a typical single-turn Q&A. This directly leads to the token surge discussion in Section 4.

3.2 Accuracy on long-tail interfaces. Core flows in mainstream apps are relatively easy to cover, but font scaling, dark mode, unusual popups, dynamic verification, and other long-tail scenarios can significantly reduce success rates. Teams need to build comprehensive evaluation sets and run regression tests after every model or app version update.

3.3 Security and privacy. Screen content often contains personal information. The scope of perception data collection, anonymization policies, and the boundary between on-device and cloud processing must all be clearly defined. Operation permissions should follow the principle of least privilege, and complete operation logs should be preserved for user review.

3.4 Evaluation framework. Single-step accuracy is not enough. The core metrics should be end-to-end task completion rate, average step count, average completion time, and sensitive operation interception rate. Only these can reflect actual user experience.

4. The Token Surge: Causes, Challenges, and Responses

4.1 Why Token Consumption Is Rising So Quickly

• Changing workload structure: Agent workloads, retrieval-augmented generation, and multi-turn conversations consume far more tokens per task than traditional Q&A.
• Widespread long-context usage: Stuffing entire documents, code contexts, and conversation histories into the context window is becoming the default approach.
• Application penetration: Usage is moving from experimentation into production, and consumption scales linearly with real business volume.

4.2 Challenges for Inference Infrastructure

• Balancing throughput and latency: Larger batches increase throughput but also push up time-to-first-token for interactive scenarios. Scheduling strategies must be differentiated by workload type.
• Cache hit rates: Agent and multi-turn workloads have high request prefix repetition. Prefix caching can significantly reduce redundant computation.
• Memory management: Longer contexts require more GPU memory. Low-bit quantization and paged memory management techniques are becoming standard in inference serving.
• Elastic scaling: Usage is highly volatile. Automatic scaling based on container platforms, monitoring, and alerting is necessary to avoid service degradation during traffic spikes.

4.3 Cost Engineering: What Developers Can Do

• Model tiering and routing: Route simple classification steps, such as checking whether a page has finished loading, to smaller models, and reserve larger models for complex planning.
• Context pruning: Inject only the UI elements and state relevant to the current step rather than the full information set.
• Result caching: Cache and reuse outputs for deterministic subtasks.
• Prompt compression: Remove redundant examples and repeated system prompts, and design prompt structures with prefix caching in mind.

5. Developer Checklist

1. Decouple the architecture: Keep perception, planning, execution, and verification as independently evolving layers, with interfaces constrained by a structured action schema.
2. Security guardrails: Sensitive action whitelist, user second confirmation, and end-to-end operation logs.
3. Cost baseline: Measure per-task token cost before launch, and set budgets and usage alerts.
4. Observability: Record every model call and every action execution for regression analysis and auditing. Major cloud platforms generally offer model serving, logging, monitoring, and alerting capabilities. Teams can reuse these foundational services rather than building the entire stack in-house. For example, Baidu AI Cloud provides such general-purpose capabilities.
5. Progressive rollout: Validate success rates in read-only, low-risk scenarios first, then gradually enable write operations.

6. Conclusion

LLMs directly controlling applications marks the evolution of agents from conversational partners to execution proxies. The sharp rise in token consumption signals that these high-consumption workloads have entered the scaling phase. The convergence of interaction paradigm and usage scale means that inference efficiency, cost engineering, and security governance will become the true differentiators in the next phase of agent application competition. For developers, establishing an engineering framework around perception, planning, execution, and verification early, while simultaneously building cost and observability baselines, is key to capturing this inflection point.