Phone Use Agents and High-Fidelity Speech Synthesis: Engineering Practices

    This article examines two capabilities that are rapidly extending the boundaries of multimodal AI agents: operating phone interfaces and generating high-fidelity speech. It defines what "Phone Use" means, explains why long multi-step flows are harder than single-step tasks, and decomposes the problem into perception, decision, and execution layers, along with the engineering practices needed to move from a demo to production. It then walks through four key parameters of high-fidelity speech synthesis—44.1 kHz sampling, lower token rates, tokenizer-free continuous-space generation, and a real-time factor (RTF) of 0.17—and the tradeoffs behind each. Finally, it presents a reference architecture that combines voice interaction with screen operation, plus practical guidance on sandboxed execution, high-risk action confirmation, structured evaluation, and common pitfalls. Engineers building or deploying agent systems will find concrete architectural patterns and evaluation metrics they can apply directly.

    Lead: The Capability Boundary of Agents Is Extending Outward

    Over the past two years, the capability boundary of agents has largely stayed at the text and knowledge level: understanding questions, retrieving information, and generating code or documents, after which the user executes the results manually. A clear recent shift is that agents are beginning to directly “operate interfaces.” In a number of open-source projects, models can now “read” a phone screen and tap, type, and swipe like a human, completing long multi-step flows of dozens of actions such as ordering food delivery or booking flights.

    At the same time, speech is evolving quickly. Newly released speech synthesis models have raised the sample rate to 44.1 kHz, halved the token generation rate, and introduced an end-to-end generation architecture that does not rely on discrete speech tokens. Speech is no longer just “text that can be read aloud”—it is an interactive channel with near-human timbre and real-time streaming output.

    These two technical paths may seem independent, but they point to the same key capability of multimodal agents: letting agents “see, do, and speak.” This article breaks down the technical essentials of phone agents and high-fidelity speech synthesis, and provides reference architectures and engineering practices that can be applied directly.


    1. Phone Agents: From “Being Able to Talk” to “Being Able to Do”

    1.1 Defining the Capability: What Is “Phone Use”?

    Phone Use refers to an agent’s ability to take screenshots or interface control information as input, control a device by simulating touch events, and complete high-level tasks delivered by the user. It differs fundamentally from traditional automation scripts:

    • Interface-driven: it does not rely on fixed coordinates or fixed control IDs. Instead, it observes the screen in real time to understand which page it is on and what regions are clickable.
    • Goal-driven: the user only describes the goal (“order a non-spicy meal delivered to my door”), and the agent plans the intermediate path itself.
    • Long-horizon execution: a single task often spans dozens of interface states, and any one misstep can throw off all subsequent steps.

    An analogy: a traditional script is like a subway running on a fixed route—once the track changes, service stops. A phone agent is more like a driver who watches the road signs: it makes judgments, takes detours, and is also more likely to run into the unexpected.

    1.2 Typical Task Breakdown: Why Long Flows Are Harder Than Answering Hard Questions

    Take food delivery as an example. A complete run roughly involves: launching the app → confirming login status → searching for an item → reviewing options (taste, portion, add-ons) → adding to cart → entering checkout → verifying delivery address and time → confirming the order → completing payment → returning to the home screen to verify.

    Take flight booking as another example: choosing origin and destination cities → selecting dates → filtering flights and prices → entering passenger details → selecting seats and add-on services → verifying the total price → payment → checking order status.

    These tasks share three common difficulties:

    1. Many states and branches: option selection, date switching, and discount calculation each spawn different interface branches.
    2. Frequent interruptions: promotional pop-ups, ad overlays, expired sessions, and network errors can break the flow at any time.
    3. Errors accumulate: even at a 95% per-step success rate, the probability of 30 consecutive correct steps is only 0.95^30 ≈ 21% (assuming the steps are independent). This shows that the bottleneck of long-horizon tasks is not per-step “cleverness” but overall “stability.”

    1.3 Three-Layer Capability Breakdown: Perception, Decision, Execution

    (1) The perception layer: reading the screen

    Perception determines what the agent can “see.” There are three common technical approaches:

    • Structured parsing: use the system’s accessibility services or control tree to directly obtain element types, text, and clickable regions. This is accurate but depends on the specific system’s APIs and permissions.
    • Pure visual understanding: apply visual understanding directly to screenshots to output control positions and semantics. This does not depend on system APIs and generalizes well across apps, but requires an additional localization step.
    • Dual-channel fusion: structured and visual information cross-validate each other—for example, using control text to calibrate the coordinates produced by a visual model. In engineering, this is usually the most robust.

    Whichever approach is used, perception output should be as structured as possible rather than free text. The more standardized fields such as the element list, current page type, interactive regions, and overlay and keyboard state are, the more reliably the decision layer can consume them.

    (2) The decision layer: knowing what to do

    The decision layer must handle three kinds of logic simultaneously:

    • Task decomposition: breaking a high-level intent into verifiable sub-goals. For example, “select options” can be split into “confirm portion,” “choose taste,” and “confirm add-ons.”
    • Action selection: choosing the target from the candidate elements on the current interface and distinguishing between similar ones. Closely related buttons such as “book now” versus “set a booking reminder,” or “confirm payment” versus “confirm order details,” are the most common sources of mis-taps.
    • Exception handling: when encountering pop-ups, loading failures, or expired sessions, first dismiss the interruption, then decide whether to retry, and if necessary go back a step and re-plan—rather than clicking straight down an error path.

    (3) The execution layer: turning actions into device events

    The execution layer converts decision results into device events: tapping, long-pressing, swiping, text input, going back, and so on. Note that:

    • Coordinates must be normalized against the actual screen resolution.
    • App launches and page transitions require waiting for loading to complete; do not tap blindly in quick succession.
    • After each action executes, re-perceive the interface and use the result to verify “whether the action took effect” before deciding the next step.

    1.4 From “Demo-Ready” to “Production-Ready”: Key Engineering Practices

    The “flawless” runs in demo videos often differ from stable operation in real environments by a large number of engineering details. The following points are especially important.

    (1) Task sandboxing and controlled execution

    Agents should run in an isolated, controlled device environment—such as a dedicated test device, a cloud real device, or an emulator—to avoid interfering with a user’s day-to-day use. Set a maximum step count and a timeout for each task to prevent infinite loops.

    (2) A confirmation mechanism for high-risk actions

    Irreversible or financially sensitive operations—payment, order submission, modifying personal information, and login authorization—should be placed in a “staged, pending confirmation” state and released only after the user reviews them. The safer pattern is “the agent operates on your behalf, with manual confirmation at key checkpoints,” rather than fully automatic execution.

    (3) Screenshot archiving and observability

    Assign a unique identifier to each task and save every step’s screenshot, action parameters, execution result, and elapsed time along the way. When a problem arises, you can quickly pinpoint which step and which type of error occurred, and you can accumulate high-quality training and evaluation data. If screenshots contain personal information, they must be desensitized and given a retention period.

    (4) Structured evaluation

    Borrowing the field-level evaluation approach from information extraction, break the task result into verifiable fields—such as product name, options, total price, delivery address, flight number, and departure time—and compare each field for correctness. Looking only at “whether the order ultimately succeeded” is not enough: an agent may select the right product but fill in the wrong address, and this kind of error must be catchable by evaluation. It is recommended to track at least the following metrics:

    • Task success rate;
    • Average number of steps to completion;
    • Mis-operation rate (the proportion of taps on the wrong element);
    • Manual intervention rate;
    • Average completion time.

    (5) Cloud-based task scheduling

    Phone agents often need to run in batches in the cloud. Common general practices include:

    • Using a container platform to host device environments and agent processes, scaling elastically on demand.
    • Using a message queue to handle asynchronous tasks, splitting long flows into multiple “perception–decision–execution” stages for distributed processing.
    • Using object storage to archive screenshots and execution logs for playback analysis.
    • Using logging services and monitoring/alerting to track success rates, timeout rates, and the distribution of anomalies.

    2. High-Fidelity Speech Synthesis: Reading Four Key Parameters

    If a phone agent is the agent’s “hands and eyes,” speech synthesis is its “mouth.” A natural, lifelike voice often builds trust more effectively than a text reply, but users also hold speech to a higher standard: it must sound human and be fast enough. Taking a recently released open-source speech model as an example, its technical design centers on four key points.

    2.1 Raising the Sample Rate to 44.1 kHz: High-Frequency Detail Decides “How Human It Sounds”

    The sample rate is the number of times the sound waveform is sampled per second, and half of it (the Nyquist frequency) determines the highest representable frequency. Telephone speech commonly uses an 8 kHz sample rate, which can only represent sound below 4 kHz; a 16 kHz sample rate covers up to 8 kHz, which is enough for clear intelligibility. Raising it to 44.1 kHz pushes the representable frequency ceiling to about 22 kHz.

    The human hearing range is roughly 20 Hz–20 kHz. Sibilants and fricatives in consonants, along with a speaker’s timbral characteristics, are largely distributed in the mid-to-high frequency range—exactly the information that determines whether it “sounds like this person” and whether it sounds natural. For zero-shot voice cloning, differences between the reference audio and the target timbre also show up heavily in high-frequency detail. So in high-fidelity cloning, raising the sample rate usually yields a more natural listening experience and a timbre closer to the target.

    However, raising the sample rate is not “free”:

    • For the same audio duration, the number of waveform sample points increases by about 2.76x (44.1 ÷ 16 ≈ 2.76).
    • If you model the raw waveform point by point, the sequence length and computation rise accordingly.

    For this reason, high-fidelity speech systems typically use hierarchical modeling: a low-time-resolution acoustic representation carries the main information first, and a decoder then reconstructs the high-sample-rate waveform.

    2.2 Lowering the Token Rate: Same Quality at Half the Computation

    In autoregressive speech modeling, the token rate is the number of acoustic units (tokens) the model generates per second. At a token rate of 12.5 Hz, each token corresponds to 80 ms of audio; dropping to 6.25 Hz makes each token correspond to 160 ms. To generate the same audio duration, the number of autoregressive steps the model needs is directly halved.

    The impact on inference cost is very direct:

    • Fewer computational steps roughly halve the generation time for a single utterance;
    • In long-speech or batch-synthesis scenarios, throughput and concurrency improve significantly;
    • The risk of autoregressive error accumulating over longer sequences is also reduced.

    The cost is a coarser time resolution. With each token covering 160 ms, the model must rely more on context to judge prosody and coarticulation, and short events such as plosives and pauses may be harder to align precisely. This is usually compensated by a high-quality decoder and by modeling long-range structure during training. In other words, “halving the token rate” is fundamentally a tradeoff between precision and efficiency, and diffusion-based decoding is precisely what makes this tradeoff worthwhile.

    2.3 Tokenizer-Free: Generating Speech in a Continuous Space

    Traditional speech generation mostly follows a “discretization” route: a speech encoder first slices audio into frame-by-frame discrete tokens, and a language model then generates them token by token. The advantage of this route is that it can directly reuse the modeling paradigm of language models; the drawback is that quantization inevitably loses information, and codebook capacity and bitrate directly cap the achievable audio quality and timbre.

    Continuous-space generation instead treats acoustic representations as continuous vectors (such as the latent variables of a variational autoencoder, VAE), performs autoregressive generation in that continuous space, and finally uses a diffusion-based decoder to reconstruct the waveform. Its core advantages are three:

    1. Less information loss: no quantization rounding is needed, so detail is preserved more fully.
    2. Decoupled time resolution: the time step of the acoustic encoding can be designed separately from the generation step, which is the prerequisite for achieving a low token rate.
    3. Quality and efficiency decoupled: text-to-acoustic-representation generation happens at a low time resolution, while audio-quality reconstruction is handed off to the decoder.

    It is worth noting that this idea is not first introduced in this version—continuous representations plus diffusion decoding have a long track record in image and audio generation. The value of this update lies in systematically optimizing the sample rate, token rate, and architecture together.

    2.4 A Real-Time Factor (RTF) of 0.17: The Performance Baseline for Streaming Synthesis

    The Real-Time Factor (RTF) is calculated as:

    1. RTF = time to generate audio / duration of generated audio

    An RTF below 1 is a basic requirement for real-time interaction. Take 0.17 as an example: generating 10 seconds of speech takes about 1.7 seconds of compute. Supporting streaming synthesis on a consumer-grade GPU means the model can generate and play at the same time, rather than waiting for the whole sentence to finish.

    But note that RTF is only a throughput metric; first-packet latency (the time from text to the first audible segment) also determines the user experience. In real deployments, the number of concurrent requests, input length, batching strategy, and hardware all affect final performance. Production environments should measure and set target values separately for single-request RTF and first-packet latency, rather than fixating on a single average.


    3. Combined View: A Reference Architecture for Speech + Screen Operation

    Putting these two capabilities together forms a closed loop of “voice issues the task, the agent performs the operation, and voice reports the result”:

    1. User voice command
    2. │
    3. ▼
    4. ┌─────────────┐ intent ┌────────────────┐
    5. │ Voice │ understand │ Task │
    6. │ interaction│──────────────▶│ orchestration │
    7. │ layer │ │ planning / │
    8. │ (ASR / TTS) │◀──────────────│ verification │
    9. └─────────────┘ status └───────┬────────┘
    10. ▲ broadcast │ action
    11. │ ▼
    12. │ execution ┌────────────────┐
    13. │ result │ Device │
    14. │ / feedback │ execution │
    15. └───────────────────────│ layer │
    16. │ perceive / do │
    17. └───────┬────────┘
    18. │ screenshots / logs
    19. ▼
    20. ┌────────────────┐
    21. │ Support │
    22. │ services │
    23. │ schedule / │
    24. │ storage / │
    25. │ monitoring │
    26. └────────────────┘
    • Voice interaction layer: performs speech recognition and synthesis, and handles wake, interruption, and announcements.
    • Task orchestration layer: converts voice intent into a structured task, decomposes sub-goals, and verifies the result of each step.
    • Device execution layer: handles screen perception and action execution, and abstracts away differences between devices and systems.
    • Support services layer: schedules tasks through a message queue, archives trajectories through object storage, and tracks operational status through logging and monitoring.

    Below is illustrative pseudocode for the execution loop. The focus is on showing the structure of “perceive–decide–confirm high-risk actions–execute–verify”; all interface names are illustrative:

    1. MAX_STEPS = 40 # Step budget to prevent infinite loops
    2. def run_phone_task(task: str, session):
    3. state = session.observe() # Get the current interface state
    4. session.speak("Sure, let me handle that for you")
    5. for step in range(MAX_STEPS):
    6. # 1. Check whether the task is done or has drifted off target
    7. status = verify(task, state)
    8. if status.done:
    9. session.speak("All done")
    10. return True
    11. if status.derailed:
    12. state = recover(session, state) # Return to a known safe state
    13. continue
    14. # 2. Decide: choose the next action
    15. action = decide(task, state)
    16. # 3. High-risk actions: confirm first, then execute
    17. if action.risk_level == "high":
    18. if not session.confirm(action.summary):
    19. session.speak("Cancelled, nothing was executed")
    20. return False
    21. # 4. Execute and verify the result
    22. result = session.execute(action)
    23. if not result.ok:
    24. session.speak("That step didn't work, let me try another way")
    25. state = session.observe() # Re-observe to close the loop
    26. session.speak("Step limit reached, please check manually")
    27. return False

    This code embodies several key design principles:

    • Observation-driven decisions: every step re-acquires the interface state rather than relying on assumptions from history.
    • Budget constraints: a step limit guarantees the task will terminate.
    • Risk tiers: high-risk actions must be confirmed by the user.
    • Visible state: every step has a clear voice or on-screen feedback.

    4. Common Pitfalls and Practical Recommendations

    1. Treating “demo success” as “product readiness”: demos are usually curated and tuned, while pop-ups, network fluctuations, and version updates in real environments all affect stability. Before launch, run large-scale evaluation across diverse real-world scenarios.
    2. Looking only at the final answer, not the intermediate process: structured field-level comparison catches “looks successful but is actually wrong” problems and is an indispensable part of the evaluation system.
    3. Ignoring privacy and compliance: screenshots may contain sensitive information such as chat records, addresses, and orders. Stick to the principle of least privilege, enforce desensitized storage and time-limited cleanup, and always keep a manual confirmation step for operations involving money and personal information.
    4. Chasing audio quality while ignoring latency: no matter how lifelike the timbre, first-packet latency beyond one or two seconds significantly degrades the interaction experience. Constrain both RTF and first-packet latency during model selection.
    5. Piling on parameters instead of matching the scenario: 44.1 kHz sampling and a low token rate are not needed in every scenario. Voice-assistant announcements, long-text narration, and voice cloning each have different requirements for audio quality and latency, so choose the appropriate configuration per scenario.

    Conclusion

    Phone agents solve the problem of “doing things on the user’s behalf,” while high-fidelity speech synthesis solves the problem of “making users willing to use it.” The former tests long-horizon planning, interface understanding, and fault-tolerant recovery; the latter tests the balance between information fidelity and inference efficiency.

    From an engineering standpoint, parameter metrics are a starting point, not the finish line. Only when metrics like sample rate, token rate, RTF, and task success rate are put back into real scenarios and weighed together with security, privacy, and cost can agents truly move from “impressive in a demo” to “usable day to day.”