Full-Duplex Multimodal Interaction: Driving Facial Animation from a Unified Speech Token Stream

1. Problem Statement: Three Bottlenecks of the Half-Duplex Paradigm

Traditional voice assistants and customer-service bots mostly use a turn-based, half-duplex interaction pattern: the system speaks, waits for the user to finish, and then starts the next turn. In real-time multimodal scenarios, this paradigm exposes three structural bottlenecks.

First, turn detection relies on a fixed silence threshold. A common strategy such as “if silence exceeds 600–800 ms, the user has finished speaking” is forced into a tradeoff in real networks and with real speech rates. A short threshold causes false interruptions, while a long threshold introduces noticeable response delay. A more serious issue is that it may cut off a user who has already organized a longer sentence, disrupting the natural rhythm of expression.

Second, audio and visual generation are separated, causing uncontrollable synchronization drift. A typical cascaded pipeline looks like this: text → synthesized speech → phoneme/viseme alignment → lip-sync driving. Each stage adds queuing and decoding latency. Lip alignment can only be fitted after the fact, and paralinguistic cues such as emotion, stress, and pauses are already flattened during the text-based intermediate stage. The result is often “an enthusiastic voice with a wooden face.”

Third, latency accumulates. In a cascaded pipeline, each stage usually has to wait for the upstream stage to produce a complete output before it can begin. Time-to-first-byte becomes the sum of serial stages and can easily exceed 2 seconds, crossing the threshold where interaction feels unusable.

2. Target Form: Full-Duplex and Co-Generated Audio-Visual Output

A usable real-time multimodal interaction system should satisfy three engineering invariants:

1. A shared time axis: Audio samples and visual frames should be driven by the same representation and share the same monotonically increasing token or timestamp sequence, instead of being aligned separately by two independent pipelines.

2. Interruptible at any time: The system should be able to receive user speech at any moment, including while the model is generating output, and switch states immediately. Interruption response should not need to wait for the current generation unit to finish.

3. Degradable but not decoupled: Under weak network conditions or limited compute, the system may reduce resolution, frame rate, or audio quality, but audio-visual synchronization deviation must remain bounded.

The third point is often overlooked. A common industry principle is to treat audio-visual synchronization as a hard constraint and clarity or frame rate as a soft constraint: it is better to reduce resolution than to lose synchronization.

3. Unified Representation: Discrete Speech Tokens as the Cross-Modal Interface

The approach used in publicly disclosed comparable technical trials is that the acoustic model outputs not a waveform directly, but discrete speech tokens that encode both speech content and expressive style. Timbre, prosody, stress, pauses, and emotional tendencies are encoded into the token sequence. The audio decoder and the avatar decoder then consume the same token stream at the same time.

This brings two direct benefits:

• Paralinguistic information is not lost. The visual side consumes expressive tokens rather than plain text, so stress, changes in speech rate, hesitation, and light laughter can be mapped to facial expressions and gestures without requiring additional emotion labels.

• Alignment changes from “fitting” to “same-source.” Lip movement is no longer an after-the-fact alignment against audio; it shares the same generation source with audio. Synchronization error is compressed into the decoding and rendering stages.

Illustrative code:

# Single token stream, dual parallel consumers, shared time base
for tok in speech_token_stream(session):          # incremental output; no need to wait for full result
    if session.interrupted:
        session.drop_pending()
        break
    ts = session.clock.tick()
    audio_sink.push(acoustic_decoder(tok), ts)     # waveform / encoded audio frame
    visual_sink.push(avatar_decoder(tok), ts)      # lip sync, expression, body motion

The key point is that session.clock is the only clock. If the two paths maintain their own timestamps, accumulated drift will eventually appear after long-running sessions.

4. Latency Budget: Breaking Time-to-First-Byte into Optimizable Parts

The bottleneck in real-time interaction experience is not average latency, but the waiting period from the moment the user stops speaking to the arrival of the first synchronized audio and video bytes. A practical approach is to first establish a latency budget table, and then optimize each stage separately:

Stage	Main cost sources	Optimization methods
Endpoint / turn detection	VAD and semantic endpoint windows	Dual-threshold VAD + semantic endpoint model; early release of the first token
First token generation	Model TTFT and context length	Streaming KV cache reuse, context pruning, speculative decoding
Acoustic decoding	Per-frame vocoder computation	Causal/streaming vocoder, chunked decoding
Visual decoding	Per-frame diffusion or generation iterations	Few-step distillation, keyframes + inter-frame reuse
Packaging and transmission	Encoding, queuing, RTT, jitter buffering	Chunked transmission, low-latency encoding profile, edge access

For order-of-magnitude reference, publicly disclosed comparable technical trials reported approximately 870 ms to the first synchronized audio-video byte, with an output specification of 448×768 portrait video at 25 fps. These numbers show two things: first, compressing time-to-first-byte below one second is feasible in engineering; second, high resolution and high frame rate are not the primary metrics for real-time interaction. Spec restraint is a prerequisite for achieving low latency.

5. Distillation: Reducing Per-Frame Computation from Hundreds to a Few

Another notable data point from the above trials is that the visual model reduced the number of model computations required per video segment from approximately 120 to 2 through distillation. This is a core technique for low-latency visual generation. Its general principle and tradeoffs are as follows.

• Teacher model: Multi-step iterative sampling, such as diffusion-based models. Quality is high, but each frame may require dozens of forward passes, making real-time generation difficult.

• Student model: Few-step or single-step generation that learns the teacher’s one-step mapping through trajectory matching or output distribution alignment.

• Cost: Fine textures and high-frequency motions such as hands and hair may show jitter or blur. Over long durations, drift and identity inconsistency can occur.

Typical engineering compensation methods include:

• High-quality generation of keyframes, combined with motion interpolation or optical-flow reuse for intermediate frames. This idea is similar to inter-frame prediction in video encoding.

• Modeling low-frequency components such as pose separately, and generating textures at low resolution before upsampling.

• Limiting the duration of a single continuous generation and periodically resetting state to suppress drift.

6. Interruption and Turn-Taking: Model as a State Machine, Not if-else

The first requirement of full-duplex interaction is that “listening” and “speaking” run in parallel. Engineering practice recommends an explicit state machine, with interruption response latency monitored as a separate metric.

IDLE, USER_SPEAKING, ASSISTANT_SPEAKING, INTERRUPTING = range(4)
def on_user_audio(chunk):
    if state is ASSISTANT_SPEAKING and is_barge_in(chunk):
        state = INTERRUPTING
        clear_audio_jitter_buffer()      # discard unplayed audio
        cancel_generation()              # cancel in-flight generation tasks
        stop_avatar_track()              # avatar returns to listening state
    if state in (IDLE, USER_SPEAKING):
        state = USER_SPEAKING
        asr.push(chunk)
def on_assistant_token(tok):
    if state in (ASSISTANT_SPEAKING, INTERRUPTING):
        state = ASSISTANT_SPEAKING
        dispatch(tok)

Three details are easy to get wrong:

• Echo cancellation should take priority over VAD. When the assistant’s voice is played through speakers, the model’s own output can easily be detected as user speech, causing self-interruption. Echo cancellation, reference signal alignment, and reduced uplink sensitivity during generation must be in place together.

• Interruption requires a three-part stop: stop audio playback, cancel the generation task, and reset the avatar state. Stopping only one of them can result in “audio stops but the face keeps moving,” or “audio stops but context is lost for the next turn.”

• False interruption rate and missed interruption rate must be evaluated as a pair. Optimizing only one side simply moves the problem to the other side.

7. Streaming Transmission and Cloud Deployment Considerations

Transmission layer: Audio and video are recommended to be transmitted over separate channels and reassembled at the receiving end using a unified sequence number and timestamp. Audio should be prioritized. Under weak network conditions, video frames should be dropped before reducing audio quality. The depth of the jitter buffer should be adaptive: too large harms latency, while too small harms continuity.

Cloud deployment: Real-time inference services are usually deployed on container platforms and scaled elastically by session. Burst traffic can be smoothed with message queues to avoid queuing during session establishment. Model weights and assets can be stored in object storage and distributed from nearby locations. Raw session data, if retained, should be desensitized and minimized, and can be used for later quality review. Logging and monitoring should cover time-to-first-byte P50/P95/P99, interruption response latency, timeout rate, GPU utilization, and queue depth.

Canary release and rollback: Iteration of real-time models carries high risk. It is advisable to release by traffic percentage and keep hot-standby instances of the previous model version, so that rollback can be completed in seconds when anomalies occur.

8. Evaluation Metrics

Before launch, the following metrics should be fixed at a minimum:

• Time to first byte: measured from the moment the user stops speaking to the first audio/video byte, reported as P50/P95/P99.

• Interruption response latency, together with false interruption rate and missed interruption rate.

• Audio-visual synchronization deviation. Industry practice often considers lip-audio deviation at the tens-of-milliseconds level to be perceptible, so it should be treated as a hard gate.

• Lip-sync accuracy, which can be automated using phoneme-viseme consistency metrics for regression testing.

• Subjective quality: naturalness MOS, identity consistency, and emotion alignment.

• Unit session cost: compute and bandwidth cost normalized per minute of interaction.

9. Input-Side Expansion: Vision Perception and Compliance Boundaries

Full-duplex multimodal interaction is not limited to output. When a camera is connected, the system can combine scene understanding to adjust tone and expression, such as slowing down speech when a user appears to frown. However, compliance boundaries must be designed at the same time:

• Prefer on-device preprocessing.
• Upload only necessary features rather than raw video.
• Clearly define retention periods.
• Provide a one-click option to disable the visual channel.
• Encrypt biometric-related data and maintain access auditing.

This part is often the threshold for whether a product can enter enterprise scenarios, and it is not merely a model-capability issue.

10. Rollout Roadmap

1. Phase 1: Streaming text + streaming speech. First, reduce TTFT and time-to-first-byte.

2. Phase 2: Introduce discrete speech token representation and validate whether paralinguistic information can drive facial expression.

3. Phase 3: Add visual generation. Use distillation and keyframe reuse to reduce per-frame computation to a real-time range.

4. Phase 4: Complete the full-duplex state machine and echo cancellation, and run dedicated interruption testing.

5. Phase 5: Extend toward on-device and wearable platforms. Under compute and power constraints, implement model tiering: lightweight on-device models plus cloud-enhanced models.

Conclusion: The competitive focus of real-time multimodal interaction has shifted from “whether the model can generate” to “whether the representation can be unified, whether latency can be decomposed and optimized, and whether state can be rigorously modeled.” If speech tokens are used as a unified cross-modal interface, time-to-first-byte is treated as a latency budget to be optimized item by item, and interruption is implemented as a testable state machine, then stable interaction experiences can be delivered under real network conditions and realistic compute budgets.