Full-Duplex Real-Time Voice Interaction: Architecture Design and Engineering Practices

1. From Turn-Based Switching to Full-Duplex

Traditional voice dialogue systems mostly use a half-duplex pipeline: first detect that the user has stopped speaking, then perform speech recognition, pass the result to a language model to generate a reply, and finally play the response through speech synthesis. Each stage runs serially, and latency accumulates at every step. If the user wants to interject while the system is speaking, they often have to wait for playback to finish or press a button, which makes the interaction feel rigid.

The core idea of full-duplex is that the microphone and speaker work at the same time. While outputting speech, the system continuously receives audio and decides at high frequency whether it should keep speaking, stop speaking, keep listening, interrupt itself, or trigger external tools. This mode is closer to human conversation: people listen while preparing a response, and they can interject at any time when the other party says something wrong or goes off topic.

Implementing full-duplex requires solving three types of problems. The first is perception: the system must accurately determine when the user starts and stops speaking in noisy environments. The second is decision-making: the system must choose the correct action within milliseconds. The third is signal processing: sound played by the speaker is captured by the microphone, and without handling, the system may mistake its own voice for user input.

2. Breaking Down Core Capabilities

2.1 Continuous Listening and Incremental Understanding

A full-duplex system cannot wait for an entire utterance to end before it starts processing. Audio enters the system in fixed frame lengths, commonly 10 to 30 milliseconds. The recognition module needs to output incremental results, including partial transcripts, confidence scores, and timestamps. Downstream language modules can use these results to prepare responses in advance and reduce the waiting time perceived by the user.

Incremental understanding has a cost: intermediate results may be corrected by subsequent audio. The design should distinguish between “temporary results” and “stable results.” Only stable segments should enter irreversible stages, such as speech that has already started playing or tool requests that have already been sent.

2.2 Interaction Decisions: Speak, Listen, Stop, Interrupt, and Invoke Tools

The decision layer is the brain of a full-duplex system. A common action set includes:

• Continue listening: the user is still speaking, or the system determines that the available information is insufficient.
• Start responding: the user’s utterance has ended and is semantically complete.
• Pause output: the user has started speaking, and the system needs to yield the floor.
• Interrupt output: the user’s interjection contains substantive content, such as a correction or a new question.
• Invoke tools: the answer requires querying data, performing calculations, or operating an external system.

Decisions can be implemented with an explicit state machine or delegated to a small model. A state machine is predictable and easy to test, but coverage of edge cases depends on manually defined rules. A combination of both often works well: use rules as a fallback for critical safety paths, and use models for semantic judgments, such as deciding whether an “mm” is a backchannel or an interruption.

2.3 Echo Cancellation and Self-Speech Filtering

Content played through the speaker is captured again by the microphone through the air or hardware paths. Without handling, the system will repeatedly trigger “user speech detected” while it is speaking, causing oscillation.

In engineering practice, acoustic echo cancellation (AEC) algorithms are commonly used. They take the playback signal as a reference, estimate the echo component, and subtract it. On this basis, speaker differentiation or voiceprint features can also be combined to reduce interference from ambient human voices and the system’s own voice. For devices without hardware echo cancellation capability, a software solution should be planned early in the design phase, and testing should cover different volume levels and room conditions.

2.4 Barge-In Handling

Barge-in is the most error-prone part of the full-duplex experience. There are two common types of misjudgment: first, background noise is mistaken for an interruption, causing the system to pause frequently; second, the user’s backchannel words, such as “right” or “okay,” are treated as new instructions, causing the answer to be cut off.

A layered strategy can be used:

1. Physical-layer judgment: only when energy exceeds a threshold and lasts longer than a minimum duration is it considered a possible interjection.
2. Semantic-layer judgment: combine the current transcript to determine whether it constitutes valid input.
3. Fallback strategy: if it is judged as a false interruption, the system resumes playback and compensates for the lost segment in the buffer.

3. Reference Architecture

A deployable full-duplex voice system usually contains the following modules:

Microphone --> Audio preprocessing (AEC, noise suppression) --> Voice activity detection
                                              |
                                              v
Speaker <-- Audio post-processing <-- Speech synthesis <-- Generation and decision <-- Incremental recognition
                                    |
                                    v
                              Tool execution layer --> External services
                                    |
                                    v
                       Session state storage / logs / monitoring

The responsibilities of each module are as follows:

• Audio capture and transport layer: handles encoding and decoding, jitter buffering, and network transmission. Real-time scenarios should prioritize low-latency codecs and control buffer depth.
• Perception layer: includes voice activity detection, incremental recognition, and endpoint detection.
• Decision layer: maintains session state and outputs action instructions.
• Generation layer: generates reply text or structured actions based on context.
• Synthesis and playback layer: converts text into speech, supports streaming output, and can be stopped immediately.
• Tool execution layer: calls external capabilities asynchronously and writes results back into the session context.
• State and observability: stores session state, records key events, and supports replay and troubleshooting.

4. Implementation Notes for the Decision Loop

The decision loop is critical to system latency and stability. The following illustrative code shows a basic structure. Interfaces and class names are generic examples and should be replaced according to the actual framework:

# Illustrative code: decision loop for a full-duplex session
class DuplexSession:
    def __init__(self, mic, speaker, vad, echo_canceller, policy, executor):
        self.mic = mic
        self.speaker = speaker
        self.vad = vad
        self.echo_canceller = echo_canceller
        self.policy = policy
        self.executor = executor
        self.state = "LISTENING"
        self.active = True
    async def run(self):
        while self.active:
            frame = await self.mic.read_frame()          # e.g., a 20 ms frame
            ref = self.speaker.reference_frame()         # reference signal currently being played
            clean = self.echo_canceller.process(frame, ref)
            event = self.vad.detect(clean)               # speech start, continuation, end
            action = self.policy.decide(self.state, event, clean)
            self.state = await self.executor.apply(action, self.state)

Implementation should pay attention to the following points:

• Avoid blocking calls inside the loop. Time-consuming operations, such as network requests and model inference, should be placed in independent tasks, with results returned through queues.
• State transitions must be traceable. Every action should record the input event, decision result, and timestamp to make problem reproduction easier.
• The playback buffer must be abortable. After receiving an interruption instruction, the playback layer should clear unplayed data within milliseconds.

5. Concurrency and Safety in Tool Calls

In a full-duplex system, tool calls may occur at the same time as speech output. For example, the system may say “Let me check that for you” while requesting a data API. This introduces three design requirements.

First, filler phrases and real results must be managed separately. The system can output a short prompt first to reduce the silence of waiting, but the prompt content should not contain unverified conclusions.

Second, when a tool result arrives, the system needs to determine whether the current session is still valid. If the user has already moved to a new topic, the old result should be discarded or downgraded.

Third, tools involving write operations, funds, or permissions must be confirmed first. The confirmation step should be designed as an explicit voice question-and-answer and require a clear affirmative response, avoiding treating background noise or ambiguous answers as authorization.

Tool permissions should follow the principle of least privilege. Voice entry points are usually easier to trigger accidentally than graphical interfaces, so high-risk actions should require additional verification.

6. Latency Budget and Key Metrics

The quality of the full-duplex experience depends heavily on latency. The end-to-end chain can be divided into several segments, with a budget set for each. The table below shows a common division; specific values should be adjusted according to business scenarios and network conditions:

Stage	Metric	Suggested target
Audio capture and transport	One-way network latency	Keep within 100 ms where possible
Voice activity detection	Endpoint decision latency	Match user pause habits; measure empirically
Incremental recognition	Time to first stable segment	As short as possible; coordinate with generation
Generation and decision	Time to first action decision	Determines whether the response is timely
Speech synthesis	Time to first audio packet	Affects when the user perceives speaking has started
Playback	Time to stop playback after interruption	Should be near-instant

In addition to latency, the following quality metrics should also be monitored:

• False interruption rate: the proportion of cases where the system is interrupted when the user did not truly interject.
• Missed interruption rate: the proportion of cases where the user clearly interjects but the system does not respond.
• Echo residual rate: the proportion of playback audio mistakenly recognized as user input.
• Conversation completion rate: the proportion of users who complete the task without needing to repeat themselves.

7. Testing and Evaluation

Full-duplex systems are harder to test than ordinary voice features because the input is continuous and mutually influential. It is recommended to build the following scenario set:

• Normal turn: the user finishes a sentence, and the system responds.
• Hesitation pause: the user pauses mid-sentence, and the system should not respond too early.
• Backchannel words: the user says “mm” or “right,” and the system should continue playback.
• Genuine interruption: the user corrects midway, and the system should stop and respond promptly.
• Noisy environment: TV, traffic noise, and multiple people talking.
• Echo conditions: different volume levels, devices, and room layouts.
• Tool asynchrony: slow tool responses, failures, and timeouts.

Testing should use real recordings for playback wherever possible, combined with automated scripts to calculate the metrics above. Relying only on manual listening evaluation makes it easy to miss intermittent issues.

8. Deployment and Operations Recommendations

The deployment approach depends on latency requirements and privacy constraints. Edge nodes close to users can reduce network latency and are suitable for scenarios sensitive to first-packet time. Centralized deployment simplifies unified model and data management and is suitable for complex reasoning tasks. Hybrid deployment is also common; for example, audio preprocessing and voice activity detection can run on the device, while generation and tool calls run in the cloud.

At the operations level, the following practices can be considered:

• Use encrypted channels at the transport layer, and encrypt audio data both in transit and at rest.
• Define a clear audio retention policy, save only necessary logs, and desensitize sensitive content.
• Use message queues to decouple perception, decision-making, and tool execution modules to avoid single-point blocking.
• Implement elastic scaling through container platforms, but note the impact of cold starts on first-packet latency.
• Use logging services and monitoring alerts to track latency percentiles, interruption success rates, and error rates, and set threshold alerts.

9. Common Issues in Practice

• Large differences in user pause duration: endpoint decision thresholds can be adjusted dynamically based on user habits, but upper and lower bounds should be set to prevent over-adjustment.
• Multiple people speaking at the same time: speaker differentiation is needed; otherwise, the system may easily respond to irrelevant conversations.
• Context expansion in long conversations: historical content should be compressed periodically, retaining key information relevant to the current task.
• Mismatch between model output and speech rhythm: sentence segmentation strategies suitable for reading aloud can be added during text generation to reduce abnormal pauses during synthesis.

The value of full-duplex interaction lies in making conversations more natural, but it is highly sensitive to engineering details. Teams should first define clear latency and quality targets, then choose specific approaches for perception, decision-making, and execution, and iterate gradually through playback tests based on real scenarios.