Next-Generation Video Coding for Real-Time Interaction: Selection and Testing
This article examines VP9 and AV1 as open, royalty-free video coding options for real-time interactive systems, with HEVC and VVC considered as licensing and hardware comparisons. It explains why real-time evaluation must focus on tail latency and recovery metrics, then covers layered encoder complexity, low-latency reference structures, content-aware bitrate allocation, and audio-video synchronization for speech and multimodal interaction.
1. Defining the Scope of Video Coding Standards
The open video coding standards discussed here are mainly VP9 and AV1. VP9 was introduced by Google, and AV1 was developed by the Alliance for Open Media (AOM). Both use royalty-free licensing. Mainstream browsers generally support VP9 decoding, and AV1 decoding is available in Chrome, Firefox, and Edge; Safari support depends on the device and OS version. For specific support, refer to the current documentation for each platform.
HEVC (H.265) and VVC (H.266) are developed by ITU-T and ISO/IEC and require patent licensing fees. In many scenarios they also offer strong compression efficiency, but selection should evaluate both licensing cost and endpoint hardware support. This article treats them as comparison objects, not default options.
2. How Real-Time Interaction Changes Evaluation Criteria
Video-on-demand services can use preloading and offline transcoding to obtain a stable playback experience. Real-time interactive services have very little tolerance for error. A complete interaction chain usually includes: user input capture, business logic processing, scene or image generation, video encoding, network transmission, receiver decoding, and rendering/display. Queuing or burst jitter in any part of the chain increases end-to-end latency.
Therefore, the objectives of coding technology can be broken down as follows:
- Reduce transmitted data at the same quality;
- Retain more detail at the same bitrate;
- Shorten the encoding and decoding paths;
- Reduce the time to restore the image after key operations;
- Maintain interaction continuity under bandwidth fluctuation;
- Adapt to mobile endpoints, edge devices, and dedicated hardware.
Average latency cannot represent the real experience. An occasional long stall often affects user perception more than a slightly higher but stable latency, so tail latency must also be measured.
The metric definitions and measurement methods are as follows:
- First-frame time: the time from the start of a session request to the completion of rendering of the first frame on the receiver. The sender records the session start time, and the receiver records the timestamp in the render callback; subtract the two.
- End-to-end latency: the time from the capture-frame timestamp on the sender to the completion of rendering on the receiver. The two ends need clock synchronization, either through NTP or by mapping RTP timestamps with RTCP reports. Report P95 and P99 over a 1-minute window.
- Keyframe recovery time: the time from when the receiver detects packet loss or decoding failure to when the first decodable and rendered frame after recovery is available.
- Stall rate: 300 ms without a new frame rendered counts as one stall. Count stalls per minute and record the total stall duration as a proportion of session duration.
3. Key Directions for Next-Generation Coding
1. Layered Complexity Instead of One Parameter Set
More advanced encoders use more flexible block partitioning, finer motion search, and more complex loop filtering. These tools improve compression efficiency, but they increase server computation, mobile device heat and power consumption, and encoding queue time.
Therefore, encoder parameters should be layered by scenario. Offline content can use two-pass encoding, lookahead, and higher encoding effort. Real-time content should use real-time mode—for example, libaom’s realtime usage mode—with a higher cpu-used value, lookahead and two-pass encoding disabled, and the number of reference frames limited. Rate control typically uses CBR or constrained VBR to adapt stably to the network.
Resolution, frame rate, and bitrate caps should be tiered by endpoint capability. For example, high-performance devices use 1080p 30fps, medium devices use 720p 30fps, and low-end mobile devices use 540p or 360p. Hardware encoders often have limits on maximum resolution, reference frame count, and some encoding tools, and differences between vendors are significant. At startup, probe hardware capabilities first; if the hardware does not support the current configuration, fall back to software encoding or reduce resolution.
Downgrade triggers can be quantified. At 30fps, the per-frame budget is about 33 ms. If encoding time exceeds 70% of the per-frame budget for 3 seconds, lower the encoding speed by one step; if encoding time is below 40% for 10 seconds, restore one step. This avoids frequent switching.
2. Trade-Offs in Low-Latency Coding Structures
Long-distance inter-frame prediction improves compression efficiency, but it increases decoding dependency, expands error propagation, and slows random access. Real-time services usually need to control reference chain depth and keep the keyframe interval within an acceptable range.
Shortening the keyframe interval speeds recovery, but the cost is clear. Keyframes are usually much larger than P frames, so a shorter interval increases average bitrate. A longer interval means a longer wait for the next keyframe after packet loss. A compromise is to keep a medium interval and, when packet loss or decoding failure is detected, have the receiver request an immediate keyframe through PLI or FIR rather than always shortening the interval.
For scenarios with dramatic image changes, such as cloud gaming and remote desktop, the encoder should allocate more bitrate to the areas users focus on. For meeting images, it can reduce background update frequency and reserve bitrate for faces, mouth shapes, and gestures.
3. Content-Aware Bitrate Allocation
Static backgrounds, fast motion, text interfaces, and natural scenes have very different bitrate requirements. A fixed bitrate strategy can hardly balance these contents.
A more reasonable process is to first analyze image motion intensity, identify text, people, and high-contrast edges, determine whether the current frame has a scene change, and then adjust the target bitrate based on user focus areas and network status.
Buttons, subtitles, code, and small text in interactive interfaces are often more important than ordinary texture. Even if overall image quality is good, blurred or jittery text edges can make users feel that system quality is poor. Therefore, the strategy should prioritize clarity in important areas rather than only pursuing average quality across the whole image.
4. Audio-Video Coordination and Synchronization Mechanisms
Speech synthesis and multimodal interaction push real-time systems from single video transmission toward audio-video coordination. Speech requires not only natural sound quality but also rhythm, pauses, character consistency, and response speed that matches the visuals.
For synchronization, audio should be used as the master clock. The audio sampling clock is stable, and the human ear is more sensitive to audio interruptions than to slight video jumps. Video frames are mapped to the same timeline according to capture timestamps, and the playback end decides when to render frames based on the audio clock.
The perception threshold for lip sync can reference ITU-R BT.1359. When audio leads video by about 45 ms or lags video by about 125 ms, viewers can easily notice it. In engineering, the target can be set to an absolute deviation of no more than 80 ms, with an alert triggered when it exceeds 125 ms.
First-packet and first-frame alignment needs separate handling. If audio arrives first and the first video frame arrives more than 150 ms late, the playback end can play audio first while the character remains on a standby frame, so motion does not jump abruptly. If video arrives first, wait for the first audio packet before starting playback to avoid lip movement ahead of audio.
Multi-character dialogue must maintain each character’s state. Each character should be bound to a voice ID, an avatar ID, and current status.
