From Award to Next Generation: Video Coding Moves Toward Real-Time Interaction
An open-source, royalty-free video codec has earned recognition in the film and broadcast engineering community, while the standards alliance behind it has outlined a next-generation codec that puts real-time interaction at the center of its design. This article traces how video coding has evolved from chasing image quality toward balancing latency, explains why real-time interaction has become a core design goal, and examines the key challenges and technical directions—latency budgeting, screen content coding, content-adaptive encoding, perceptual rate-distortion optimization, packet-loss resilience, hardware/software and device-cloud collaboration, and AI integration. It then offers practical guidance for building low-latency real-time video pipelines, covering latency budget planning, adaptive bitrate control, differentiated configurations for screen content and gaming, quality evaluation, and observability with canary rollouts.
1. Background: An Award Is Recognition—and a New Starting Point
Among the most influential annual awards in film and video engineering, recognition went to a general-purpose video coding format that was jointly initiated by industry and released under a royalty-free license. This marks another milestone in how open-source communities and industry collaboration are driving the adoption of video technology. At the same time, the open standards alliance behind the format disclosed the direction of its next-generation coding standard and stated clearly that real-time interaction will be a major design goal of the new standard.
There are three reasons this award is worth attention:
- Royalty-free licensing lowers the barrier to adoption: Small and mid-sized developers and emerging businesses can access advanced coding capabilities at lower cost, extending use cases from film production to internet applications.
- Open-source collaboration has changed the pace of standard evolution: Tool proposals, reference implementations, and real-world production feedback are tightly linked, so the standard iterates noticeably faster than through traditional processes.
- Evaluation criteria have become more diverse: The award weighs not only compression metrics measured in the lab, but also compatibility, engineering difficulty, and ecosystem activity in real deployments.
The award recognizes past work; the next-generation coding standard is the forward-looking move. The alliance lead also emphasized that the new standard is still in an early stage and will focus on emerging forms such as screen content and multi-view video, as well as coding performance under real-world conditions.
The phrase “designed for real-time interaction” may sound simple, but it could reshape the technical roadmap of video coding. The analysis below is based on publicly disclosed directions and general technical principles; specific technical proposals are subject to the final information released by the standards organization.
2. Evolution: From “Chasing Image Quality” to “Balancing Latency”
Looking back over decades of video coding development, we can broadly divide it into three generations:
| Generation | Representative form | Core goal | Main tradeoff |
|---|---|---|---|
| First | Early international video coding standards | Deliver SD TV over limited bandwidth and store on optical discs | Limited compression efficiency |
| Second | High Efficiency Video Coding (HEVC) and its successors | Popularization of HD/UHD streaming and mobile video | High coding complexity, costly patent licensing |
| Third | Open-source, royalty-free general-purpose coding formats | Balance high compression efficiency with low-cost adoption | Even higher coding complexity |
Each generation rebalances the triangle of compression efficiency, computational complexity, and licensing cost. The emergence of open-source, royalty-free formats is essentially a breakthrough on the licensing-cost dimension; these formats now broadly support UHD video-on-demand and media playback in mainstream browsers.
One notable observation is that one of the most important innovations of the streaming era came from the transport side: adaptive bitrate (ABR) streaming, in which the player switches among multiple bitrate renditions based on real-time network conditions, greatly improving the viewing experience on weak networks. This shows that video experience has never been determined by the codec alone. Listing real-time interaction as a priority for the next-generation standard likewise requires codecs, transport, and scheduling to evolve together—only then can gains at the coding layer be realized as an experience users can actually perceive.
3. Why Has Real-Time Interaction Become a Core Problem?
Unlike live streaming and video-on-demand, real-time interactive applications require a closed loop of “user action → system response → result presentation,” and it must be completed within an acceptable time window.
- Typical scenarios: cloud gaming, cloud desktops, remote design collaboration, interactive classrooms, online medical consultation, video conferencing, and conversational human-machine applications.
- Core difference: on-demand video focuses on startup time, stall rate, and clarity, and can tolerate latency of a few seconds. In real-time interaction, even a few hundred milliseconds of extra delay can destroy the feel of an operation or interrupt the rhythm of a conversation.
- Business relevance: coding efficiency gains in real-time interactive scenarios translate directly into lower downstream bandwidth consumption, or higher clarity at the same bandwidth.
Whether coding efficiency can be realized in real-time scenarios depends on whether the new-generation toolchain is “real-time friendly”:
- Share of low-latency tools: the previous generation included many inter-frame prediction tools that improve compression. They rely on longer reference frame sequences and introduce extra processing latency, so real-time scenarios can only enable them partially. To unlock its gains, the next-generation standard needs more tools that still perform well under short references and low reordering.
- Reference buffer pressure on memory bandwidth: richer reference frame management means more off-chip memory accesses. High-resolution real-time applications on phones and low-end devices must account for power consumption and heat.
- Coding complexity and hardware encoder adoption: new tools may be quite time-consuming on general-purpose processors, so their real-world rollout speed depends on how much hardware encoders support them. Whether a codec is “new or old” must ultimately be weighed together with “whether a low-latency hardware encoding path is available.”
4. Coding Challenges and Key Technical Directions in Real-Time Interaction
4.1 Latency Budget: Breaking Total Time Down by Stage
The end-to-end latency of a real-time pipeline can be split into several stages. The table below lists common magnitudes from engineering experience (illustrative only; actual values vary widely by scenario, network, and hardware):
| Stage | Typical time magnitude | Notes |
|---|---|---|
| Capture and preprocessing | A few milliseconds | Capture, scaling, color conversion, denoising, etc. |
| Encoding | A few to tens of milliseconds | Strongly tied to resolution, frame rate, and software/hardware encoding |
| Packetization and sending | ~1 to a few milliseconds | Protocol encapsulation, smooth transmission |
| Network transmission | Tens to over a hundred milliseconds | Depends on physical distance, congestion, and routing |
| Jitter buffer | A few to tens of milliseconds | Trade-off between latency and stalling |
| Decoding | A few milliseconds | Mostly done in hardware |
| Rendering and display | A few milliseconds to one refresh cycle | Tied to refresh rate and the composition pipeline |
A few easily overlooked details in the encoding stage:
- Frame time budget: resolution and frame rate together determine the time budget per frame. At 1080p/60fps, each frame has only about 16.7 ms, and encoding must finish within that window—otherwise frames pile up and latency compounds.
- B-frame reordering overhead: the coding order for bidirectional prediction differs from the display order, so the decoder must buffer and reorder frames, adding a fixed latency of several frames. Real-time encoding therefore usually limits or disables B-frames, trading some compression for latency.
- Buffer verification model constraints: to smooth bitrate fluctuations, encoding often uses VBV/HRD-like buffer model constraints on instantaneous bitrate. In real-time scenarios, the buffer size must be tightened to suppress short-term bursts.
4.2 Screen Content: From “Camera Thinking” to “Interface Thinking”
Screen sharing is a high-frequency need in remote work and teaching, but the signal characteristics of screen content differ markedly from natural scenes:
- Signal characteristics: large flat color blocks, sharp text edges, a limited color palette, and asynchronous regional changes (only the cursor or a single window is moving).
- Common artifacts: encoders optimized for natural video tend to introduce ringing, blurring, or “creeping” on text, which is especially noticeable at small font sizes.
- Coding responses: tools such as palette mode, block copying, and frame repetition are optimized for “concentrated colors and stable details” and can keep text sharp at low bitrates.
- Scene switching: a single meeting may mix camera footage, presentation slides, and screen sharing, so the coding configuration needs to switch automatically by content type.
It is worth emphasizing that the core of screen content coding is not compression ratio but readability. If a decimal point in a title, a line of code, or a table becomes blurred, information transfer can fail.
4.3 Content-Adaptive Encoding: Guiding Coding Effort by Spatial Complexity
Real-time video content varies widely, including games, remote-control footage, live people on camera, and virtual digital humans. Take game video as an example: rich high-frequency textures, intense motion, and hard-to-predict frame changes mean that full-frame encoding of such content often needs several times the bitrate of ordinary content to stay sharp.
A general and effective approach is content-adaptive encoding (CAE):
- The encoder first performs spatial complexity analysis of the frame, typically based on residual energy after intra prediction or on texture strength.
- The frame is partitioned into regions of different complexity by block (e.g., macroblock or coding unit), allocating more of the bitrate budget to key regions such as text, UI, and faces, and compressing less-sensitive regions such as the background.
- This is essentially a scheduling strategy that prioritizes “the human eye and business value”: since total bitrate is limited, the most important information is guaranteed first.
4.4 Rate-Distortion Optimization and Perceptual-Driven Coding
Traditional encoding minimizes a distortion metric as its optimization objective, but the human eye is not equally sensitive to all distortions. Perceptual-driven coding introduces the laws of visual perception into coding decisions. Common techniques include:
- Leveraging temporal and spatial masking effects to relax quantization in fast-moving regions;
- Protecting the edge structure of text and lines in screen content;
- Tilting quality toward skin tones and facial regions.
If the next-generation coding standard introduces finer perceptual models into rate-distortion optimization, two bitstreams with similar objective metrics could show a more noticeable difference in actual viewing experience.
4.5 Packet-Loss Resilience and Network Adaptation: The Source of Robustness for Real-Time Links
Real-time links run over uncontrollable networks, so the coding and transport layers must be designed together:
- Transport mechanisms: prefer transport schemes that maintain ordered delivery over unreliable channels and support selective retransmission and congestion feedback.
- Forward error correction (FEC): sending redundant data in advance lets the receiver recover some packet loss without waiting for retransmission. This suits links with higher round-trip latency but comes with a fixed bandwidth overhead.
- Frame-dropping strategy: when the network is persistently congested, actively dropping frames with shallower reference relationships recovers faster than blindly retransmitting.
- Keyframe interval: a longer interval yields higher compression efficiency, but if packet loss causes decoding errors, recovery to a clear picture takes longer. Real-time scenarios need a compromise between the two.
- Layered coding: with structures such as temporal layering, packet loss only affects higher-layer frames while the base layer still decodes normally, so the picture degrades briefly rather than breaking.
- Congestion control: continuously estimate available bandwidth from receiver feedback and dynamically adjust the send rate to avoid sustained overload and queuing latency.
4.6 Hardware/Software and Device-Cloud Collaboration
The fact that a coding format can be widely adopted in browsers owes to both mature software implementations and the large-scale penetration of hardware decoders. Real-time scenarios place higher demands on hardware support at the encoding side: hardware encoding typically processes a single frame within a fixed latency of a few milliseconds, with more controllable power consumption and heat, whereas pure software encoding, though flexible, may not finish encoding in time at high resolution and high frame rates.
Therefore, real-time interaction should follow these principles:
- Prioritize enabling the low-latency hardware encoding path;
- Confirm through capability negotiation whether the endpoint has the corresponding coding capability;
- Gracefully fall back to a more compatible coding format when it is not supported;
- For “encode in the cloud, decode on the terminal” architectures such as cloud gaming and cloud phones, also pay attention to the elastic scaling of the cloud-side encoding resource pool and the proximity scheduling of encoding instances to edge nodes.
4.7 Integration with AI Capabilities
Artificial intelligence is influencing real-time video processing from two directions:
- Pre-processing: lightweight super-resolution, denoising, and frame interpolation models can restore perceptual quality after low-resolution, low-bitrate transmission, mitigating clarity loss on weak networks.
- Post-processing and generation: generative models can “complete” picture details, improving subjective quality at extremely low bitrates.
This does not mean AI can replace standard coding:
- Generative completion may alter original details, posing accuracy risks in scenarios such as medical imaging and industrial inspection;
- The extra latency of model inference must be counted into the end-to-end budget;
- Terminal compute varies widely, and model distribution and version management are costly.
A more realistic path is to treat AI as an enhancement layer on top of standard coding, enabled on demand by the business and validated for its gains in key scenarios through quality evaluation.
5. Engineering Practice: Building a Low-Latency Real-Time Video Pipeline
5.1 Latency Budget Management
Set an explicit budget for each stage, and continuously collect and optimize through instrumentation:
# Illustrative latency budget plan (values are starting points from experience and must be tuned to real measurements)total_budget_ms = 200 # Real-time interaction experience target (end-to-end)budget = {"capture": 10, # Capture and preprocessing"encode": 30, # Encoding"network": 80, # Network transmission (including jitter)"jitter": 20, # Jitter buffer"decode": 10, # Decoding"render": 30, # Rendering and display}assert sum(budget.values()) <= total_budget_ms
In practice, record first-frame latency, end-to-end latency, stall count, and stall duration separately, and pay particular attention to high-percentile metrics such as P95/P99. For real-time interaction, the worst-case experience often determines whether users stay in the product.
5.2 Adaptive Bitrate and Quality Switching
The adaptation logic needs to consider both network bandwidth and content complexity:
def update_target_bitrate(estimate_bps, loss_rate, rtt_ms, base_bitrate):"""Adjust the target bitrate based on bandwidth estimate, loss rate, and round-trip time (illustrative logic)."""# Fast fallback with a safety margin under sustained congestionif loss_rate > 0.05 or rtt_ms > 400:return max(base_bitrate * 0.3, int(estimate_bps * 0.6))# Ramp up gradually when bandwidth is ample, avoiding frequent oscillationif estimate_bps > base_bitrate * 1.5 and loss_rate < 0.01:return min(base_bitrate * 1.5, int(estimate_bps * 0.8))# Otherwise converge toward 80% of the bandwidth estimatereturn int(estimate_bps * 0.8)
Beyond bitrate, you can jointly adjust frame rate, resolution, and the combination of coding tools. On weak networks, moderately lowering resolution to trade for stable frame rate often improves the feel of interaction more than simply lowering the frame rate.
5.3 Differentiated Configurations for Screen Content and Gaming
Screen content coding should focus on:
- Preferring products or solutions that support screen content coding tools;
- Preserving adequate resolution (especially for small fonts) and avoiding over-downsampling just to save bandwidth;
- Automatically lowering the frame rate for static presentation screens and raising it only when content changes;
- For video conferencing, splitting camera footage and shared screens into separate bitstreams to encode independently.
Game streaming should focus on:
- Adopting a low-latency coding configuration, disabling or strictly limiting B-frames;
- Enabling content-adaptive encoding, tilting the budget toward UI and text;
- Enabling temporal layering to achieve graceful degradation under packet loss.
5.4 Quality Evaluation: Do Not Look at Averages Alone
Evaluating a coding solution should cover three dimensions—quality, bitrate, and latency:
- Content-stratified sampling: test natural video, screen content, games, close-ups of people, and other types separately, so that overall averages do not mask weaknesses.
- Subjective evaluation: use methods such as Double Stimulus Continuous Quality Scale (DSCQS) to let viewers compare different solutions alternately within the same scenario.
- Objective metrics: use PSNR, SSIM, and VMAF to measure quality, and plot rate-distortion curves to compare the quality gap between encoders.
- Real-time-specific metrics: end-to-end latency, stall rate, first-frame time, and audio-video sync offset.
5.5 Observability and Canary Rollouts
Optimizing real-time systems depends on data feedback. It is recommended to build a monitoring system covering:
- Quality metrics: receive-side bitrate, resolution, frame rate, packet loss rate, and decode error count;
- Latency metrics: capture-to-send time, end-to-end time, and first-frame time;
- Experience metrics: stall count and duration, and switching success rate;
- Alerting: using cloud log services and monitoring/alerting capabilities, configure threshold alerts and anomaly detection for metrics such as latency and stalls to locate problems quickly.
For launching a new coding format, a canary strategy is recommended: enable it on a small fraction of traffic first, run an A/B comparison against the existing coding solution, and only scale up gradually after validating the compression gains and latency costs.
6. Conclusion
Recognition of an open-source video coding format by an award in the film engineering field is the result of open-source collaboration and royalty-free licensing driving technology adoption; listing real-time interaction as a core direction for the next-generation coding standard signals that video coding is moving from “for distribution” toward “for interaction.” For developers and businesses, three points deserve priority:
- Start from the latency budget: plan the full pipeline from capture to rendering, avoiding the overall imbalance caused by local optimizations;
- Make content adaptation the core: differentiated coding for screen content, games, and people-on-camera scenes is the key to improving experience;
- Iterate with a data-driven approach: combine subjective evaluation, objective metrics, and online monitoring to continuously optimize coding strategies.
For businesses that need to quickly build real-time audio/video capabilities, the capabilities of Baidu AI Cloud in real-time audio/video communication, audio/video processing, and edge distribution can serve as a starting point for solution selection. On this basis, targeted tuning for your own business scenarios is what ultimately balances experience and cost.
In the era of real-time interaction, video coding is no longer just the technology of “making the picture smaller,” but a key piece of infrastructure that connects people with people and people with services. Whoever does this link more stably, faster, and more clearly will gain the initiative in the next round of competition.
