The Next Generation of Video Coding for Real-Time Interaction

    This article explains how real-time interaction is reshaping video coding. It covers the fundamental differences between offline transcoding and live pipelines, and the four core requirements for real-time encoders: ultra-low latency, packet loss resilience, scalable coding, and adaptive bitrate control. It then outlines key advances, including screen content coding, perceptual and interaction-aware encoding, AI-assisted enhancement, and real-time audio-video coordination. Practical examples for cloud gaming, real-time calls, and cloud rendering are followed by a sample encoder configuration and deployment guidance. The article closes with the remaining trade-offs and outlook. The content is intended for developers and architects who design and operate real-time media systems.

    1. Introduction: Real-time interaction is reshaping the direction of video coding

    For more than a decade, the main goal of video coding has been to achieve higher visual quality at a lower bitrate. Success was usually measured by compression efficiency (BD-Rate) and offline transcoding throughput. That picture is changing. Cloud gaming, real-time audio and video calls, cloud rendering, remote collaboration, and immersive interaction are growing rapidly. As a result, the central requirement for coding technology is shifting fundamentally: from “optimal offline” to “usable in real time”, and from “quality first” to “latency and robustness first”.

    Recently, a new generation of open video coding standards received recognition from a major international technical award. An open-source community and browser vendors have also shared technical directions for next-generation coding standards. One term that keeps coming up is “real-time interaction”. This shows that the industry has reached a consensus: the ability to code for real-time interactive use cases is becoming a core yardstick for designing and deploying the next generation of codecs.

    This article uses real-time interaction as its main thread. It breaks down what these scenarios fundamentally require of encoders, outlines where the next generation of coding technology is heading, and offers practical engineering guidance.

    2. Real-time interaction vs. offline transcoding: fundamental differences

    To understand why real-time interaction is reshaping coding technology, it helps to first clarify how its constraints differ from those of traditional offline video processing.

    In offline transcoding scenarios such as video on demand and film or TV distribution, the encoder has a generous time budget. It can use strategies such as multi-pass encoding, complex motion search, and offline rate control to push quality and compression as far as possible. Latency is almost irrelevant, and it is perfectly acceptable for a file to take a few extra minutes to encode.

    In real-time interaction scenarios, the end-to-end latency budget for the entire pipeline is often squeezed to within a few hundred milliseconds. The system must also satisfy the following constraints:

    • Sensitivity to end-to-end latency: Capture, encoding, transmission, decoding, and rendering all draw from the same total budget. The encoder itself has very little time, so it cannot afford complex multi-pass optimization.
    • Unpredictable networks with high jitter: Real-time streams usually travel over the public internet, where packet loss, jitter, and sudden bandwidth drops are the norm rather than the exception.
    • An interaction loop: User actions (such as clicks and drags in cloud games) immediately trigger changes on screen. The encoder must react quickly to sudden visual changes and guarantee quality in regions relevant to key interactions.
    • Diverse content: Besides natural-scene video, there is a large amount of screen content, including screen sharing, slides, code editors, and documents. Their requirements for text clarity, color block edges, and so on differ greatly from those of natural video.

    This combination of high latency sensitivity, strong network uncertainty, and immediate interactivity is the fundamental driver of next-generation coding standards toward real-time interaction.

    3. Four core requirements for real-time encoders

    3.1 Ultra-low latency: not just “fast encoding”, but a “fast pipeline”

    Real-time encoding latency comes from several stages: frame capture buffering, internal encoding buffers (such as the reordering delay caused by B-frames), packetization and transmission, network jitter buffers, and decoding and rendering. Reducing end-to-end latency takes more than optimizing the encoder itself:

    • Avoid coding tools that depend on future frames (such as unstructured B-frame reordering), or use low-latency encoding modes.
    • Shrink the encoder’s internal lookahead window, balancing compression efficiency against latency.
    • Adaptively shrink jitter buffers in coordination with the network layer, rather than using a fixed large buffer.

    In real-time scenarios, the goal is usually not maximum per-frame compression. It is to push latency as low as possible while meeting a minimum quality bar.

    3.2 Packet loss resilience and error recovery: “stable” matters more than “beautiful”

    Packet loss on the public internet can cause blocking artifacts, stuttering, or even prolonged recovery failures at the decoder. Next-generation real-time coding standards generally emphasize the following mechanisms:

    • Periodic keyframes (IDR) and predictable refresh points: A well-chosen keyframe interval lets receivers rebuild the picture quickly after packet loss.
    • Reference frame management and intra refresh: Limiting how deeply a frame depends on earlier frames reduces the chance that a single lost packet causes prolonged cascading corruption.
    • Coordination of forward error correction (FEC) with retransmission (NACK/PLI): Design a feedback channel between the coding layer and the transport layer so that packet loss can trigger keyframe requests or error-correcting retransmissions.
    • Temporal layering: Organize frames into different temporal layers so that, even when higher layers are lost, lower layers still maintain basic visual continuity.

    3.3 Scalable coding: encode once, adapt to many endpoints

    Real-time interaction scenarios often use one-to-many or many-to-many topologies, such as meetings and interactive live streaming. Endpoints can differ dramatically in computing power, network conditions, and screen size. Scalable coding lets the same bitstream be “trimmed” to fit each endpoint’s capabilities:

    • Temporal scalability: A base layer ensures a smooth baseline, while higher layers provide higher frame rates.
    • Spatial scalability: A base layer provides a low-resolution, fast preview, while higher layers add high-resolution detail.
    • Quality scalability (SNR): Multiple quality layers at the same resolution.

    With scalability, the server only needs to encode one stream or a limited number of streams. It can then adapt to a massive number of endpoints through layering or nearby transcoding, which significantly reduces server computing and bandwidth costs.

    3.4 Adaptive bitrate: a real-time negotiation with network conditions

    Real-time encoding must match the available bandwidth dynamically. Otherwise, a bitrate that is too high causes congestion and packet loss, while one that is too low causes a sharp drop in quality. Engineering practice typically combines:

    • Bandwidth estimation, such as probing available bandwidth based on receiver feedback.
    • Congestion control, which dynamically adjusts the sending rate at the transport layer.
    • Encoder rate control integration, which adjusts quantization parameters and the target bitrate in real time based on bandwidth estimates.

    These three elements must form a tight feedback loop to sustain the highest quality possible in a fluctuating network.

    4. Where next-generation coding technology is heading

    In response to these requirements, coding technology for real-time interaction is evolving along several dimensions.

    4.1 Screen content coding (SCC)

    Remote work, online education, and cloud desktops all involve large amounts of screen content. Unlike natural video captured by a camera, screen content has large flat areas, sharp text edges, precise color blocks, and high-contrast lines. To handle these characteristics, next-generation encoders commonly introduce:

    • Intra prediction tools designed for text and lines, which reduce ringing and blurring.
    • Palette mode, which uses indexed encoding for UI elements with a limited color palette.
    • Dedicated rate control strategies for screen content, aimed at keeping text readable even at very low bitrates.

    4.2 Interaction-aware and perceptual coding

    Traditional coding targets pixel-level distortion metrics such as PSNR and SSIM. In real-time interaction, however, the user’s subjective experience is the ultimate standard. As a result, perceptual coding and task-oriented coding are receiving more attention:

    • When bandwidth is limited, prioritize quality in regions where the user’s attention is focused, such as interaction focus areas, faces, and text.
    • Take human visual characteristics into account. Make moderate trade-offs on high-frequency details, and spend bitrate where users can actually perceive it.
    • Use more stable coding strategies for interaction-sensitive regions, so that quality fluctuations do not affect the operation experience.

    4.3 AI-enhanced coding and post-processing

    Deep learning is being introduced into real-time encoding pipelines, mainly to trade compute for bandwidth or quality:

    • Super-resolution and quality enhancement: Transmit at a lower resolution and let lightweight models reconstruct high-definition output at the receiver.
    • Neural codecs and intelligent denoising: Improve perceived quality in capture scenarios with low light or heavy noise.
    • Intelligent ROI (region of interest): Automatically identify important regions and allocate them more bitrate.

    It is worth noting that these capabilities mostly exist as “general technical directions”. Their actual implementation and effectiveness depend heavily on engineering tuning and scenario adaptation. There is no “one model fits all”.

    4.4 Real-time audio-video and multimodal coordination

    Real-time interaction is not only about video, but also about coordinating sound and multimodal content. Recent industry progress is also reflected in stronger real-time speech generation capabilities, such as synthesizing speech with greater expressiveness, rhythm control informed by context, and consistency across multiple speakers. When incorporating these capabilities into real-time pipelines, several factors need special attention: lip-sync between audio and video, end-to-end latency alignment, and quality degradation strategies under network jitter. Audio is usually more “resilient” than video, so when bandwidth is tight, continuity and clarity of speech should be prioritized.

    5. Implementation in typical real-time interaction scenarios

    5.1 Cloud gaming: an extreme balance between latency and quality

    Cloud gaming is extremely sensitive to end-to-end latency; in practice, the round trip from user action to visual feedback typically needs to stay within a very tight range. Engineering practice usually involves:

    • Using low-latency encoding modes and disabling high-latency coding tools.
    • Responding quickly to key frames after player actions and prioritizing image quality in interaction regions.
    • Combining predictive rendering with fast keyframe refresh to shorten recovery time after packet loss.

    5.2 Real-time audio and video calls: robustness first

    For meetings, interactive live calls, and voice chats, the core requirement is “stay stable and never drop”. Key practices include:

    • Scalable encoding combined with selective forwarding at the server to serve endpoints on weak networks.
    • A feedback loop combining FEC with keyframe requests to quickly repair damage caused by packet loss.
    • Prioritizing audio: on weak networks, lower the video bitrate first to keep speech continuous.

    5.3 Cloud rendering and immersive interaction: compute and experience in sync

    AR/VR and cloud rendering scenarios demand both low latency and high quality. Beyond encoding, they also need:

    • Careful trade-offs between front and back buffers to reduce motion-to-photon (MTP) latency.
    • Foveated rendering driven by eye tracking or head pose, coordinated with encoding resource scheduling.
    • Edge nodes deployed close to users to shorten physical transmission distances.

    6. Engineering implementation: sample encoder configuration

    The following illustrative pseudocode shows how encoding parameters can be configured for real-time interaction scenarios. It is not any specific product interface and is intended only to explain general concepts.

    1. # Encoder configuration for real-time interaction (illustrative, not a specific product interface)
    2. encoder_config = {
    3. "mode": "realtime", # low-latency real-time mode instead of offline-optimal mode
    4. "bitrate_kbps": 2500, # target bitrate, dynamically updated by bandwidth estimation
    5. "rc_mode": "cbr_adaptive", # rate control mode: adaptive, following network conditions
    6. "gop_frames": 60, # keyframe interval, balancing random access and bitrate
    7. "idr_interval_sec": 2.0, # periodic IDR keyframes for fast recovery after packet loss
    8. "b_frames": 0, # disable B-frames to avoid reordering latency
    9. "lookahead_frames": 0, # disable lookahead to reduce encoding latency
    10. "temporal_layers": 2, # temporal layering; higher layers can be dropped on weak networks
    11. "error_resilience": {
    12. "enable_fec": True, # forward error correction for random packet loss
    13. "enable_nack": True, # allow feedback-based retransmission
    14. "on_pli_request_idr": True, # quickly refresh when a keyframe is requested
    15. },
    16. }
    17. def on_bandwidth_update(estimated_kbps, config):
    18. """Adjust the target bitrate when the bandwidth estimate changes"""
    19. target = max(MIN_BITRATE, min(estimated_kbps * 0.8, MAX_BITRATE))
    20. config["bitrate_kbps"] = int(target)
    21. # On weak networks, tighten the keyframe interval to speed up recovery
    22. if target < LOW_BANDWIDTH_THRESHOLD:
    23. config["idr_interval_sec"] = 1.0
    24. return config

    This configuration illustrates three principles: disable high-latency tools, converge the bitrate dynamically with bandwidth, and treat recovery from packet loss as a first-class concern. In real engineering work, these parameters need continuous tuning based on the specific scenario, endpoint distribution, and network profiles. There is no one-size-fits-all set of optimal values.

    7. Deployment architecture and cloud coordination

    Services built for real-time interaction usually need coordination with cloud capabilities. An elastic architecture can be built using general-purpose cloud product categories:

    • Audio/video processing and transcoding: Use elastic compute for real-time transcoding, layer packaging, and format adaptation.
    • Edge computing and nearby access: Push encoding and transcoding nodes down to the edge, close to users, to shorten physical latency.
    • Content delivery network (CDN): Handle large-scale distribution and nearby origin fetching.
    • Object storage: Store recordings and media assets for playback and further processing.
    • Message queues: Decouple signaling, events, and asynchronous tasks in the real-time pipeline.
    • Logging and monitoring with alerting: Collect and alert on core metrics such as end-to-end latency, stall rate, packet loss rate, and first-frame time in real time, forming an observability feedback loop.

    At the deployment level, it is recommended to separate the signaling control plane from the media data plane. The control plane handles session negotiation and scheduling, while the data plane focuses on low-latency media forwarding. This keeps control logic from blocking the real-time pipeline.

    8. Challenges and outlook

    The evolution of coding for real-time interaction still faces several challenges:

    • The triangle of latency, quality, and robustness: It is difficult to optimize all three at once, so trade-offs need to be made for each scenario.
    • Complexity of adapting to heterogeneous endpoints and networks: Endpoint computing power varies widely, so layering and degradation strategies need careful design.
    • Compute overhead of AI-enhanced capabilities: There is tension between model inference and real-time constraints, requiring coordinated scheduling between edge and endpoints.
    • Standardization and ecosystem compatibility: New tools need full-chain support across endpoints, servers, and the transport layer. How fast they are adopted depends on how mature the ecosystem is.

    Looking ahead, real-time interaction coding will keep evolving toward lower latency, stronger robustness, and smarter bitrate allocation, with increasing integration of AI capabilities. For developers and architects, the key to building an excellent real-time experience is to understand the fundamental constraints that real-time interaction places on coding. From there, good coordination across parameter configuration, pipeline optimization, and cloud architecture will make all the difference.