Sub-frame Encoding for Video Streaming Packet Pacing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional video streaming methods face challenges with packet loss and latency, leading to visual artifacts like stutter and corruption due to the reliance on single bursts of data packets for frame transmission, which can result in increased latency and frame delay.
Innovation Solution
The approach involves encoding video frames into multiple sub-frames, prioritizing the transmission of the most critical sub-frames, and applying entropy-based masks to identify high-bit macroblocks, with enhanced error correction for the first sub-frame to ensure timely and error-free delivery of motion data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single burst of data packets is used to transmit an encoded frame, then transmission speed is improved, but packet loss increases leading to visual artifacts like stutter and corruption
Solution Approach 1:
The encoded frame data is divided into multiple sub-frames (first sub-frame, second sub-frame, etc.), where each sub-frame contains different types of data (e.g., motion data, texture data). These sub-frames are transmitted as separate packet bursts with different pacing strategies, allowing critical motion data to be delivered faster while less critical texture data is delivered with more error protection.
2Reliability
If smaller data packet bursts with predetermined delays are used for packet pacing, then packet loss is reduced, but latency increases resulting in visually disruptive artifacts
Solution Approach 1:
Different sub-frames receive different levels of error protection and pacing. The first sub-frame containing motion data is transmitted with higher priority and more aggressive pacing (less delay) because motion information is critical for visual quality. The second sub-frame containing texture data receives more conservative pacing with greater error protection, as texture data can tolerate more delay and error correction without significantly impacting visual quality.
3Reliability
If conventional packet pacing with predetermined delays is applied to all packets, then corruption from packet loss is reduced, but frame delay increases due to increased latency
Solution Approach 1:
The system dynamically adjusts packet pacing and error protection levels based on the importance of the data being transmitted. Critical sub-frames (e.g., motion data) receive aggressive pacing with minimal delay and stronger error correction, while less critical sub-frames (e.g., texture data) receive more conservative pacing with greater delay tolerance. This dynamic allocation optimizes the balance between frame integrity and frame delay.
Data Source
AI summary
In various examples, a frame may be encoded as multiple sub-frames. For example, data particularly relevant to conveying visual motion between frames may be encoded in a first sub-frame(s) with remaining data being encoded in a second sub-frame(s). Other information may be included in the first sub-frame(s), such as high entropy data. The high entropy data may be estimated using quantization and dequantization of macroblocks. Packet pacing may be applied at least between the encoded sub-frames. As the first sub-frame(s) may include the most important information for frame updates at the client device, if the second sub-frame(s) is not received and/or displayed the first sub-frame may be displayed providing high quality results. More error correction may be used for the first sub-frame than the second sub-frame to increase the likelihood that the first sub-frame is received at a client device.