Saliency-Guided Packet Loss Mitigation for Video Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional packet loss mitigation methods in content streaming, such as video streaming, often increase network bandwidth consumption, latency, and reduce available bitrate, particularly in low-latency applications like high-definition video and video game streaming, due to the indiscriminate application of error correction techniques across the entire frame.
Innovation Solution
Selective application of packet loss mitigation methods, guided by saliency maps, to identify and prioritize important regions of a frame, applying techniques like forward error correction and packet duplication only to those regions, thereby reducing bandwidth usage and maintaining video quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If forward error correction packets are transmitted to mitigate packet loss, then video quality and reliability are improved, but bandwidth consumption increases and available bitrate for content decreases
Solution Approach 1:
The video frame is segmented into multiple regions with different saliency values, and error correction is selectively applied only to high-saliency regions rather than the entire frame. This segmentation allows the system to protect critical visual information while reducing overall bandwidth consumption for error correction data.
Solution Approach 2:
Different error correction strategies are applied to different regions of the frame based on their saliency values. High-saliency regions receive stronger protection (e.g., redundant packets), while low-saliency regions receive minimal or no error correction, optimizing the trade-off between video quality and bandwidth usage.
2Reliability
If error mitigation techniques are applied across the entire frame, then packet loss is reduced, but latency increases due to additional processing and transmission overhead
Solution Approach 1:
By dividing the frame into salient and non-salient regions, the system processes and transmits error correction data only for critical regions. This reduces the total amount of data that requires error mitigation processing, thereby reducing latency while maintaining reliability for important visual content.
Solution Approach 2:
Instead of applying error mitigation to the entire frame (excessive action), the system applies it only to high-saliency regions (partial action). This partial application is sufficient to maintain perceived video quality while significantly reducing the processing and transmission overhead that causes latency.
3Loss of information
If packet duplication or forward error correction is applied to all regions, then data integrity is improved, but the bitrate available for actual video content is reduced
Solution Approach 1:
The system applies different levels of error protection to different frame regions based on saliency. High-saliency regions (e.g., faces, text, action areas) receive packet duplication or FEC to ensure data integrity, while low-saliency regions receive minimal protection, maximizing the bitrate available for actual video content while maintaining perceived quality.
Solution Approach 2:
The frame is segmented into regions of varying importance, allowing the system to allocate bitrate efficiently. Critical regions receive redundant data for integrity protection, while non-critical regions use fewer bits, overall increasing the bitrate available for video content compared to uniform protection across the entire frame.
Data Source
AI summary
Approaches presented herein provide systems and methods to selectively apply packet loss mitigation methods to one or more regions of a frame that have a sufficient importance value. The importance value may be determined by a saliency map generated for the frame that determine the most important content elements or regions of the frame. An importance value may be computed based on the saliency map and then, for areas of sufficient importance, selective mitigation methods may be used to reduce bandwidth, conserve compute resources, and provide error correction or duplication for important regions of the frame.