Quality-Aware Error Concealment for Streaming Media
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing error concealment methods in game and video streaming fail to consider the complexity of streaming content, leading to stutters and severe corruption due to packet losses, as they either rely on bandwidth-limited Forward Error Correction or client-side reference picture invalidation, which drops frames regardless of content complexity.
Innovation Solution
A quality-aware error concealment (QEC) method that determines whether to discard or select decoded frames as reference frames based on their complexity, using frame statistics such as average bits per macro-block and motion information to reduce frame corruption and error propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reference picture invalidation is used to conceal errors, then packet loss corruption is reduced, but frame stutters and playback interruptions occur
Solution Approach 1:
The patent changes the parameter of reference frame selection from a binary decision (use or discard) to a quality-aware decision based on complexity metrics. By introducing complexity thresholds and quality assessment, the system dynamically adjusts which frames are selected as reference frames, thereby maintaining both frame quality and streaming continuity without the severe stutters caused by conventional invalidation methods.
Solution Approach 2:
The system dynamically adapts reference frame selection based on real-time assessment of frame complexity and content characteristics. Instead of a static invalidation approach, the patent implements dynamic decision-making that considers motion information, bit depth, and other content-specific parameters to determine optimal reference frame usage, thus balancing quality and continuity.
2Reliability
If Forward Error Correction is applied at the source, then packet loss resilience is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts the error concealment function from the source encoding side and relocates it to the client decoding side. By moving the complexity-based reference frame selection to the client, the system eliminates the need for bandwidth-intensive Forward Error Correction at the source, while still achieving robust packet loss resilience through intelligent client-side reference frame management.
Solution Approach 2:
The patent introduces complexity metrics and quality assessment mechanisms as intermediaries between packet loss and visual quality. Instead of directly combating packet loss through FEC, the system uses these intermediary assessments to selectively manage reference frames, providing an efficient middle ground that achieves resilience without excessive bandwidth consumption.
3Reliability
If all frames with packet losses are discarded, then error propagation is prevented, but user experience deteriorates due to frequent frame drops
Solution Approach 1:
The patent applies local quality assessment to individual frames based on their complexity characteristics. Instead of uniformly discarding all frames with packet losses, the system evaluates each frame's motion information, bit depth, and content complexity locally, making selective decisions about which frames to retain as reference frames. This localized approach prevents error propagation in critical areas while maintaining user experience through selective frame preservation.
4Productivity
If complex frames are used as reference frames, then coding efficiency is improved, but error propagation risk increases under packet loss
Solution Approach 1:
The patent implements dynamic reference frame selection that adapts to packet loss conditions. During normal operation, complex frames are utilized for coding efficiency. However, when packet loss is detected, the system dynamically switches to simpler, more robust reference frames based on complexity thresholds. This dynamic adaptation resolves the contradiction by allowing the system to exploit coding efficiency when conditions permit while mitigating error propagation risk when packet loss occurs.
Data Source
AI summary
A technique for streaming and a client device that uses the technique are disclosed herein. The disclosed technique determines context complexity of streamed data and determines whether to discard or select the streamed data for a future reference frame based on the context complexity of the streamed data. The streamed data is discarded if the content complexity is higher than a content complexity threshold, and the streamed data is selected if the content complexity is not higher than a content complexity threshold. This is based on the realization that error propagation in the case of a less complex video sequence is not very bothersome to the end user experience whereas corruption will be very severe in cases of highly complex sequences.