Video Reframing Using Saliency and Coding Cost Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video reframing methods do not effectively consider the bit rate and distortion of the output encoded video signal, leading to increased coding costs and decreased video quality due to uncapped cropping window adjustments.
Innovation Solution
The method involves computing a saliency map and a macroblock coding efficiency cost map to determine optimal reframing window positions and sizes that minimize coding costs for the reframed video signal, using techniques like Kalman filters and macroblock cost maps to constrain window parameters and improve temporal consistency and coding efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the cropping window is dynamically adjusted to track the region of interest, then the user attention is maintained, but the coding complexity increases due to multiple zooming and panning operations
Solution Approach 1:
The patent pre-calculates and stores macroblock coding efficiency cost maps before actual encoding. These pre-computed cost maps are used to predict and avoid high-complexity reframing operations, thereby reducing the coding complexity burden during real-time encoding while maintaining region of interest tracking capability
Solution Approach 2:
The patent introduces macroblock coding efficiency cost maps as an intermediary layer between the saliency map and the reframing decision. This intermediary provides coding complexity information that mediates the trade-off between tracking region of interest and maintaining low coding complexity, allowing the system to select reframing operations that balance both requirements
2Adaptability or versatility
If the cropping window is adjusted without considering coding efficiency, then the region of interest is maintained, but the bit rate increases and video quality decreases
Solution Approach 1:
The patent implements a feedback mechanism where macroblock coding efficiency cost maps are computed and used to evaluate the impact of potential reframing operations. This feedback loop allows the system to adjust the cropping window while considering coding efficiency, thereby maintaining video quality and controlling bit rate while still tracking the region of interest
Solution Approach 2:
The patent changes the parameter selection criteria for the cropping window from solely based on saliency to a combined criterion that includes macroblock coding efficiency cost. This parameter change allows the system to optimize both region of interest tracking and video quality preservation by selecting window positions and sizes that minimize coding cost
Data Source
AI summary
Reframing is used to re-size an image or video content, e.g. for displaying video signals with a given aspect ratio on a display having a different aspect ratio. Window cropping parameters (position and size over time) are constrained in order to optimise the rate/distortion of the encoded output video signal. Initial reframing is improved by computing a saliency map representing a user attention map which considers the video coding context and by providing a macroblock coding efficiency cost map and then taking the saliency map or a combined saliency/coding cost map into account so that the coding cost for said reframed video signal is smaller than the coding cost for other candidate reframing windows' sizes and positions.