Video Retargeting via Saliency Map Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video retargeting methods for displaying videos on devices with different formats, such as handheld devices, often result in distorted images or unused display space due to linear scaling or letter-boxing, which compromises the viewing experience and efficiency.
Innovation Solution
A computationally efficient approach for non-linear video warping that computes two-dimensional saliency maps and projects them to create horizontal and vertical saliency profiles, allowing for scaling while preserving the aspect ratio of salient regions, enabling efficient retargeting of high-definition video content in real-time using low processor clock speeds and modest hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If linear scaling is used to retarget video to different display formats, then the video can be displayed on devices with different resolutions, but the image becomes distorted when aspect ratios differ significantly
Solution Approach 1:
The patent applies local quality by differentiating the scaling treatment between salient and non-salient regions. Salient regions (containing important visual information) are scaled to preserve their aspect ratio, while non-salient regions can be scaled differently. This allows the system to maintain image quality for important content while adapting to different display formats, resolving the contradiction between versatility and aspect ratio preservation.
Solution Approach 2:
The patent segments the image into salient and non-salient regions based on content analysis. By dividing the image this way, the system can apply different scaling strategies to different segments, enabling both format adaptability and aspect ratio preservation for the critical salient portions.
2Manufacturing precision
If letter-boxing is used to preserve aspect ratio, then the original video aspect ratio is maintained, but the maximum usable display area is not utilized
Solution Approach 1:
Instead of uniformly preserving aspect ratio across the entire image (which creates black bars and wastes display area), the patent applies local quality by preserving aspect ratio only for salient regions while allowing non-salient regions to be scaled to fill the display. This eliminates wasted display space while maintaining aspect ratio for important content.
Solution Approach 2:
The patent dynamically adjusts the scaling behavior based on content analysis. The system determines which regions are salient and applies different scaling rules accordingly, allowing the display to adaptively fill available space while preserving aspect ratio where necessary, rather than using a static letter-boxing approach.
3Manufacturing precision
If non-linear video warping is used to retarget HD video, then visually pleasing results are achieved with proper aspect ratio preservation, but computational complexity and hardware requirements increase
Solution Approach 1:
The patent applies partial action by computing saliency maps and applying non-linear warping only to the extent necessary to identify and preserve salient regions. Rather than applying complex warping to the entire image uniformly, the system uses saliency information to selectively apply transformations only where needed, reducing overall computational complexity while maintaining retargeting quality.
Solution Approach 2:
The patent introduces saliency maps as an intermediary that guides the warping process. By first computing a saliency map that identifies important regions, the system can use this intermediate representation to direct the warping operation, reducing the computational burden of analyzing and transforming every pixel equally.
Data Source
AI summary
Techniques are disclosed for retargeting images. The techniques include receiving one or more input images, computing a two-dimensional saliency map based on the input images in order to determine one or more visually important features associated with the input images, projecting the saliency map horizontally and vertically to create at least one of a horizontal and vertical saliency profile, and scaling at least one of the horizontal and vertical saliency profiles. The techniques further include creating an output image based on the scaled saliency profiles. Low saliency areas are scaled non-uniformly while high saliency areas are scaled uniformly. Temporal stability is achieved by filtering the horizontal resampling pattern and the vertical resampling pattern over time. Image retargeting is achieved with greater efficiency and lower compute power, resulting in a retargeting architecture that may be implemented in a circuit suitable for mobile applications such as mobile phones and tablet computers.