Video Upsampling Using Sensor Motion Estimates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video coding/decoding systems introduce data loss and artifacts like shimmering, particularly during slow panning operations, due to incorrect motion estimation and prediction, leading to image quality issues and noticeable content discontinuities.
Innovation Solution
A method that estimates camera motion using motion sensor data to detect non-camera motion and engage upsampling processes only when within predetermined limits, merging forward and backward estimates of image content to reduce artifacts and improve image quality, while applying rolling shutter correction and spatial filtering as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If video coding/decoding is performed using pixel block-based predictive coding, then data compression efficiency is improved, but image quality deteriorates due to data loss and coding artifacts
Solution Approach 1:
The video sequence is divided into multiple segments or regions, with different coding strategies applied to different segments. High-importance regions use higher quality coding methods while low-importance regions use more aggressive compression, resolving the contradiction between overall compression efficiency and localized image quality.
Solution Approach 2:
Different quality levels are applied to different regions of the video content based on their importance. Critical regions maintain high quality while less important regions accept higher compression, allowing the system to achieve good overall compression while preserving quality where it matters most.
2Productivity
If motion estimation and prediction are used to exploit temporal redundancy, then compression efficiency is improved, but shimmering artifacts are introduced due to incorrect motion estimation
Solution Approach 1:
The system uses feedback mechanisms to detect and correct motion estimation errors. By monitoring for shimmering artifacts and adjusting motion estimation parameters accordingly, the system maintains compression efficiency while reducing the harmful shimmering effect caused by incorrect motion prediction.
Solution Approach 2:
The motion estimation parameters are dynamically adjusted based on content characteristics. By changing parameters such as search range, block size, or motion model complexity adaptively, the system achieves accurate motion compensation without introducing shimmering artifacts while maintaining good compression.
3Stability of the object's composition
If rolling shutter correction is applied to correct distortion during camera panning, then image stability is improved, but device complexity increases
Solution Approach 1:
Rolling shutter correction is applied as a preliminary processing step before main video coding. By correcting the distortion early in the processing pipeline, subsequent coding operations work with already-stabilized content, achieving image stability without requiring complex processing throughout the entire system.
Solution Approach 2:
An intermediate processing stage is introduced that handles rolling shutter correction separately from both capture and main coding. This intermediary module isolates the complexity of distortion correction, making it manageable while still achieving stable, artifact-free output video.
Data Source
AI summary
Embodiments of the invention provide techniques for upsampling a video sequence for coding. According to the method, an estimate of camera motion may be obtained from motion sensor data. Video data may be analyzed to detect motion within frames output from a camera that is not induced by the camera motion. When non-camera motion falls within a predetermined operational limit, video upsampling processes may be engaged. In another embodiment, video upsampling may be performed by twice estimating image content for a hypothetical new a frame using two different sources as inputs. A determination may be made whether the two estimates of the frame match each other sufficiently well. If so, the two estimates may be merged to yield a final estimated frame and the new frame may be integrated into a stream of video data.