Video Depth Map Scaling for Flicker-Free Temporal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing depth estimation techniques in video frames often result in flickering due to inconsistent relative depth changes between frames, particularly when there are no significant scene changes or camera movements, leading to undesirable flicker in depth maps.
Innovation Solution
A method involving generating a single image depth map for each frame, scaling it using grid point-based scale values derived from temporally related frames, and applying spatio-temporal filtering to generate a consistent depth map, utilizing machine learning and optical flow analysis to align and average depth values across frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If single image depth estimation is applied to each frame independently, then depth map generation is computationally efficient, but temporal consistency deteriorates causing flicker
Solution Approach 1:
The patent performs preliminary actions by generating depth maps for multiple frames in advance and computing scale values based on temporal relationships before final depth map generation. This allows the system to maintain temporal consistency without reprocessing all data at the final stage, balancing computational efficiency with stability.
Solution Approach 2:
The patent introduces scale values as an intermediary element that mediates between individual frame depth estimates and the final temporally consistent depth map. These scale values adjust depth estimates to maintain consistency across frames while preserving the computational efficiency of single-image depth estimation methods.
2Stability of the object's composition
If scale values are computed using multiple temporally related frames, then temporal consistency is improved, but computational complexity increases
Solution Approach 1:
The patent applies local quality by computing scale values at grid points rather than for every pixel, and by using neighboring grid point information locally. This localized approach maintains temporal consistency at critical points while avoiding the computational burden of processing all pixels with equal complexity.
Solution Approach 2:
The patent performs partial action by computing scale values only at grid points and using interpolation for intermediate pixels, rather than computing full-resolution depth maps for all frames. This provides sufficient temporal consistency while significantly reducing computational complexity compared to processing all pixels explicitly.
3Stability of the object's composition
If depth values are averaged across multiple frames, then temporal stability is improved, but depth precision may deteriorate due to motion blur
Solution Approach 1:
The patent applies dynamics by adapting the depth averaging process to account for motion. The system dynamically adjusts which frames are averaged and how depth values are combined based on detected motion and optical flow, maintaining precision for stationary regions while achieving stability for moving objects through selective temporal aggregation.
Solution Approach 2:
The patent changes parameters by using scale values to adjust the contribution of different frames to the final depth estimate. Rather than simple averaging, the system modifies the weighting parameters based on temporal relationships and motion characteristics, preserving precision while achieving stability.
Data Source
AI summary
A method and system for generating a depth map corresponding to a frame of a sequence of frames in a video clip is disclosed. This can involve generating a single image depth map for each of a plurality of frames, scaling the single image depth maps, and processing a time sequence of scaled single image depth maps to generate said depth map corresponding to the frame of the sequence of frames in the video clip.