Optical Flow Estimation for Occluded Video Pixels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processing techniques are inefficient in handling occluded pixels due to foreground and background object movements, leading to invalid flow vectors and artifacts in interpolation algorithms, particularly in real-time or near-real-time applications with limited compute and memory resources.
Innovation Solution
A lightweight method to estimate optical flow vectors for occluded pixels by using optical flow data to determine occluded regions and fill them with estimated flow vectors, including reversing known backward flow vectors for pixels moving out of the frame and using global flow for foreground object motion, thereby reducing the number of pixels requiring further processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional interpolation algorithms are used to generate intermediate frames, then frame quality can be maintained, but processing speed decreases and memory requirements increase
Solution Approach 1:
The patent segments the image into superpixels and identifies occluded regions separately, then applies different processing strategies to different regions. Non-occluded regions use standard optical flow while occluded regions use specialized handling, improving overall processing efficiency without sacrificing frame quality
Solution Approach 2:
The patent applies different quality levels of processing to different regions of the image. Occluded regions receive targeted flow vector estimation while non-occluded regions use more efficient methods, optimizing the balance between quality and processing speed for each specific region
2Measurement precision
If traditional occlusion handling methods are used, then flow vector accuracy can be maintained, but memory and processing requirements increase
Solution Approach 1:
The patent extracts and identifies occluded regions separately from the rest of the image, then applies specialized flow vector estimation only to those regions. This extraction approach maintains accuracy for occluded pixels while reducing overall computational burden by not applying complex methods to non-occluded regions
Solution Approach 2:
The patent performs preliminary identification of occluded regions before final flow vector computation. By pre-segmenting occluded areas and planning the processing strategy in advance, the system avoids unnecessary computations and reduces memory requirements while maintaining flow vector accuracy
3Loss of information
If comprehensive optical flow estimation is performed for all pixels, then flow vector completeness is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by focusing computational resources only on occluded regions that require flow vector estimation. Non-occluded regions are handled more efficiently, providing sufficient flow information without the full computational overhead of comprehensive estimation methods
Solution Approach 2:
The patent performs preliminary identification of occluded regions to determine where flow vector estimation is actually needed. This advance planning allows the system to complete necessary estimations faster by avoiding unnecessary processing in non-occluded areas while ensuring completeness where required
Data Source
AI summary
Systems and methods estimate optical flow vectors for occluded pixels between frames of a video sequence. Regions of occluded pixels may be identified and a cause of their occlusion may be determined. Different estimation techniques may be applied based, at least in part, on the cause of occlusion to provide a lightweight, less resource intensive estimation of optical flow data. Optical flow vectors for pixels that are occluded due to movement out of a frame may be estimated using a first technique while optical flow vectors for pixels that are occluded due to foreground movement may be estimated using a second technique.