Stereo Pixel Reclassification for Accurate 3D Dynamic Scene Depth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to accurately reconstruct three-dimensional scenes with dynamically moving content, particularly in augmented and virtual reality applications, due to challenges in distinguishing between static and dynamic pixels caused by global and local motion.
Innovation Solution
A method for re-classifying static and dynamic pixels using a combination of position sensor data and image-based tracking, followed by iterative refinement using deep neural networks and bundle adjustment, to optimize depth information and reconstruct the scene accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dynamic scene reconstruction methods are used to handle moving content, then accuracy of three-dimensional scene reconstruction is improved, but computational complexity and processing time increase significantly
Solution Approach 1:
The patent segments pixels into static and dynamic categories based on motion detection. Static pixels are processed using efficient static reconstruction algorithms, while only dynamic pixels require complex dynamic reconstruction processing. This segmentation resolves the contradiction by applying computational complexity only where needed rather than uniformly across the entire scene.
Solution Approach 2:
The patent applies different reconstruction quality and processing methods to different regions of the image based on their motion characteristics. Dynamic regions receive detailed dynamic reconstruction with higher computational resources, while static regions use simpler, more efficient algorithms. This local differentiation optimizes the balance between reconstruction accuracy and computational complexity.
2Measurement precision
If iterative refinement methods are applied to all pixels, then depth information accuracy is improved, but processing time increases
Solution Approach 1:
The patent divides pixels into static and dynamic sets and applies iterative refinement only to dynamic pixels that require it. Static pixels undergo simpler processing without full iterative refinement, reducing overall processing time while maintaining high accuracy for the critical dynamic portions of the scene.
Solution Approach 2:
Instead of applying full iterative refinement uniformly to all pixels (excessive action), the patent applies the refinement process only partially to dynamic pixels that need it. This partial action approach reduces processing time while sufficient for achieving the required depth accuracy in moving regions.
3Reliability
If position sensor data is combined with image-based tracking, then reliability of motion detection is improved, but device complexity increases
Solution Approach 1:
The patent merges position sensor data with image-based tracking data to create a more reliable motion detection system. By combining multiple data sources (inertial measurement units and visual tracking), the system achieves higher reliability through cross-validation and complementary information, while managing complexity through integrated processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In one embodiment, a method includes identifying, in each image of a stereoscopic pair of images of a scene at a particular time, every pixel as either a static pixel corresponding to a portion of a scene that does not have local motion at that time or a dynamic pixel corresponding to a portion of a scene that has local motion at that time. For each static pixel, the method includes comparing each of a plurality of depth calculations for the pixel, and when the depth calculations differ by at least a threshold amount, then re-labeling that pixel as a dynamic pixel. For each dynamic pixel, the method includes comparing a geometric 3D calculation for the pixel with a temporal 3D calculation for that pixel, and when the geometric 3D calculation and the temporal 3D calculation are within a threshold amount, then re-labeling the pixel as a static pixel.