3D Key Point Reconstruction Using Occlusion Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D pose estimation techniques based on multiple input camera views face inaccuracies due to incorrect 2D key point detection, especially when objects are occluded, and challenges in identifying corresponding key points across different camera views.
Innovation Solution
A method for 3D key point reconstruction that involves obtaining multiple 2D images from different cameras, detecting 2D key points, determining occlusion measures such as frustum occlusion and mutual instance occlusion, and excluding occluded instances from 3D key point estimation to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple input camera views are used for 3D pose estimation, then the coverage and completeness of object detection is improved, but the accuracy of 3D key point reconstruction deteriorates due to occlusion errors
Solution Approach 1:
The system performs preliminary occlusion analysis on all detected 2D key points before performing 3D reconstruction. By calculating occlusion measures (such as depth-based occlusion, texture consistency, and geometric plausibility) in advance, the system identifies and filters out key points from occluded instances, ensuring that only high-quality key points are used in the subsequent 3D reconstruction process, thus maintaining accuracy while using multiple views.
Solution Approach 2:
The system creates virtual copies of key points from different camera views and uses occlusion measures to evaluate their reliability. By comparing multiple copies of the same key point across different views and selecting those with consistent occlusion characteristics, the system reconstructs 3D positions more accurately while maintaining the benefits of multi-view coverage.
2Quantity of substance
If 2D key points from occluded instances are included in 3D reconstruction, then the quantity of data for reconstruction is increased, but the reliability of the reconstruction deteriorates due to inaccurate key point detection
Solution Approach 1:
The system extracts and removes 2D key points from occluded instances by calculating occlusion measures for each detected key point. Using criteria such as depth discontinuities, texture inconsistencies, and geometric implausibility, the system identifies and extracts only the reliable key points from the full set, excluding those from occluded instances before performing 3D reconstruction. This ensures high reliability while maintaining sufficient quantity of valid key points.
3Device complexity
If conventional matching techniques are used to identify corresponding key points, then the simplicity of the method is maintained, but the precision of correspondence identification deteriorates when objects are occluded or viewed from different positions
Solution Approach 1:
The system introduces occlusion measures as an intermediary evaluation criterion between 2D key point detection and 3D correspondence identification. By calculating occlusion measures (depth consistency, texture similarity, geometric plausibility) for each potential correspondence, the system mediates the matching process to filter out incorrect matches from occluded or poorly visible instances, thereby improving correspondence precision without significantly increasing overall system complexity.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A method and system for reconstructing 3D key points. A plurality of 2D images, each providing a different view of a scene, are obtained and processed to detect 2D key points for one or more object instances. The detected 2D key points are processed to determine one or more measures of occlusion for each instance, and only 2D key points belonging to instances for which all of the one or more measures of occlusion satisfy one or more respective predetermined conditions are used to estimate a plurality of 3D key points for the scene.