3D Human Reconstruction with Occlusion Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for reconstructing humans in 3D face challenges such as missing color and depth information due to occlusions, leading to loss of detail and incomplete representations.
Innovation Solution
A method that involves capturing multiple color and depth images from different viewpoints, using image segmentation to identify occluded regions, generating a point cloud, discounting points corresponding to occluding items, and adjusting a parametric model to reconstruct the human subject with occluded parts included.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If color and depth images are captured to form a point cloud, then detail preservation is improved, but completeness deteriorates due to occlusions
Solution Approach 1:
The method segments the point cloud into multiple components: visible surface points, occluded region points (identified through depth discontinuities and boundary analysis), and inferred points. This segmentation allows different processing strategies for different regions, preserving detail where visible while inferring complete geometry where occluded.
Solution Approach 2:
The method transitions from 2D image space to 3D point cloud space, then to a complete 3D mesh representation. By working in three dimensions and analyzing depth maps, the system can identify occluded regions and infer their geometry, recovering information that is lost when staying in 2D image space.
2Productivity
If deformable model fitting is used, then reconstruction speed is improved, but detail completeness deteriorates due to occlusions by hair or clothing
Solution Approach 1:
The method performs preliminary analysis of the point cloud to identify occluded regions before final mesh generation. By detecting depth discontinuities and boundary artifacts in advance, the system can prepare appropriate filling strategies, ensuring that occluded regions like hair and clothing are properly reconstructed before the final model is created.
Solution Approach 2:
The method introduces an intermediary processing stage between point cloud generation and final mesh creation. This intermediate stage includes occlusion detection, region classification, and selective point filtering, which prepares the data for complete reconstruction while maintaining the efficiency of automated processing.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A method of generating a three-dimensional (3D) reconstruction of a human is provided. The method comprises obtaining at least one colour image and corresponding depth image of a scene comprising a human subject that is at least partially occluded by one or more items. The regions in the at least one colour image corresponding to a human subject are identified. A point cloud of the scene is generated based on the at least one colour image and corresponding depth image. One or more parameters of a parametric model are then adjusted based on the regions of the point cloud identified as corresponding to the human subject. The adjusted parametric model is then used to generate a 3D reconstruction of the human subject; wherein the parametric model provides an estimate for points in the point cloud corresponding to the occluded parts of the human subject. A corresponding system is also provided.