3D Point Cloud Upsampling Using 2D Image Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Three-dimensional (3D) models generated using 3D scanning technologies often suffer from incomplete data due to illumination changes, resulting in 'holes' or missing information on object surfaces, particularly for objects like satellites or spacecraft, where illumination varies widely, leading to insufficient feature detection and incomplete 3D models.
Innovation Solution
A method and system that combine 2D and 3D sensor fusion, using a 2D imaging sensor and a 3D imaging sensor to capture images of an object, where a processor generates an upsampled 3D point cloud by filling missing points with local entropy data from 2D images, and merges multiple upsampled point clouds from different viewpoints to create a denser and more complete 3D model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If 3D scanning technology is used to generate 3D models, then 3D model generation is achieved, but the model contains holes and missing information due to illumination changes
Solution Approach 1:
A 2D image from a 2D imaging sensor is used as an intermediary to fill missing 3D points. The 2D image provides pixel intensity information that is mapped to corresponding 3D locations, serving as a mediator to recover data lost due to illumination changes in the 3D scanning process
Solution Approach 2:
The patent combines data from heterogeneous sensors - a 3D imaging sensor (LiDAR) and a 2D imaging sensor (camera). By merging the 3D point cloud with 2D image data through sensor fusion, the system compensates for illumination-related deficiencies in the 3D scanning process
2Reliability
If 3D scanning is performed with varying illumination, then 3D model acquisition is possible, but feature detection becomes inconsistent across images
Solution Approach 1:
The system creates a composite representation by combining 3D geometric data from LiDAR with 2D photometric data from a camera. This composite approach leverages the strengths of both sensors - the illumination-invariant 3D geometry and the illumination-sensitive 2D texture - to achieve consistent feature detection across varying lighting conditions
3Quantity of substance
If only 3D imaging sensor is used, then 3D point cloud is captured, but the point cloud is sparse and incomplete
Solution Approach 1:
The patent merges data from two different sensor types - a 3D imaging sensor (LiDAR) and a 2D imaging sensor (camera). This combination increases the quantity of data points available for 3D model construction while maintaining a relatively simple sensor system architecture
Solution Approach 2:
The 2D imaging sensor serves multiple functions: it captures 2D images for texture information and simultaneously provides data for filling missing 3D points through entropy-based upsampling. This multi-functionality increases data quantity without proportionally increasing system complexity
Data Source
AI summary
A method for generating a 3D model point cloud of an object includes capturing a 2D image of the object and a 3D image of the object. The 3D image of the object includes a point cloud. The point cloud includes a multiplicity of points and includes a plurality of missing points or holes in the point cloud. The method additionally includes generating an upsampled 3D point cloud from the 3D image using local entropy data of the 2D image to fill at least some missing points or holes in the point cloud and merging a model point cloud from a previous viewpoint or location of a sensor platform and the upsampled 3D point cloud to create a new 3D model point cloud. The method further includes quantizing the new 3D point cloud to generate an updated 3D model point cloud.