3D Scene Colorization Using ML and Occlusion Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D measurement systems are complex, require specialized personnel, and can be bulky, making them inefficient in time-sensitive situations such as crime or accident scene investigations. Additionally, these systems struggle with generating realistic 3D scenes with missing color information and occluded regions.
Innovation Solution
A system comprising a 3D scanner, a camera with a different viewpoint, and processors that access point clouds and 2D images to generate 3D scenes. This system uses a trained machine learning model, such as a generative adversarial neural network, to estimate colors for occluded and missing regions based on reflectance images and occlusion masks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 3D scanner with movable structure is used to capture 3D data, then measurement precision is improved, but device complexity and bulkiness increase
Solution Approach 1:
The system separates the 3D scanning function from the color capture function, using a stationary 3D scanner for precise geometric measurement and a camera for color information. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining measurement precision.
Solution Approach 2:
A processor acts as an intermediary that combines the point cloud data from the 3D scanner with the color image data from the camera. The processor maps color information onto the 3D point cloud, eliminating the need for complex integrated scanning devices while achieving accurate colorized 3D measurements.
2Measurement precision
If specialized personnel operate the scanning equipment, then measurement precision is improved, but loss of time increases due to setup and operation complexity
Solution Approach 1:
The system performs automated processing of the point cloud and color image data through a processor that automatically aligns, maps, and integrates the data streams. This self-service automation eliminates the need for specialized operators to manually process the data, reducing both the skill requirement and the time needed to complete the scanning process.
3Manufacturing precision
If multiple captures are used to obtain complete color information, then color accuracy is improved, but productivity decreases
Solution Approach 1:
The system captures all necessary color information in a single image taken before or during the 3D scanning process. The processor then maps this pre-captured color information onto the point cloud data, eliminating the need for multiple sequential captures and significantly improving scanning speed while maintaining color accuracy.
Data Source
AI summary
Techniques are described to generate a 3D scene by mapping a point cloud with a 2D image, and colorize portions of the 3D scene synthetically. An input is received to select, from the 3D scene, a portion to be colorized synthetically. The colorizing includes generating a reflectance image based on an intensity image of the point cloud. The colorizing further includes generating an occlusion mask that identifies the selected portion in the reflectance image. The colorizing further includes estimating, using a trained machine learning model, a color for each of the one or more points in the selected portion based on the reflectance image, the occlusion mask, and the 2D image. The 3D scene is updated by using the estimated colors from the trained machine learning model to colorize the selected portion.