3D View Synthesis Model for RGB Assignment Beyond Camera FOV
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to assign RGB values outside the field angle range in images, particularly at night when color information is lacking, and moving objects interfere with stationary object identification, especially in urban mapping using passive sensors like visible light cameras and active sensors like LIDAR.
Innovation Solution
A learning device and method that utilizes three-dimensional coordinate values, line-of-sight direction, and point cloud data to learn a model for generating images with RGB assignment from arbitrary viewpoints, incorporating point cloud data to enhance color estimation beyond the field angle range.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If simple superimposition is used to assign RGB values to point cloud data, then the process is simple and fast, but RGB values cannot be assigned outside the field angle range and moving objects are transferred to stationary object locations
Solution Approach 1:
The patent segments the image processing by separating stationary objects from moving objects through temporal comparison of multiple images. By identifying pixels that change between frames, the system isolates moving objects and excludes them from the RGB assignment process, preventing them from being incorrectly transferred to stationary object locations.
Solution Approach 2:
The patent performs preliminary actions by pre-processing images to identify and remove moving objects before assigning RGB values. The system compares multiple images in advance, determines which pixels correspond to moving objects, and creates a mask to exclude these regions from subsequent RGB assignment operations.
2Reliability
If data is acquired at night when few moving objects appear, then moving object interference is reduced, but color information cannot be acquired by passive sensors due to lack of sunlight irradiation
Solution Approach 1:
The patent merges data from multiple sources and time points. It combines point cloud data with RGB information from daytime images, while using temporal comparison to filter out moving objects. The system integrates shape information from LIDAR with color information from cameras, creating a comprehensive representation that works effectively at night.
Solution Approach 2:
The patent creates a digital copy of the scene's geometric structure through point cloud data, which can then be overlaid with RGB information from separate daytime images. This allows the system to assign colors to stationary objects based on their 3D position in the point cloud, matching them to corresponding pixels in the daytime images.
3Productivity
If LIDAR is used to acquire shape information at night, then shape information can be efficiently acquired, but color information in wavelengths other than laser wavelength cannot be acquired
Solution Approach 1:
The patent implements a multi-functional system that uses LIDAR for shape acquisition and separate daytime camera images for color information. The point cloud data from LIDAR serves as a universal geometric framework that can be combined with RGB information from different sources and times, making the system versatile for both night and daytime operations.
Data Source
AI summary
Provided is a learning device 10 including: an acquisition unit 101 that acquires three-dimensional coordinate values, information on a line-of-sight direction, and point cloud data as input data and images captured from a plurality of directions as teacher data; and a learning unit 102 that learns a model for outputting an image from a designated line-of-sight direction by outputting a color and a density for each pixel using the input data and the teacher data.