2D-3D Image Labeling via Point Cloud Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image labeling technologies are inadequate for 3D environments, as they struggle to consistently label objects across multiple viewpoints and fail to effectively represent 3D shapes in 2D images, especially when surface color attributes are involved.
Innovation Solution
A spatial analysis system that combines 2D and 3D data processing, using labeling, association, weighting, and propagation logic to assign labels to objects in 2D data and propagate them to 3D data, leveraging surface information from 3D meshes to enhance labeling accuracy and consistency across different views.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If 2D image labeling is used to label objects in 3D environments, then labeling can be performed on individual images, but label consistency across multiple viewpoints deteriorates
Solution Approach 1:
The patent transitions from 2D image labeling to 3D point cloud labeling by introducing depth information. Instead of labeling objects in separate 2D images from different viewpoints, the system creates a unified 3D representation where objects are labeled once in three-dimensional space, automatically providing consistent labels across all viewpoints that would capture the object.
Solution Approach 2:
The patent merges multiple 2D image data sets into a single 3D point cloud representation. By combining information from multiple viewpoints and integrating depth data, the system consolidates what would otherwise be separate labeling tasks into one unified 3D labeling process, ensuring consistency across all original viewpoints.
2Area of stationary object
If multiple 2D images are used to cover entire rooms or spaces, then complete coverage is achieved, but the amount of data to be labeled increases substantially
Solution Approach 1:
The patent uses 3D spatial representation to efficiently cover large areas. Instead of requiring multiple overlapping 2D images to map an entire room, the system creates a single 3D point cloud that inherently represents the full three-dimensional space, reducing the number of data sets needed while maintaining complete coverage.
Solution Approach 2:
The patent combines multiple data types (color information from 2D images and depth information from 3D sensors) into a composite 3D point cloud representation. This composite structure efficiently encodes both spatial layout and visual appearance in a unified format, reducing overall data volume while preserving all necessary information.
3Ease of manufacture
If 2D attributes are applied to represent 3D shapes, then simple labeling is possible, but accurate representation of 3D geometry deteriorates
Solution Approach 1:
The patent applies labels directly to 3D points in point cloud data rather than to 2D image pixels. This allows the labeling system to work natively in three-dimensional space, preserving geometric accuracy while maintaining labeling simplicity. Objects are defined by their 3D coordinates and spatial relationships rather than by projecting them onto 2D surfaces.
Data Source
AI summary
2D and 3D data of a scene are linked by associating points in the 3D data with corresponding points in multiple different 2D images within the 2D data. Labels assigned to points in either data can be propagated to the other data. Labels propagated to a point in the 3D data are aggregated, and the labels ranked highest are kept and propagated back to the 2D images. 3D data including labels produced in this manner allow partially obscured objects in certain views to be more accurately identified. Thus, an object can be manipulated in all 2D views of the 2D data in which the object is at least partially visible, in order to digitally remove, alter, or replace it.