3D Object Retrieval from 2D Images via Deep Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for 3D object retrieval from 2D images face challenges such as sensitivity to image quality, requirement for specific constraints, inability to handle partially occluded objects, and scalability limitations, often relying on heavy annotations and calculations.
Innovation Solution
A computer-implemented method using a deep neural network trained on both rendered and natural 2D images, which provides a similarity metric for retrieving 3D objects from 2D images without specific data constraints, improving scalability and discriminative power by leveraging end-to-end learning from raw data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If shape-based approaches with silhouette analysis are used for 3D object retrieval from 2D images, then the method can operate with simple 2D input, but the performance is sensitive to the quality of automatic contour detection which remains a great challenge
Solution Approach 1:
The patent introduces an intermediary step of generating multiple candidate 3D models from the 2D image, then selecting the best match. This mediator approach (candidate generation and selection) resolves the contradiction by not relying directly on perfect contour detection, but rather using the 2D image to generate and evaluate multiple 3D hypotheses, thereby reducing sensitivity to contour quality while maintaining ease of 2D input operation
2Ease of operation
If composite feature vectors combining Zernike moments and HOG features are used, then the approach can retrieve objects from single 2D images, but the features are insufficient to distinguish between similar objects with the same overall shape and fail with partially occluded objects
Solution Approach 1:
The patent transitions from 2D feature extraction to 3D model-based comparison. By generating 3D candidate models and comparing them against the query image through 3D rendering and matching, the system adds the third dimension to the analysis. This resolves the contradiction by providing sufficient discriminative power for similar objects and handling occlusions through 3D geometric reasoning, while still accepting simple 2D image input
3Measurement precision
If 3D exemplar models with heavy annotations and calculations are used for object detection and pose estimation, then fine pose estimation can be achieved, but the methods require heavy annotations and calculations making them not scalable
Solution Approach 1:
The patent performs preliminary action by pre-generating multiple candidate 3D models that could potentially match the query image. These candidate models are prepared in advance with their associated 3D representations and rendering parameters. This preliminary preparation resolves the contradiction by enabling accurate pose estimation through pre-computed 3D candidates without requiring heavy runtime annotations or calculations, thus improving scalability while maintaining precision
4Extent of automation
If dense pixel-level correspondences are established between natural images and rendered images for 3D reconstruction, then objects can be reconstructed from single views, but the method is sensitive to the quality of segmentation and thus could fail with images with complex backgrounds and partially occluded objects
Solution Approach 1:
The patent inverts the traditional approach by not trying to segment the 2D image to reconstruct 3D, but rather generating 3D candidate models and rendering them to compare with the 2D image. This inversion resolves the contradiction by eliminating the need for segmentation entirely, making the method reliable with complex backgrounds and occlusions while maintaining automatic 3D reconstruction capability through 3D-2D matching
Data Source
Figure 1
Figure 2~4
Figure 5~6
AI summary
The invention notably relates to a computer-implemented method for recognizing a three-dimensional modeled object from a two-dimensional image. The method comprises providing a first set of two-dimensional images rendered from three-dimensional modeled objects, each two-dimensional image of the first set being associated to a label; providing a second set of two-dimensional images not rendered from three-dimensional objects, each two-dimensional image of the second set being associated to a label; training a model on both first and second sets; providing a similarity metric; submitting a two-dimensional image depicting at least one object; and retrieving a three-dimensional object similar to the said at least one object of the two-dimensional image submitted by using the trained model and the similarity metric.