3D-Aware Image Search Using Rotation Loss Feature Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image search models trained on 2D datasets fail to capture 3D information such as object orientation, leading to search results that are not filtered based on 3D properties, requiring users to manually filter results, which is time-consuming and inefficient.
Innovation Solution
A method and system that utilize a machine learning model trained with a rotation loss term to generate feature vectors that include 3D orientation information, allowing for the comparison and selection of images based on similar object orientations, using a multi-task loss function that incorporates object synthesis and rotation loss terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If image search models are trained on 2D datasets, then the training process is simpler and faster, but the search results do not capture 3D information such as object orientation
Solution Approach 1:
The patent transitions from 2D image processing to 3D-aware feature extraction by introducing rotation loss terms and 3D orientation information into the training process. This allows the model to capture depth and orientation data while maintaining compatibility with standard 2D image inputs, effectively adding a dimensional aspect to the feature space without requiring full 3D input data.
Solution Approach 2:
The patent modifies the loss function parameters by adding rotation loss terms (L_rotation) to the standard image synthesis loss. This parameter change enables the model to optimize for both visual fidelity and correct 3D orientation, allowing it to learn rotation-aware features during training without fundamentally changing the network architecture or data requirements.
2Device complexity
If conventional image search models are used, then the implementation is simpler, but users must manually filter search results which is time-consuming
Solution Approach 1:
The patent enables the image search model to automatically filter and rank results based on 3D orientation matching. The model performs self-service by computing rotation-aware feature vectors for both query and database images, then automatically comparing these vectors to retrieve orientation-matched results without requiring manual user intervention for filtering.
Solution Approach 2:
The patent incorporates rotation loss feedback during training, where the model receives feedback on both image synthesis quality and rotation accuracy. This dual feedback mechanism guides the model to learn features that simultaneously capture visual appearance and 3D orientation, enabling automatic orientation-based filtering at query time.
3Measurement precision
If 3D orientation information is incorporated into feature vectors, then search results are more accurate regarding object orientation, but the feature vector computation becomes more complex
Solution Approach 1:
The patent segments the feature vector computation into distinct components: standard image synthesis features and rotation-aware features. By separating these concerns and computing them through different loss terms during training, the model can maintain efficient forward propagation while achieving precise orientation matching through the combined feature representation.
Data Source
AI summary
Systems and methods for performing image search are described. An image search method may include generating a feature vector for each of a plurality of stored images using a machine learning model trained using a rotation loss term, receiving a search query comprising a search image with object having an orientation, generating a query feature vector for the search image using the machine learning model, wherein the query feature vector is based at least in part on the orientation, comparing the query feature vector to the feature vector for each of the plurality of stored images, and selecting at least one stored image of the plurality of stored images based on the comparison, wherein the at least one stored image comprises a similar orientation to the orientation of the object in the search image.