Box Embeddings for Asymmetric Image Overlap
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for determining visual overlap of images are costly and inefficient, particularly when evaluating a query image against a large gallery, as they require local feature matching and geometric verification for 3D re-localization and reconstruction applications.
Innovation Solution
An image matching system that encodes images into box embeddings and estimates asymmetric surface overlaps using a convolutional neural network, calculating an asymmetric overlap factor by determining the intersection and volume of boxes representing the images in a multi-dimensional embedding space, thereby reducing the need for feature matching and geometric verification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional local feature matching and geometric verification methods are used to determine visual overlap of images, then measurement precision of surface overlap is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent creates a simplified copy of the image data in the form of box embeddings that capture the essential spatial information needed for overlap determination. Instead of performing complex feature matching on the original images, the system uses these compact box representations to estimate surface overlap, dramatically reducing computational time while maintaining acceptable precision for the application.
Solution Approach 2:
The box embeddings serve as disposable, computationally inexpensive representations that can be quickly generated and discarded after use. These simple bounding box structures require minimal processing resources compared to conventional feature matching methods, enabling rapid evaluation of many image pairs without the high computational cost of traditional approaches.
2Reliability
If conventional local feature matching and geometric verification are performed for every pair of images in a large gallery, then reliability of 3D re-localization is improved, but device complexity and computational resources required increase
Solution Approach 1:
The patent extracts only the essential spatial information from images by representing them as box embeddings in a multi-dimensional space. This extraction process removes unnecessary computational complexity while retaining the key geometric relationships needed for reliable 3D re-localization and reconstruction tasks.
Solution Approach 2:
The system transforms image data from the original pixel domain into a different parameter space (box embeddings in multi-dimensional space). This parameter transformation simplifies the comparison operation from complex feature matching to straightforward geometric intersection calculations, reducing system complexity while maintaining reliability.
3Measurement precision
If exhaustive image evaluation against a large gallery is performed using conventional methods, then completeness of image matching results is improved, but productivity and processing efficiency decrease
Solution Approach 1:
By using box embeddings as simplified copies of image spatial information, the system enables rapid evaluation of many image pairs. This copying approach maintains the ability to comprehensively evaluate the query image against the entire gallery while dramatically increasing processing throughput compared to conventional methods.
Data Source
AI summary
An image matching system for determining visual overlaps between images by using box embeddings is described herein. The system receives two images depicting a 3D surface with different camera poses. The system inputs the images (or a crop of each image) into a machine learning model that outputs a box encoding for the first image and a box encoding for the second image. A box encoding includes parameters defining a box in an embedding space. Then the system determines an asymmetric overlap factor that measures asymmetric surface overlaps between the first image and the second image based on the box encodings. The asymmetric overlap factor includes an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.