Scalable 3D Object Recognition with Depth-Based Bounding Boxes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems face challenges in accurately recognizing and generating realistic interactions between virtual and real objects due to limitations in 2D object recognition methods, leading to incomplete and noisy representations of the physical world.
Innovation Solution
A scalable 3D object recognition system that integrates depth sensors, image sensors, and head pose tracking to construct a more complete and less noisy representation of the physical world, using cloud computing for efficient processing of sensor data and generating 3D bounding boxes for objects through a combination of 2D and 3D object recognition algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If 2D bounding box generation system is used for object recognition, then the system complexity is low, but the measurement precision and representation completeness of physical world objects deteriorate
Solution Approach 1:
The patent transitions from 2D bounding box generation to 3D bounding box generation by incorporating depth information from depth sensors. The system processes 2D images and combines them with depth data to create three-dimensional representations of objects, enabling more accurate spatial understanding and object recognition in augmented reality environments.
Solution Approach 2:
The system combines multiple data sources including 2D images, depth information, and sensor data to create a composite representation of the physical world. This multi-modal data fusion approach integrates different types of information to achieve more precise and complete object models than any single data source could provide alone.
2Productivity
If 2D object recognition system is used, then the processing speed is fast, but the representation completeness and accuracy of physical world deteriorate
Solution Approach 1:
The system adds the depth dimension to traditional 2D object recognition by integrating depth sensor data. This enables the generation of 3D bounding boxes that preserve spatial information and object geometry while maintaining processing efficiency through optimized pipelines that handle both 2D and depth data concurrently.
Solution Approach 2:
The patent divides the object recognition process into separate modules: 2D image processing, depth information processing, and 3D bounding box generation. This segmentation allows each module to be optimized independently, maintaining processing speed while accumulating complete 3D representations through the combination of processed results.
3Measurement precision
If cloud computing environment is used for 3D object recognition, then the measurement precision improves, but the loss of time in data transmission increases
Solution Approach 1:
The system performs preliminary processing of 2D images and depth data locally on the device before transmitting to the cloud. This pre-processing reduces the amount of data that needs to be transmitted and allows the cloud computing environment to focus on the computationally intensive 3D bounding box generation, thereby reducing overall transmission time while maintaining precision.
Solution Approach 2:
The patent segments the computation workload between local device and cloud environment. Local processing handles data acquisition and preliminary processing, while cloud computing handles the complex 3D reconstruction algorithms. This distribution optimizes the balance between precision and transmission time by placing each task in the most appropriate computational environment.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.