3D Object Detection With Pose and Size Estimation on Mobile
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision systems face challenges in performing efficient 3D object detection and pose estimation from a single image without prior knowledge, particularly on mobile devices where real-time processing with limited resources is required, and there is a lack of adequate training data for 9 degrees of freedom detection.
Innovation Solution
A multi-task machine-learned model with a multi-headed architecture is trained using synthetic data generated by augmented reality engines, capable of determining object properties such as centroid heat maps, distance fields, and segmentation masks, and utilizes camera intrinsics and AR data to estimate 3D bounding boxes in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a machine-learned model performs 3D object detection and pose estimation on mobile devices, then real-time processing capability is achieved, but device resource limitations (computational power, memory) are exceeded
Solution Approach 1:
The model is divided into multiple specialized heads (centroid heat map head, distance field head, segmentation mask head) that each handle specific detection tasks. This segmentation allows the model to process different aspects of object detection separately, improving efficiency and reducing overall computational resource consumption on mobile devices.
Solution Approach 2:
The machine-learned model performs multiple functions simultaneously including 3D bounding box detection, pose estimation, and size estimation from a single image input. This multi-functionality consolidates what would otherwise require multiple separate models, reducing total computational overhead and enabling real-time processing on resource-constrained mobile devices.
2Adaptability or versatility
If 3D object detection is performed without prior knowledge of objects, then model adaptability is improved, but detection precision and measurement accuracy deteriorate
Solution Approach 1:
The model introduces intermediate representations including centroid heat maps, distance fields, and segmentation masks as mediators between the raw image input and the final 3D pose/size estimates. These intermediates provide structured information that improves measurement precision while maintaining the ability to detect objects without prior knowledge.
Solution Approach 2:
The model transitions from 2D image input to 3D output by estimating depth, pose, and size dimensions. This dimensional transformation enables accurate 3D measurement estimation without requiring prior knowledge, as the model learns to infer three-dimensional properties directly from two-dimensional images.
3Loss of time
If training data is limited for 9 degrees of freedom detection, then data acquisition time is reduced, but model training precision and reliability worsen
Solution Approach 1:
The model performs preliminary actions by generating intermediate representations (centroid heat maps, distance fields, segmentation masks) during training that capture essential object properties. These preliminary outputs serve as robust training signals that improve model reliability even when labeled training data is limited, as they provide multiple supervisory cues from a single training example.
Data Source
AI summary
The present disclosure is directed to systems and methods for performing object detection and pose estimation in 3D from 2D images. Object detection can be performed by a machine-learned model configured to determine various object properties. Implementations according to the disclosure can use these properties to estimate object pose and size.