Industrial Machine 3D Tracking From 2D Images Without LiDAR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D object detection methods for industrial machines rely on expensive 3D LiDAR sensors and require 3D bounding box annotations, which are difficult to obtain, making them impractical for robots without such sensors.
Innovation Solution
A method for tracking industrial machine movements using 2D images, generating bounding boxes, and projecting a 3D model based on these boxes, optimizing poses using 2D projections, and employing neural networks trained on 2D data to estimate orientation and dimensions, allowing for reliable 3D tracking without 3D sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D LiDAR sensors are used for 3D object detection, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent creates a 3D model copy of the industrial machine based on 2D image data and bounding box information. Instead of directly measuring the object in 3D space with LiDAR, the system generates a virtual 3D representation by projecting 2D bounding boxes into 3D space and optimizing the pose parameters, thereby achieving 3D detection capability using only 2D cameras
Solution Approach 2:
The patent replaces the mechanical 3D LiDAR sensing system with an optical 2D imaging system combined with computational algorithms. By substituting the physical 3D measurement mechanism with 2D image processing and 3D model optimization, the system eliminates the need for expensive 3D sensors while maintaining detection capability
2Measurement precision
If 3D bounding box annotations are used for training, then training data accuracy is improved, but ease of manufacture deteriorates due to annotation difficulty
Solution Approach 1:
The patent transitions the annotation task from 3D space to 2D space. Instead of requiring annotators to draw complex 3D bounding boxes in three-dimensional space, the system only requires 2D bounding box annotations on image planes. This dimensional reduction makes annotation significantly easier while the 3D model optimization process recovers the full 3D pose information from these simplified 2D annotations
3Measurement precision
If 3D point cloud data is used for tracking, then measurement precision is improved, but use of energy increases due to higher computational requirements
Solution Approach 1:
The patent extracts only the essential 2D projection information from the 3D model and uses this extracted data for tracking comparisons. Instead of processing complete 3D point cloud data which requires intensive computation, the system extracts 2D bounding box projections and compares these with detected 2D bounding boxes, significantly reducing computational energy while maintaining tracking accuracy
Data Source
AI summary
A computer-implemented method for tracking movements of an industrial machine includes, for each image of a sequence of two-dimensional images that captures the industrial machine: generating a first bounding box and a second bounding box, generating a three-dimensional model based on the first bounding box and the second bounding box, projecting the three-dimensional model on the image resulting in a two-dimensional projection, and optimizing pose of the three-dimensional model based on the two-dimensional projection. The first and second bounding boxes identifies a first and second component of the industrial machine, respectively. The three-dimensional model includes a first geometric shape and a second geometric shape representing the first component and the second component, respectively. The method further includes tracking movements of the industrial machine over time based on the optimized poses of the three-dimensional model for each image of the sequence.