Joint 2D-3D Object Tracking for Noisy 3D Position Estimates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object tracking solutions for autonomous systems primarily focus on 3D tracking, leading to inaccuracies and object misses due to the lack of integration with 2D image features, especially when DNNs provide noisy 3D position estimates.
Innovation Solution
A joint 2D and 3D object tracking approach that simultaneously tracks objects in both 2D image space and 3D world space, using a deep neural network object detector to enhance accuracy by incorporating both 2D and 3D features for object state representation, prediction, association, and update operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D-centric object tracking is used, then object tracking in world space is achieved, but tracking accuracy deteriorates due to noisy DNN 3D position estimates and lack of 2D image features
Solution Approach 1:
The patent introduces 2D image space tracking as an additional dimension to complement 3D world space tracking. By maintaining tracked states in both 2D bounding box coordinates and 3D world coordinates, the system leverages the accuracy of 2D DNN detections while preserving 3D spatial information, thereby improving overall tracking accuracy and robustness against noisy 3D estimates.
Solution Approach 2:
The patent merges 2D and 3D tracking frameworks into a unified joint tracking system. The association metric combines 2D IoU (Intersection over Union) and 3D distance metrics with equal weighting, allowing the system to simultaneously utilize 2D image features and 3D spatial information. This combination resolves the contradiction by making tracking accuracy dependent on both 2D and 3D measurements rather than relying solely on noisy 3D estimates.
2Productivity
If only 3D tracking is performed, then computational efficiency is maintained, but velocity estimation accuracy deteriorates due to lack of initial velocity from DNN
Solution Approach 1:
The patent performs preliminary velocity estimation using 2D DNN detections before the main tracking update. The DNN provides initial velocity estimates in 2D space, which are then transformed and integrated into the 3D velocity calculation. This preliminary action allows the system to leverage pre-computed DNN velocity information, improving velocity estimation accuracy without significantly increasing computational burden.
3Ease of operation
If 3D position jumps occur from DNN estimates, then tracking updates are performed, but object tracking fails due to sudden position discontinuities
Solution Approach 1:
The patent introduces 2D tracked state as an intermediary between consecutive 3D position estimates. When a new 3D detection arrives, the system first updates the 2D tracked state using 2D DNN detections and association metrics. This 2D intermediate representation smooths out sudden 3D position jumps because 2D DNN detections are generally more stable. The smoothed 2D state then guides the 3D position update, preventing tracking failures due to discontinuities.
Data Source
AI summary
In various examples, techniques for multi-dimensional tracking of objects using two-dimensional (2D) sensor data are described. Systems and methods may use first image data to determine a first 2D detected location and a first three-dimensional (3D) detected location of an object. The systems and methods may then determine a 2D estimated location using the first 2D detected location and a 3D estimated location using the first 3D detected location. The systems and methods may use second image data to determine a second 2D detected location and a second 3D detected location of a detected object, and may then determine that the object corresponds to the detected object using the 2D estimated location, the 3D estimated location, the second 2D detected location, and the second 3D detected location. The systems and method then generate, modify, delete, or otherwise update an object track that includes 2D state information and 3D state information.