3D Object Tracking via 2D Semantic Keypoints in Occlusions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autonomous driving systems rely on expensive 3D bounding box annotations for object tracking, which lead to unstable tracking in occlusions and unusual vehicle sizes, and introduce re-projection and calibration issues when using LIDAR point clouds.
Innovation Solution
The method employs semi-supervised 3D object tracking using 2D semantic keypoints, leveraging sparse annotations and multi-view consistencies to propagate keypoint labels across frames, and uses a shared image backbone with a differentiable keypoint-flow network to lift 2D measurements to 3D, enabling robust tracking even in occlusions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 3D bounding box annotations are used for object tracking, then tracking coverage is comprehensive, but annotation cost is expensive and tracking stability deteriorates in occlusions
Solution Approach 1:
The patent uses 2D semantic keypoints as a simplified copy or projection of 3D object characteristics. Instead of annotating full 3D bounding boxes, the system extracts and tracks distinctive 2D keypoint positions and descriptors from images, which can be easily copied and matched across frames. This reduces annotation complexity while maintaining tracking reliability through the use of descriptor-based matching that is invariant to scale and viewpoint changes.
Solution Approach 2:
The patent extracts only the essential features needed for tracking - specifically 2D semantic keypoints and their descriptors - from the complete 3D object information. By taking out only the critical 2D projection elements rather than using full 3D annotations, the system reduces annotation cost and complexity while preserving the ability to track objects reliably through occlusions and viewpoint changes.
2Measurement precision
If LIDAR point clouds are used for object tracking, then 3D structure information is obtained, but re-projection and calibration issues arise
Solution Approach 1:
The patent replaces the mechanical and calibration-intensive LIDAR point cloud system with a purely image-based 2D keypoint detection and matching system. Instead of relying on 3D-to-2D re-projection geometry and sensor calibration, the system uses 2D descriptor matching that is inherently invariant to viewpoint and scale, eliminating the need for complex calibration procedures while still providing accurate object tracking information.
Solution Approach 2:
The patent transitions from working in 3D space with LIDAR point clouds to working entirely in 2D image space with semantic keypoints. By changing the dimensional domain from 3D to 2D and using descriptor-based matching, the system avoids re-projection issues and calibration requirements while maintaining the ability to track objects across different viewpoints and scales.
3Adaptability or versatility
If conventional keypoint detection is used, then object features are detected, but invariance to lighting and viewpoint changes is insufficient
Solution Approach 1:
The patent creates a composite representation by combining 2D keypoint positions with descriptor vectors that encode appearance information. This composite structure - merging geometric position data with photometric descriptor data - provides both precision in keypoint location and invariance to lighting and viewpoint changes through the descriptor matching component, which captures object appearance characteristics that remain consistent across different viewing conditions.
Data Source
AI summary
A method for 3D object tracking is described. The method includes inferring first 2D semantic keypoints of a 3D object within a sparsely annotated video stream. The method also includes matching the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame. The method further includes warping the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame. The method also includes labeling a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.