3D Object Tracking via 2D Camera Correction

The patent combines three-dimensional imaging data from lidar with two-dimensional imaging data from cameras to create a complementary sensing system. The 3D data provides depth and spatial positioning while the 2D camera data captures visual information including objects with non-reflective surfaces that lidar may miss. By merging these data sources through registration and fusion processes, the system achieves both accurate depth measurement and reliable detection of all object types regardless of surface reflectivity.

The patent implements a feedback mechanism where two-dimensional imaging data is used to verify and correct three-dimensional object data. The 2D camera images provide visual feedback on object geometry, dimensions, and boundaries that are used to refine the 3D bounding boxes and object representations. This feedback loop compensates for clipping and merging errors in the 3D imaging by comparing against the detailed visual information from the 2D images and adjusting the 3D object parameters accordingly.

The patent introduces an intermediary registration process that aligns the coordinate systems and temporal synchronization of multiple imaging sensors before data fusion. This intermediary step creates a unified reference framework that simplifies the subsequent fusion operations by pre-resolving coordinate transformations and timing mismatches. The registration module acts as a mediator that translates data from different sensor modalities into a common representation, reducing the computational complexity of the fusion process while maintaining high detection accuracy.