Graph Convolution Keypoint Matching for 3D-2D Localization

The patent creates a 3D map representation (virtual copy) of the environment from pre-built data and matches it with current 2D camera images. This copying approach allows the system to perform 3D scene reconstruction using only a monocular camera, eliminating the need for expensive LIDAR sensors while maintaining matching accuracy through graph convolutional network-based keypoint correspondence.

The patent replaces the mechanical/optical LIDAR sensing system with a computational vision system using a monocular camera combined with graph convolutional networks. The mechanical keypoint extraction and matching process is substituted with deep learning-based feature detection and graph neural network matching, reducing hardware complexity while improving robustness to adverse conditions.

By creating and matching against a pre-built 3D map copy rather than using active LIDAR scanning, the system eliminates the high energy consumption associated with LIDAR operation. The graph convolutional network processes 2D image data computationally, which consumes significantly less energy than active optical sensing.

The patent uses a simple monocular camera (cheaper and lower energy consumption) instead of expensive, high-energy LIDAR sensors. The system compensates for the camera's limitations through sophisticated algorithms (graph convolutions) that extract maximum information from passive light capture.

The patent extracts and removes the problematic descriptor-based matching component from the system. Instead of relying on appearance-based descriptors that fail under occlusion and dynamic changes, the system extracts geometric keypoint locations and uses graph convolutional networks to establish correspondences based on spatial relationships and geometric constraints, which are invariant to appearance changes.

The patent transitions from 2D descriptor space matching to 3D geometric space matching by lifting 2D keypoints into 3D using the pre-built map, then performing matching in this elevated dimensional space where geometric relationships provide more reliable correspondence information that is robust to appearance variations, occlusions, and dynamic object changes.