Cooperative Pseudo Point Cloud Stitching for Wider Vehicle 3D Sensing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth estimation and pseudo point cloud models for automotive applications have limitations such as limited field-of-view, reconstruction range, and accuracy, as well as high implementation costs, which hinder effective 3D sensing and reconstruction in autonomous driving systems.
Innovation Solution
A system and method utilizing a homography transformation learning network with a spatial self-attention mechanism to estimate relative poses of cameras between vehicles, enabling the transformation and stitching of pseudo point clouds from a transmitter vehicle to a receiver vehicle, thereby generating a dense pseudo point cloud with a wider field-of-view without the need for expensive LiDAR sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LiDAR sensors are used for 3D sensing and reconstruction, then measurement precision and reconstruction accuracy are improved, but implementation cost increases significantly
Solution Approach 1:
The patent uses camera images as a cheap copy alternative to LiDAR depth data. By training a depth estimation model to generate depth maps from 2D images, the system replicates the 3D sensing capability of LiDAR using inexpensive camera hardware, thereby reducing implementation cost while maintaining acceptable reconstruction accuracy
Solution Approach 2:
The patent replaces the mechanical LiDAR sensing system with a computational photography approach. Instead of using physical LiDAR sensors to directly measure depth, the system uses camera images processed through deep learning models to estimate depth, substituting a mechanical sensing system with an optical-computational system that is cheaper and more scalable
2Ease of manufacture
If a single camera is used for depth estimation, then implementation cost is reduced, but field-of-view and reconstruction range are limited
Solution Approach 1:
The patent merges data from multiple sources: images and depth maps from both the local vehicle and nearby connected vehicles are combined. By stitching these overlapping fields-of-view together using the estimated homography, the system creates a composite pseudo point cloud that covers a much larger area than any single camera could capture alone
Solution Approach 2:
The patent extends the 3D reconstruction from a single-vehicle perspective to a multi-vehicle collaborative space. By incorporating data from neighboring vehicles and transforming it into the local vehicle's coordinate system, the system effectively adds a spatial dimension to the field-of-view, covering areas that would be invisible to a single camera
3Area of stationary object
If depth maps from multiple vehicles are stitched together, then field-of-view and reconstruction completeness are improved, but processing complexity increases
Solution Approach 1:
The patent performs preliminary depth estimation for each vehicle independently before stitching. Each vehicle's depth estimation model processes its own images to generate depth maps in advance, and the homography transformation is pre-computed from detected overlapping features. This preliminary processing organizes the data structure before combination, making the subsequent stitching operation more manageable and efficient
Solution Approach 2:
The patent uses detected overlapping features and estimated homography matrices as intermediaries to bridge different vehicle coordinate systems. These intermediaries provide a mathematical transformation framework that automatically handles the complexity of coordinate alignment, allowing depth maps from different vehicles to be stitched together systematically rather than through ad-hoc processing
Data Source
AI summary
System, methods, and embodiments described herein relate to dynamically generating a wide field-of-view three-dimensional pseudo point cloud of an environment around a vehicle. A disclosed method may include capturing, via a camera, a first view in a first image, determining a first depth map based on the first image, obtaining, from an external system, a second image of a second view that overlaps the first view and a second depth map based on the second image, inputting the first image and the second image into a self-supervised homograph network that is trained to output a homographic transformation matrix between the first image and the second image, and generating a three-dimensional pseudo point cloud that combines the first depth map and the second depth map based on the homographic transformation matrix.