Sensor Fusion via Ray Projection for Parallax Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-collocated sensors in systems like autonomous vehicles cause parallax occlusion effects when fusing image and depth data, leading to incorrect semantic labeling and fusion issues due to differences in data timestamps and sparsity, resulting in inaccurate representation of environments.
Innovation Solution
A controller fuses image and depth data to form a mesh, projects rays to determine occlusion boundary surfaces, and uses optical flow and image warping to account for timestamp differences, while interpolating camera poses and addressing data sparsity by selecting appropriate pixels or points for fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If image data and depth data from non-collocated sensors are fused to form a mesh, then semantic information and geometry are combined, but parallax occlusion effects cause incorrect semantic labeling and fusion inaccuracies
Solution Approach 1:
The system performs preliminary actions by determining occlusion boundary surfaces from depth data before fusing image data with the mesh. By pre-identifying occlusion boundaries and projecting rays to detect intersections, the system prevents incorrect semantic labeling at the source rather than correcting errors after fusion, thus resolving the contradiction between preserving semantic information and maintaining fusion accuracy
Solution Approach 2:
The system introduces an intermediary mechanism using ray projection and occlusion boundary surfaces as a mediator between image data and depth data. The ray projection technique acts as an intermediary test to determine whether image pixels should be fused with mesh points, filtering out parallax occlusion effects before they corrupt the fusion result, thereby maintaining both semantic information integrity and fusion precision
2Duration of action of stationary object
If data from non-collocated sensors are fused at different timestamps, then temporal coverage is improved, but timestamp differences cause misalignment and fusion errors
Solution Approach 1:
The system performs preliminary temporal alignment by associating image data with depth data based on timestamp comparison before fusion. By pre-sorting and matching data points according to their timestamps, the system establishes correct temporal correspondence, preventing misalignment errors while maintaining extended temporal coverage from multitimestamp data
Data Source
AI summary
Systems, methods, controllers, and techniques for addressing the parallax occlusion effect caused by non-collocated sensors are disclosed. A controller is configured to fuse image data received from an imaging device and depth data received from a depth sensor to form a mesh, project a ray from the imaging device to a pixel of the image data fused with a point of the depth data forming the mesh, determine an occlusion boundary surface within the depth data, and in response to determining that the ray intersects the occlusion boundary surface, determine that the imaging device is occluded from a fused point in the mesh.