Semantic Free-Space Mapping With Camera-LiDAR Data Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining a drivable region in front of a vehicle using a single camera or sensor face challenges in accurately estimating depth and object type due to varying weather conditions and air transparency, which affects navigation reliability.
Innovation Solution
A method combining a camera and a sensor, such as LIDAR, to capture two-dimensional visual images and three-dimensional distance data, using neural networks for bounding box classification and semantic segmentation, transforming data into a bird's-eye view to determine the drivable space with object types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single camera or sensor is used to determine the drivable region, then the device complexity is reduced, but the measurement precision of depth and object type deteriorates due to varying weather conditions and air transparency
Solution Approach 1:
The patent combines data from multiple sensors (camera, LIDAR, radar) to determine the drivable region. The camera provides visual information about object types and semantics, while LIDAR and radar provide accurate depth and distance measurements. By merging these complementary data sources, the system achieves both low complexity and high measurement precision, overcoming the limitations of using a single sensor type.
2Device complexity
If a single camera or sensor is used to determine the drivable region, then the device complexity is reduced, but the reliability of navigation decisions deteriorates
Solution Approach 1:
The system merges data from multiple sensor types (camera, LIDAR, radar) to improve navigation reliability. The camera identifies object types and semantic information, while LIDAR and radar provide reliable depth measurements independent of weather conditions. This multi-sensor approach ensures that navigation decisions are based on comprehensive and reliable information, overcoming the unreliability of single-sensor systems in varying environmental conditions.
3Productivity
If deep neural networks are used for segmentation techniques, then the productivity of determining the drivable region is improved, but the measurement precision of object type and depth deteriorates
Solution Approach 1:
The patent combines deep neural network processing with multi-sensor data fusion to maintain both high productivity and measurement precision. The neural networks efficiently process visual data from the camera for semantic segmentation and object classification, while simultaneous LIDAR and radar data provide accurate depth and distance measurements. This parallel processing approach ensures that both semantic accuracy and geometric precision are achieved without compromising productivity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Provides accurate and reliable navigation decisions by determining the drivable space and object types, enhancing the precision and reliability of autonomous driving systems.
Implementation Method 1
Via a sensor, three-dimensional distance data is captured with respect to the vehicle for objects in the environment of the vehicle
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for determining a semantic free space in an environment of a vehicle comprises capturing a two dimensional visual image from the environment of the vehicle via a camera, and determining a limitation of a free space within the visual image. Via a sensor, distance data of objects are captured and assigned to the visual image, and the limitation of the free space is transferred to a bird's-eye view based on the assigned distance data. For objects identified in the visual image a respective bounding box and a respective classification are determined. Objects limiting the free space are selected, and their bounding box is assigned to the limitation of the free space in the bird's-eye view. Finally, segments of the limitation of the free space are classified according to the classification of each bounding box of the selected objects.