Multimodal Sensor Fusion for Driver Assistance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current driver assistance systems are vulnerable to unexpected objects on the road and lack accuracy in judging split-second decisions, particularly for elderly or disabled drivers, due to heavy dependence on visible lane markers and limited effectiveness in varying conditions such as open sunlight or moving objects.
Innovation Solution
A multimodal sensor fusion system that combines video, audio, and movement-related data from external and internal sensors to provide real-time driver assistance, using microphones, cameras, accelerometers, and gyroscopes to generate alerts for safe driving actions like lane changes, independent of road conditions or lighting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If visible lane markers and computer vision systems are used for detecting road geometry, then the system can provide lane departure warnings and collision avoidance, but the system becomes vulnerable to unexpected objects on the road and fails in varying conditions such as open sunlight
Solution Approach 1:
The patent combines multiple sensing modalities (computer vision systems, GPS systems, digital maps, ultrasonic sensors, radar, and LIDAR) into a unified sensor fusion system. This merging of different sensing approaches allows the system to overcome the limitations of individual sensors, providing reliable detection across varying conditions including open sunlight and unexpected objects on the road.
Solution Approach 2:
The sensor fusion system is designed to perform multiple functions simultaneously: detecting lane markers, identifying unexpected objects, determining vehicle position, and providing collision avoidance warnings. The system adapts to different road conditions and lighting scenarios through its multi-functional capability, making it universally applicable across diverse driving environments.
2Loss of information
If heavy dependence on maps and GPS systems is used for knowledge of road geometry, then the system can provide lane departure warnings, but the system becomes vulnerable to unexpected objects on the streets
Solution Approach 1:
The system uses GPS and digital maps to pre-establish knowledge of road geometry and expected lane positions before the vehicle encounters actual road conditions. This preliminary information serves as a reference framework that the real-time sensors can compare against, enabling the system to detect both expected road features and unexpected objects that deviate from the pre-known geometry.
Solution Approach 2:
The patent integrates GPS map data with real-time sensor inputs from computer vision systems and other detectors. By merging pre-known road geometry information with live environmental sensing, the system can reliably identify unexpected objects that do not match the expected map data, thereby maintaining detection reliability even when objects are not present in the digital maps.
3Measurement precision
If Time-of-Flight cameras or 3D sensors are used for sensing traffic, then the system can provide obstacle detection, but the accuracy is insufficient for open sunlight or moving objects
Solution Approach 1:
The patent combines Time-of-Flight camera data with inputs from other sensing modalities including computer vision systems, radar, and LIDAR. This sensor fusion approach compensates for the deficiencies of individual sensors in open sunlight conditions, as different sensors have different strengths and weaknesses that complement each other when integrated together.
Solution Approach 2:
The system uses multiple sensors as intermediary detection layers, where each sensor type acts as a mediator that can detect objects under specific conditions. When one sensor type struggles (e.g., Time-of-Flight cameras in open sunlight), other sensors serve as intermediaries to maintain detection capability, ensuring continuous and accurate obstacle detection across varying conditions.
Data Source
AI summary
An apparatus and method of providing driver assistance for performing a driving action include obtaining video information adjacent to a vehicle, obtaining audio information adjacent to the vehicle, optionally obtaining movement-related information, and generating a driver assist indication based on a combination of the video information and the audio information, and, optionally, the movement-related information.