Audio-Visual Fusion for Emergency Vehicle Detection in Autonomous Driving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous driving vehicles face challenges in accurately detecting emergency vehicles using single-source sensor data, leading to potential unnecessary stops or unsafe maneuvers.
Innovation Solution
The fusion of audio and visual data from multiple sources is achieved through machine learning models that extract feature vectors from both modalities and concatenate them for input into a multi-layer perceptron network, enabling more accurate detection of emergency vehicles and determining their position and direction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If single-source sensor data is used for emergency vehicle detection, then device complexity is reduced, but detection accuracy deteriorates leading to unnecessary stops or unsafe maneuvers
Solution Approach 1:
The patent combines audio and visual sensor data into a unified detection system. The audio capturing device detects sirens and the image capturing device detects emergency vehicle lights, merging these multiple information sources to improve detection accuracy and eliminate false alarms that would occur with single-source detection
Solution Approach 2:
The detection system is designed to handle multiple types of emergency vehicle signals simultaneously - both audio signals (sirens) and visual signals (emergency lights) - through a single integrated machine learning model that can process and fuse different modalities of sensor data
2Measurement precision
If audio and visual data fusion is implemented, then detection accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent introduces a machine learning model as an intermediary that automatically fuses audio and visual features. This intermediary component handles the complex task of combining and analyzing multiple data streams, reducing the overall system complexity while maintaining high detection accuracy
Solution Approach 2:
The patent replaces traditional rule-based or manual signal processing methods with a machine learning-based approach. This substitution enables automated feature extraction and fusion from audio and visual data, simplifying the processing architecture while improving detection performance
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
According to various embodiments, systems, methods, and mediums for operating an autonomous driving vehicles (ADV) are described. The embodiments use a number of machine learning models to extract features individually from audio data and visual data captured by sensors mounted on the ADV, and then to fuse these extracted features to create a concatenated feature vectors. The concatenated feature vector is provided to a multiplayer perceptron (MLP) as input to generate a detection result related to the presence of an emergency vehicle in the surrounding environment. The detection result can be used by the ADV to take appropriate actions to comply with the local traffic rules.