Driver Behavior Prediction Using Multimodal Neural Network Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automated driving systems face challenges in understanding human driver interactions in complex scenarios, particularly when labeled data is scarce, limiting the effectiveness of learning-based algorithms in recognizing driver behavior.
Innovation Solution
A system that utilizes a processor to generate feature vectors from image sequences and vehicle data signals using neural networks, including CNNs and RNNs, and performs data fusion using LSTM layers to recognize and predict driver behavior, leveraging 3D semantic scene representations and multimodal data fusion to detect tactical driver behaviors without relying on direct supervision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If learning based algorithms are used to search data sequences in association with driving scenarios, then the system can recognize driver behavior, but the algorithms cannot perform optimally when labelled data is not provided
Solution Approach 1:
The system uses unsupervised learning algorithms that can automatically learn from unlabeled video data without requiring manual annotation. The algorithm processes raw driving scenario data and automatically identifies patterns and behaviors, making the system self-sufficient in data labeling and eliminating dependency on extensive labeled datasets
Solution Approach 2:
The patent transforms the learning approach by changing from supervised learning (requiring labeled data) to unsupervised learning (working with unlabeled data). This parameter change in the learning methodology allows the system to maintain high recognition accuracy while working with unlimited unlabeled data instead of scarce labeled data
2Measurement precision
If multiple neural networks and data fusion techniques are used to improve driver behavior recognition, then recognition accuracy is improved, but system complexity increases
Solution Approach 1:
The system segments the driver behavior recognition task into multiple specialized neural networks, each handling specific aspects: one network processes video data, another processes sensor data, and a third handles depth information. This segmentation allows each network to be optimized for its specific function while working together through data fusion to achieve comprehensive behavior recognition
Solution Approach 2:
The patent merges multiple data streams from different sources (video cameras, sensors, depth maps) and multiple neural network outputs into a unified behavior recognition decision. The data fusion technique combines features from various networks and modalities, creating a comprehensive view of driver behavior that improves accuracy while managing complexity through systematic integration
Data Source
AI summary
Driver behavior recognition or driver behavior prediction are described herein. A first image sequence including image frames associated with a forward-facing image capture device of a vehicle and a corresponding vehicle data signal sequence may be received. A second image sequence including image frames associated with a rear or driver facing image capture device of the vehicle may be received. Feature vectors may be generated for respective sequences using neural networks, such as a convolutional neural network (CNN), a depth CNN, a recurrent neural network (RNN), a fully connected layer, a long short term memory (LSTM) layer, etc. A fusion feature may be generated by performing data fusion on any combination of the feature vectors. A predicted driver behavior may be generated based on the LSTM layer and n image frames on an image sequence and include x number of prediction frames.