In-Cabin Occupant Behavior Analysis With Spatiotemporal Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current in-cabin occupant behavior description methods, such as driver monitoring systems and occupant monitoring systems, lose temporal information and struggle to precisely describe actions when multiple occupants are present, as they rely on convolutional neural networks that degrade spatial information and generate single descriptions from video sequences.
Innovation Solution
The proposed method uses a hybrid computational system with a field programmable gate array (FPGA), graphical processor unit (GPU), and central processing unit (CPU) to process video sequences, employing deep learning and recurrent neural networks like convolutional gated recurrent units (GRUs) to extract and retain spatial and temporal information, directly generating final results like occupant locations and behavior descriptions without intermediate steps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If convolutional neural networks with global pooling are used to extract features and summarize spatial information, then processing efficiency is improved, but spatial information is degraded and temporal information is lost
Solution Approach 1:
The patent introduces an intermediary mechanism (convolutional gated recurrent unit) that acts as a bridge between spatial feature extraction and temporal sequence processing. This intermediary preserves spatial information through convolutional operations while adding temporal processing capabilities, avoiding the information loss caused by direct global pooling.
Solution Approach 2:
The patent transitions from 2D spatial feature maps to 3D spatiotemporal feature representations by incorporating the time dimension. This dimensional expansion allows the system to maintain spatial information while simultaneously processing temporal sequences, rather than compressing to 1D feature vectors that lose both spatial and temporal details.
2Device complexity
If a single description is generated for a video sequence, then processing complexity is reduced, but the ability to precisely pinpoint actions of specific occupants is lost
Solution Approach 1:
The patent segments the video sequence processing into individual occupant analysis units. Each occupant is independently tracked and analyzed through the convolutional gated recurrent unit, allowing the system to generate separate descriptions for each occupant's actions while maintaining overall system efficiency through unified architectural processing.
3Device complexity
If rule-based methodology is used to process intermediate results and generate final results, then system simplicity is maintained, but the ability to capture complex temporal behaviors is limited
Solution Approach 1:
The patent replaces the mechanical rule-based processing system with a neural network-based convolutional gated recurrent unit. This substitution enables the system to automatically learn complex temporal patterns and behaviors from data, providing greater adaptability while maintaining computational efficiency through the structured architecture of the recurrent unit.
Data Source
AI summary
A method of describing a temporal event, including receiving a video sequence of the temporal event, extracting at least one physical characteristic of an at least one occupant within the video sequence, extracting at least one action of the at least one occupant within the video sequence, extracting at least one interaction of the at least one occupant with a secondary occupant within the video sequence, determining a safety level of the temporal event within a vehicle based on at least one of the at least one action and the at least one interaction and describing the at least one physical characteristic of the at least one occupant and at least one of the at least one action and the at least one interaction of the at least one occupant.