Sonification Audio for Video Event Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated event detection using both audio and video data is challenging due to significant data set mismatch and resource-intensive processing, particularly because video data has a much larger bit rate than audio data, leading to difficulties in data synchronization and efficient event classification.
Innovation Solution
The method involves generating sonification audio data from video data, which represents object positions and movements, and combining it with original audio data to create a balanced dataset, allowing for efficient event detection and classification using machine learning techniques, thereby reducing the complexity of the classification process and resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If both audio and video data are processed together for automated event detection, then event detection accuracy is improved, but processing resource consumption increases significantly
Solution Approach 1:
The system segments video data processing from audio data processing by converting video information into sonification audio data. This segmentation allows audio processing operations to handle both original audio and video-derived audio separately, reducing the computational burden of processing raw video data while maintaining event detection accuracy.
Solution Approach 2:
The system replaces direct video data processing with an acoustic substitution approach. Video data is converted into sonification audio data that represents visual information through audio characteristics, allowing the system to use audio processing operations instead of computationally intensive video processing operations.
2Reliability
If video data is processed directly for event detection, then detection accuracy is improved, but data synchronization becomes difficult due to larger bit rate
Solution Approach 1:
The system creates an acoustic copy of video data through sonification. Instead of processing the original high-bit-rate video data directly, the system generates audio data that copies and represents visual information, making it easier to synchronize with audio data while preserving event detection capabilities.
3Productivity
If only audio data is used for event detection, then processing efficiency is improved, but event classification accuracy deteriorates
Solution Approach 1:
The system merges audio data with video-derived sonification audio data into combined audio data. This combination allows the system to maintain processing efficiency by using audio processing operations while incorporating information from both original audio and video sources, thereby improving event classification accuracy without sacrificing processing speed.
Data Source
AI summary
Computer-implemented event detection includes obtaining, at one or more processors, multimedia data including multiple frames of video data and corresponding audio data. The processor(s) process the multiple frames to detect at least one object and to track the object(s) between two or more of the frames. The processor(s) generate sonification audio data representing a position of the object(s) in the two or more frames, movement of the object(s), or both the position and the movement of object(s). The processor(s) generate combined audio data including the audio data and the sonification audio data. The processor(s) generate one or more feature vectors representing the combined audio data and provide the feature vector(s) as input to a trained event classifier to detect an event represented in the multimedia data.