Audio-Visual Speaking Classification via Neural Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaking classification systems that rely solely on visual data face challenges in accuracy and computational resource consumption, and may produce noisy predictions due to rapid transitions between 'speaking' and 'not speaking' classifications, especially when the mouth of the target person is occluded.
Innovation Solution
A system that integrates both audio and visual data using a recurrent neural network, processing video frames and audio data to generate accurate speaking classification predictions by learning correlations between mouth and body movements with corresponding audio signals, and employing an end-to-end optimization procedure for joint training of neural networks to reduce resource consumption and improve prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system processes more video frames to improve speaking classification accuracy, then prediction accuracy increases, but computational resource consumption increases
Solution Approach 1:
The patent merges audio data and visual data processing into a unified speaking classification system. By combining audio embeddings and image embeddings through a fusion neural network, the system achieves accurate speaking detection without needing to process every video frame independently, thus reducing computational resources while maintaining or improving accuracy.
Solution Approach 2:
The patent introduces audio data as an additional dimension to the traditional visual-only approach. By processing audio embeddings alongside image embeddings, the system captures speaking information from multiple modalities, allowing for more accurate classification with fewer video frames processed, thereby reducing computational load.
2Use of energy by moving object
If the system uses only visual data for speaking classification, then computational resource consumption is reduced, but prediction accuracy decreases and noisy predictions occur
Solution Approach 1:
The patent combines audio and visual processing streams that converge in a fusion neural network. This integration allows the system to leverage complementary information from both modalities, improving speaking classification accuracy while maintaining efficient resource utilization through shared computational architecture.
Solution Approach 2:
The patent introduces embedding layers as intermediaries that transform raw audio and visual data into compressed representations. These embeddings serve as efficient intermediaries that capture essential speaking characteristics, allowing the fusion neural network to make accurate predictions without processing the full original data, thus improving accuracy while controlling computational resources.
3Speed
If the system processes only mouth region images, then processing speed increases, but prediction accuracy decreases when mouth is occluded
Solution Approach 1:
The patent makes the image processing system universal by accepting images of various regions (mouth, face, body) and processing them through the same embedding and fusion architecture. This multi-functional approach allows the system to handle occluded mouths by utilizing alternative body regions or facial features, maintaining both processing speed and accuracy across different input conditions.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating predictions for whether a target person is speaking during a portion of a video. In one aspect, a method includes obtaining one or more images which each depict a mouth of a given person at a respective time point. The images are processed using an image embedding neural network to generate a latent representation of the images. Audio data corresponding to the images is processed using an audio embedding neural network to generate a latent representation of the audio data. The latent representation of the images and the latent representation of the audio data is processed using a recurrent neural network to generate a prediction for whether the given person is speaking.