Audio-Visual Speech Recognition via DNN Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech recognition technologies at a distance lack optimization for accuracy, particularly in combining audio and visual information effectively, and are limited by small datasets and small speaker/vocabulary tasks.
Innovation Solution
A method utilizing Deep Neural Networks (DNNs) for joint audio-visual speech recognition, incorporating image preprocessing and feature combination to enhance beam-forming and speech recognition, with recurrent neural networks for temporal modeling and region-of-interest extraction, enabling robust speech recognition across distances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If microphone array processing is used for distance speech recognition, then noise and reverberation are suppressed, but speech recognition accuracy is not optimized
Solution Approach 1:
The patent combines audio signals from multiple microphones with visual information from video cameras to create a fused audio-visual representation. This merging of multiple information sources allows the system to leverage both the noise suppression capabilities of microphone arrays and the spatial context from visual data, thereby achieving both noise suppression and optimized speech recognition accuracy simultaneously
Solution Approach 2:
The patent introduces an audio-visual integration module that acts as an intermediary between the microphone array processing and speech recognition components. This intermediary fuses audio and visual features, using visual information to guide audio processing and enhance the speech signal representation, thereby resolving the contradiction between noise suppression and recognition accuracy
2Measurement precision
If audio-visual integration is implemented, then speech recognition accuracy improves, but system complexity increases
Solution Approach 1:
The patent divides the audio-visual processing system into distinct modular components: audio feature extraction module, visual feature extraction module, and audio-visual integration module. Each module performs a specific function and processes its respective input independently before fusion. This segmentation reduces system complexity by making each component manageable and interchangeable, while still achieving improved speech recognition accuracy through their coordinated operation
Solution Approach 2:
The patent develops a unified audio-visual integration framework that can process both audio and visual inputs through a single integrated model. This multi-functional approach eliminates the need for separate processing pipelines for audio and video, reducing overall system complexity while maintaining the accuracy benefits of audio-visual fusion
3Reliability
If deep neural networks are used for audio-visual integration, then recognition robustness improves, but computational requirements increase
Solution Approach 1:
The patent performs preliminary feature extraction and preprocessing of audio and visual inputs before they are fed into the deep neural network. By extracting relevant features in advance and preprocessing the data to remove redundancies, the system reduces the computational burden on the DNN while maintaining recognition robustness. This preliminary action allows the DNN to focus on integrating and classifying already-processed features rather than processing raw data
Data Source
AI summary
Disclosed herein is method of performing speech recognition using audio and visual information, where the visual information provides data related to a person's face. Image preprocessing identifies regions of interest, which is then combined with the audio data before being processed by a speech recognition engine.