Guided Speech Enhancement With Beamformer-Agnostic Interference Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-microphone speech enhancement systems face challenges in effectively reducing interference such as echo, background noise, and competing speech due to the need for large training data and computational power, and they struggle to generalize well to real-world conditions with device-specific microphone geometries.
Innovation Solution
A guided speech-enhancement network that leverages the spatial filtering performed by a beamformer to further attenuate interfering signals, trained using a simulated generic beamformer model, allowing it to be agnostic to specific microphone arrangements and beamformers, and capable of real-time streaming speech enhancement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network-based end-to-end models are used to improve speech enhancement, then speech recognition accuracy is improved, but computational power requirements and training data needs increase significantly
Solution Approach 1:
The speech enhancement task is divided into two separate stages: beamforming for spatial filtering and a neural network for spectral enhancement. This segmentation allows each component to be optimized independently, reducing the overall computational burden compared to a single end-to-end model while maintaining speech recognition accuracy.
Solution Approach 2:
The beamformer performs preliminary spatial filtering to attenuate interfering signals before the neural network processes the audio data. This preliminary action reduces the complexity of the subsequent neural network processing by pre-processing the input data to remove obvious interference, thereby reducing computational power requirements.
2Measurement precision
If multi-channel audio input is used to improve spatial resolution and signal-to-noise ratio, then speech capture quality is improved, but device complexity increases
Solution Approach 1:
The guided speech enhancement network is designed to be device-agnostic and can work with different microphone geometries and beamformer configurations. This universality allows the same neural network model to be deployed across multiple device types without requiring device-specific customization, thereby managing device complexity while maintaining spatial resolution capabilities.
3Reliability
If existing speech enhancement models are trained on large-scale data, then speech enhancement performance is improved, but training time and computational resources increase
Solution Approach 1:
The beamformer performs preliminary spatial filtering to create pre-processed training data that has already had interfering signals attenuated. This preliminary action allows the neural network to be trained on more effective data with fewer samples needed, reducing training time while maintaining or improving speech enhancement performance.
Solution Approach 2:
The beamformer acts as an intermediary that transforms raw multi-channel audio into spatially-filtered audio that is more suitable for neural network training. This intermediary processing creates higher-quality training samples that improve model performance while potentially reducing the total amount of training data and time required.
Data Source
AI summary
A method includes receiving, as input, reference audio data representing a reference audio signal captured by an audio input device. The method also includes receiving, as input, from a beamformer, spatially-filtered audio data representing an output of the beamformer, the beamformer configured to spatially filter, based on additional audio data captured by one or more additional audio input devices, the reference audio data to attenuate one or more interfering signals in the spatially-filtered audio data. The method processes, using a trained guided speech-enhancement network, the reference audio data and the spatially-filtered audio data to generate, as output, enhanced audio data, the guided speech-enhancement network processing the reference audio data and the spatially-filtered audio data to further attenuate, in the enhanced audio data, the one or more interfering signals attenuated by the beamformer.