Speech Enhancement via Sequential AI Models and Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech enhancement technologies face limitations in effectively addressing multiple distortions in digital audio signals, requiring powerful hardware and large training datasets, while also losing information through signal compression and being limited by strong inductive biases.
Innovation Solution
A method utilizing three sequentially applied deep neural networks: a subtractive model for noise removal, an autoencoder for dimensionality reduction, and a generative model for enhancement, trained on simulated real-world distortions to elevate perceptual quality, operating on a lower-dimensional signal representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If deep neural networks with millions of parameters are used for speech enhancement, then model performance and power are improved, but hardware requirements and computational burden increase significantly
Solution Approach 1:
The system divides the speech enhancement task into multiple sequential sub-models, each handling specific distortion types (noise removal, reverberation, compression artifacts). This segmentation allows each sub-model to be smaller and more specialized, reducing overall hardware requirements while maintaining comprehensive enhancement capability.
Solution Approach 2:
The system transforms the high-dimensional waveform input into a lower-dimensional spectrogram representation, processes enhancement operations in this compressed domain, and then reconstructs the output. This dimensionality change reduces computational burden and hardware requirements while preserving essential speech information.
2Manufacturing precision
If high-dimensional waveform data is processed directly, then speech quality is maintained, but computational burden and data dimensionality increase
Solution Approach 1:
The system converts waveform data into spectrogram representation, which compresses the high-dimensional time-series data into a lower-dimensional time-frequency matrix. This transformation maintains speech quality information while significantly reducing computational complexity for subsequent enhancement operations.
Solution Approach 2:
The spectrogram serves as an intermediary representation between the input waveform and the enhanced output. This intermediate form facilitates more efficient processing by separating temporal and spectral information, allowing targeted enhancement of specific frequency components without manipulating the entire high-dimensional waveform.
3Device complexity
If signal compression is applied to reduce dimensionality, then computational burden is reduced, but information loss occurs
Solution Approach 1:
Instead of using traditional lossy compression methods, the system employs learned compression through neural network-based spectrogram generation. The model learns to preserve speech-relevant information during the compression to representation transformation, and subsequent enhancement sub-models recover and enhance this information, minimizing actual information loss.
Solution Approach 2:
The system uses feedback mechanisms where enhancement sub-models process the compressed representation and generate enhanced spectrograms that are transformed back to waveform. The training process incorporates feedback loops that adjust model parameters to minimize the difference between enhanced output and original clean speech, ensuring information preservation despite compression.
4Quantity of substance
If strong inductive biases are built into AI models, then training data efficiency increases, but model performance may be limited or hurt
Solution Approach 1:
The system applies segmentation to the modeling approach by creating separate sub-models for different distortion types rather than using a single heavily-biased model. Each sub-model has targeted inductive biases appropriate for its specific task, achieving good data efficiency without the performance limitations that would result from strong general biases.
Solution Approach 2:
The system changes parameters dynamically based on the input characteristics and distortion type. Different sub-models are applied based on the detected distortion profile, allowing the system to adapt its inductive biases to match the specific enhancement task, thereby maintaining both data efficiency and high performance.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
There is provided a method for speech enhancement of a digital audio signal, wherein the input digital audio signal is in a digital waveform representation, the method comprising the steps of: a) based on a first trained artificial intelligence, Al, model creating a denoised digital audio signal in the waveform representation by removing energy from the input digital audio signal; b) based on a second trained Al model mapping the denoised digital audio signal in the waveform representation into a lower dimensional signal representation; c) based on a third trained Al model determining an enhanced digital audio signal in the lower dimensional signal representation such that the perceptual quality of the voice is elevated; and d) based on the second trained Al model mapping the enhanced digital audio signal in the lower dimensional signal representation back to the waveform representation; wherein each of the first, second, and third trained Al model is based on a respective Al model comprising a plurality of model parameters which are determined based on a training process.