Generative Neural Network Speech Enhancement System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech enhancement technologies fail to effectively remove a wide range of artifacts such as noise, reverberation, clipping, silent gaps, equalization issues, and codec artifacts from audio signals in a single step, compromising the naturalness of speech recordings.
Innovation Solution
A neural network-based system comprising a generative network and a conditioning network that processes audio signals to generate enhanced outputs by combining internal representations at different temporal resolutions, using side information for adaptability, and employing auxiliary neural networks for training with loss functions to remove various artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If simple denoising algorithms are used, then noise removal is achieved, but multiple artifacts (reverberation, clipping, codec artifacts) cannot be removed simultaneously
Solution Approach 1:
The patent implements a universal speech enhancement system that can handle multiple types of artifacts (noise, reverberation, clipping, codec artifacts, equalization issues) through a single integrated neural network architecture. The generative network is designed to perform multiple functions simultaneously, replacing the need for separate specialized algorithms for each artifact type.
Solution Approach 2:
The system divides the speech enhancement task into distinct processing components: a conditioning network that processes the input speech signal to extract features, and a generative network that synthesizes the enhanced output. This segmentation allows each component to specialize in specific aspects of artifact removal while working together as a unified system.
2Adaptability or versatility
If multiple separate algorithms are used to remove different artifacts, then each artifact type can be addressed, but processing time and computational resources increase
Solution Approach 1:
The patent merges multiple artifact removal functions into a single integrated neural network system. The conditioning network and generative network work together in one processing pass to simultaneously remove noise, reverberation, clipping, and other artifacts, eliminating the need for sequential processing with multiple separate algorithms.
Solution Approach 2:
The conditioning network performs preliminary processing of the input speech signal by extracting relevant features and conditioning information before the generative network produces the final enhanced output. This preliminary action prepares the data in advance, enabling the generative network to efficiently synthesize the enhanced speech without requiring multiple iterative processing steps.
3Object-affected harmful factors
If aggressive denoising is applied, then noise removal is improved, but naturalness of speech is compromised
Solution Approach 1:
The conditioning network acts as an intermediary between the noisy input speech and the generative network. It processes the input signal to extract conditioning features that guide the generative network in producing enhanced speech that maintains naturalness. This intermediary processing ensures that noise removal does not directly compromise speech quality.
Solution Approach 2:
The generative network learns to adjust multiple parameters of the speech signal simultaneously, including noise levels, reverberation characteristics, clipping artifacts, and other degradation factors. By changing these parameters in a coordinated manner based on the conditioning information, the system achieves effective denoising while preserving speech naturalness.
Data Source
AI summary
The disclosure relates to a neural network based system for speech enhancement, comprising a generative network for generating an enhanced audio signal and a conditioning network for generating conditioning information for the generative network. The conditioning network comprises a plurality of layers and is configured to receive an audio signal as input: propagate the audio signal through the plurality of layers; and provide one or more first internal representations of the audio signal or processed versions thereof as the conditioning information, wherein the one or more first internal representations of the audio signal are extracted at respective layers of the conditioning network. The generative network is configured to receive a noise vector and the conditioning information as input; and generate the enhanced audio signal based on the noise vector and the conditioning information. The disclosure further relates to a method of training the system.