Multi-Scale CNN Speech Source Separation for Dynamic Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech source separation methods struggle to effectively extract dialog from professional content, such as movies and TV, due to the complexity of dynamic background noise, which is not adequately addressed by traditional stationary noise-focused approaches.
Innovation Solution
A Convolutional Neural Network (CNN) based method that utilizes an aggregated multi-scale architecture with parallel convolution paths, applying time-frequency transforms, weight adjustments, and cascaded pooling to generate output masks for isolating speech from noisy signals, incorporating features like leaky rectified linear units and dilation filters for enhanced feature extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If traditional spectral subtraction method is used for speech enhancement, then the method is simple and widely applicable, but it assumes stationary background noise which limits its effectiveness for dynamic background in professional content
Solution Approach 1:
The patent transforms the static, stationary noise assumption into a dynamic model by using deep neural networks that can adapt to non-stationary background conditions. The system dynamically adjusts its noise estimation and speech separation parameters in real-time based on changing acoustic environments, enabling effective processing of professional content with complex dynamic backgrounds.
Solution Approach 2:
The patent changes the fundamental parameters of the speech enhancement approach by moving from traditional spectral subtraction with fixed parameters to deep learning-based methods with learnable parameters. The neural network models learn optimal parameters for noise estimation, speech separation, and mask generation from training data, allowing the system to adapt to various dynamic background conditions in professional content.
2Measurement precision
If deep neural networks are used for speech separation, then performance is greatly improved, but the model complexity and computational requirements increase
Solution Approach 1:
The patent segments the speech separation task into multiple specialized neural network components: a noise estimation network, a speech separation network, and a mask generation network. Each component focuses on a specific aspect of the separation problem, allowing for more efficient training and inference while maintaining high performance. This modular architecture reduces overall system complexity compared to a monolithic model.
Solution Approach 2:
The patent introduces time-frequency masking as an intermediate representation dimension, transforming the speech separation problem from the time domain to the time-frequency domain. This dimensional transformation allows the neural networks to operate on spectrogram representations, capturing both temporal and spectral characteristics of speech and noise, thereby improving separation performance while enabling more efficient processing.
3Adaptability or versatility
If multi-scale CNN architecture is used for feature extraction, then both stationary and non-stationary background separation is improved, but the number of parallel convolution paths and computational load increase
Solution Approach 1:
The patent designs a multi-scale CNN architecture where parallel convolution paths with different kernel sizes serve multiple functions: capturing local speech patterns, modeling broader contextual information, and adapting to various noise conditions. Each path contributes to a unified speech separation objective, making the system universally applicable to both stationary and non-stationary backgrounds while sharing computational resources through the aggregated output mechanism.
Data Source
AI summary
Described herein is a method for Convolutional Neural Network (CNN) based speech source separation, wherein the method includes the steps of: (a) providing multiple frames of a time-frequency transform of an original noisy speech signal; (b) inputting the time-frequency transform of said multiple frames into an aggregated multi-scale CNN having a plurality of parallel convolution paths; (c) extracting and outputting, by each parallel convolution path, features from the input time-frequency transform of said multiple frames; (d) obtaining an aggregated output of the outputs of the parallel convolution paths; and (e) generating an output mask for extracting speech from the original noisy speech signal based on the aggregated output. Described herein are further an apparatus for CNN based speech source separation as well as a respective computer program product comprising a computer-readable storage medium with instructions adapted to carry out said method when executed by a device having processing capability.