Neural Network Speech Separation with Iterative Phase Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech separation and speech enhancement applications face a phase inconsistency problem due to the use of mixture phase for time-domain re-synthesis, leading to inconsistent STFT representations and poor speech separation results.
Innovation Solution
An end-to-end approach for single-channel speaker-independent multi-speaker speech separation is implemented, using time-frequency (T-F) masking, short-time Fourier transform (STFT), and its inverse as layers within a deep network, along with iterative phase reconstruction algorithms to improve phase consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If magnitude separation is performed in the time-frequency domain using conventional methods, then the speech separation process is simplified, but phase inconsistency occurs leading to poor reconstruction quality
Solution Approach 1:
The patent replaces conventional magnitude-based separation methods with a neural network-based end-to-end system that operates directly on complex spectrograms. The neural network learns to separate speech components while preserving phase information, substituting traditional signal processing mechanics with intelligent computational methods that simultaneously achieve simplicity and precision.
Solution Approach 2:
The patent changes the fundamental parameters of speech separation by working with complex spectrograms (magnitude and phase) rather than just magnitude. The neural network is trained to output complex spectrograms with consistent phase relationships, fundamentally changing how separation is achieved from magnitude-only manipulation to joint magnitude-phase processing.
2Ease of operation
If mixture phase is directly used for time-domain re-synthesis, then the processing workflow is simplified, but the STFT representation becomes inconsistent
Solution Approach 1:
The patent introduces feedback mechanisms where the neural network's output is evaluated for phase consistency, and the network adjusts its predictions accordingly. The system uses loss functions that measure phase coherence and provide feedback signals to guide the network toward producing consistent STFT representations, ensuring reliability while maintaining operational simplicity.
Solution Approach 2:
The patent performs preliminary phase alignment and consistency checks within the neural network before final time-domain re-synthesis. By pre-processing the complex spectrograms to ensure phase consistency prior to inverse STFT, the system guarantees reliable STFT representations without requiring complex post-processing workflows.
3Manufacturing precision
If iterative phase reconstruction algorithms are implemented, then phase consistency is improved, but the system complexity increases
Solution Approach 1:
The patent merges iterative phase reconstruction capabilities directly into the neural network architecture, combining the strengths of iterative algorithms with the efficiency of deep learning. This integration allows the system to achieve high phase consistency without the overhead of separate iterative processing stages, reducing overall system complexity while maintaining precision.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Systems and methods for an audio signal processing system for transforming an input audio signal. A processor implements steps of a module by inputting an input audio signal into a spectrogram estimator to extract an audio feature sequence, and process the audio feature sequence to output a set of estimated spectrograms. Processing the set of estimated spectrograms and the audio feature sequence using a spectrogram refinement module, to output a set of refined spectrograms. Wherein the processing of the spectrogram refinement module is based on an iterative reconstruction algorithm. Processing the set of refined spectrograms for the one or more target audio signals using a signal refinement module, to obtain the target audio signal estimates. An output interface to output the optimized target audio signal estimates. Wherein the module is optimized by minimizing an error using an optimizer stored in the memory.