GAN-Based Multi-Person Voice Separation via Masking Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face significant challenges in separating target speech from multiple human voices in noisy acoustic environments, as they struggle to differentiate between speakers due to similar speech characteristics, making speech separation more difficult than speech denoising.
Innovation Solution
A multi-person speech separation method and apparatus using a Generative Adversarial Network (GAN) model to extract masking coefficients and determine masking matrices from hybrid speech signals, allowing for the separation of multiple speech signals by identifying and isolating individual voices within a complex acoustic scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional speech denoising solutions are used, then speech can be separated from noise, but speech cannot be effectively separated from multiple human voices with similar characteristics
Solution Approach 1:
The patent segments the mixed speech signal into multiple independent speech signals by training the neural network to output separate masking matrices for each speaker. The segmentation is achieved by dividing the frequency spectrum into multiple bands and assigning each band to a specific speaker through the masking matrix, thereby separating overlapping speech signals in the time-frequency domain.
Solution Approach 2:
The patent changes the parameter representation by transforming the speech signal from the time domain to the frequency domain using short-time Fourier transform (STFT). The neural network operates on spectral features (magnitude and phase) rather than raw waveforms, and outputs masking matrices in the frequency domain that are then transformed back to time domain for final separation.
2Ease of manufacture
If speech denoising methods are applied to speech separation, then processing simplicity is maintained, but separation performance deteriorates due to similar characteristics of different speakers
Solution Approach 1:
The patent introduces masking matrices as intermediary elements that act as filters between the mixed speech signal and the separated speech signals. The masking matrix M(f,t) serves as a time-varying frequency mask that selectively passes or blocks specific frequency components at each time frame, enabling the separation of speakers with similar characteristics by learning speaker-specific spectral patterns.
Solution Approach 2:
The patent moves the separation problem from the time domain to the time-frequency domain by applying STFT. In this transformed domain, overlapping speech signals can be separated by manipulating frequency bins independently through masking matrices, adding a frequency dimension to the separation process that is not available in the raw time domain.
3Measurement precision
If the GAN model is used for speech separation, then separation accuracy is improved, but computational complexity increases
Solution Approach 1:
The GAN model is segmented into two independent but interacting components: a generator network that creates separated speech signals and a discriminator network that evaluates their authenticity. This segmentation allows each network to be optimized for its specific function while working together to achieve high separation accuracy through adversarial training.
Solution Approach 2:
The generator network performs preliminary separation of the mixed speech signal by producing estimated speech signals for each speaker before the discriminator evaluates them. This preliminary action allows the system to learn from the difference between generated and real speech, progressively improving separation accuracy through iterative adversarial training.
Data Source
Figure 1~2
Figure 3~4-a
Figure 4-b~4-d
AI summary
A multi-person voice separation method and apparatus, used for implementing separation between voices in a multi-person voice scenario, and comprising: a terminal extracts mixed voice features from a mixed voice signal to be separated, N human voices being mixed in the mixed voice signal, and N being a positive integer greater than or equal to 2 (101); the terminal uses a generative adversarial network model to implement masking coefficient extraction of the mixed voice features to obtain a masking matrix corresponding to the N human voices (102); and the terminal uses the generative adversarial network model to implement voice separation of the masking matrix corresponding to the N human voices and the mixed voice signal, and outputs N separate voice signals corresponding to the N human voices (103).