Causal Speech Separation Training With Pseudo-Autoregressive Conditioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech separation models are inadequate for real-time, high-fidelity processing of mixed audio streams due to limitations in offline networks and inefficient training methods, leading to significant degradation in separation quality and increased computational burden.
Innovation Solution
A pseudo-autoregressive Siamese training approach for causal neural networks, utilizing two forward passes through a Siamese-style network, where the current step separation is conditioned on past steps, and ground-truth outputs are replaced with neural network outputs during training, enabling efficient utterance-level processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If offline speech separation networks are used for streaming processing, then device complexity is reduced, but separation quality degrades significantly due to inability to access future inputs
Solution Approach 1:
The patent divides the speech separation task into multiple frames, where each frame is processed separately but conditioned on previous frame outputs. This segmentation allows the model to handle streaming data in real-time while maintaining quality through sequential processing of divided audio segments.
Solution Approach 2:
The patent uses teacher forcing during training, where ground truth labels are provided as conditional inputs for each frame. This preliminary action of providing correct labels during training enables the model to learn optimal separation patterns before deployment, improving inference quality without increasing runtime complexity.
2Manufacturing precision
If autoregressive training is used to improve separation quality, then manufacturing precision improves, but training time increases significantly due to sequential processing requirements
Solution Approach 1:
The patent implements a hybrid training approach where teacher forcing is applied periodically during training rather than continuously during inference. This allows the model to learn from ground truth at regular intervals while maintaining autoregressive behavior during actual processing, balancing training speed and inference quality.
Solution Approach 2:
The patent creates multiple copies of the model for parallel training, where each copy processes different frames simultaneously with teacher forcing. This copying strategy enables parallel computation during training, significantly reducing training time while maintaining the quality benefits of autoregressive processing.
3Ease of operation
If teacher forcing is used during training, then ease of operation improves, but reliability decreases due to error expansion during inference at high frame rates
Solution Approach 1:
The patent implements dynamic switching between teacher forcing and autoregressive modes. During training, teacher forcing is used to simplify the learning process, but during inference, the model dynamically relies on its learned patterns without ground truth inputs. This dynamic approach maintains training simplicity while improving inference reliability by preventing error expansion.
Data Source
AI summary
A method and system for supervised training of a causal neural network for a streaming audio processing application is provided. The method comprises acquiring an input mixture signal corresponding to two or more speakers. Further, the method comprises training the causal neural network to transform the input mixture signal into an output signal matching a ground truth signal. To that end, the training comprises processing the input mixture signal conditioned on a causal input including a delayed version of the input mixture signal transformed by the causal neural network without the causal input.