Speech Enhancement Neural Training for Small Far-Field Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning approaches for speech enhancement in audio signals require significant training data and are not well-suited for resource-constrained platforms or far-field applications, leading to large model sizes and suboptimal performance.
Innovation Solution
A neural network is trained using multi-tasking to infer both a noise-invariant speech component and a noise-variant SNR component, requiring less training data and enabling deployment on resource-constrained devices like mobile phones and IoT devices, while improving performance in far-field scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional machine learning approaches are used for speech enhancement, then speech enhancement performance can be achieved, but the model size becomes too large for resource-constrained platforms
Solution Approach 1:
The patent segments the speech enhancement task into two separate neural network models: a first model trained on clean speech data for noise-invariant features, and a second model trained on noisy speech data for noise-variant features. This segmentation allows each model to be smaller and more suitable for resource-constrained platforms while maintaining overall enhancement performance through feature fusion.
2Reliability
If traditional machine learning approaches are used for speech enhancement, then speech enhancement can be performed, but extensive training data is required
Solution Approach 1:
The training process is segmented into two separate training phases with different data requirements. The first model is trained on clean speech data to learn noise-invariant features, while the second model is trained on noisy speech data to learn noise-variant features. This segmentation reduces the overall training data burden compared to training a single comprehensive model.
Solution Approach 2:
The first neural network model trained on clean speech data serves multiple purposes: it learns noise-invariant features that are universally applicable across different noise conditions, and provides a foundation that can be adapted to various noisy environments through the second model, reducing the need for extensive noise-specific training data.
3Reliability
If traditional machine learning approaches are used, then speech enhancement may work for near-field applications, but performance is suboptimal for far-field applications
Solution Approach 1:
The system creates a universal speech enhancement approach that works across different field conditions by fusing features from both clean-speech-trained and noisy-speech-trained models. The noise-invariant features provide robustness for near-field applications, while the noise-variant features adapt to far-field conditions, making the overall system versatile across application scenarios.
Solution Approach 2:
The patent combines features from two different neural network models (trained on different data types) to create a composite enhancement output. This is analogous to using composite materials where the strengths of different components are combined to achieve performance that neither component alone could provide, particularly improving far-field adaptability.
Data Source
AI summary
A method of training neural networks may include receiving a sequence of audio frames, and mapping a first audio frame in the sequence of audio frames to a first output frame based on a neural network. The first output frame may represent a noise-invariant component of the first audio frame. The method may also include determining a first loss value based on differences between the first output frame and a first ground truth frame. The method may include mapping the first audio frame to a second output frame based on the neural network. The second output frame may represent a noise-variant component of the first audio frame. The method may further include determining a second loss value based on differences between the second output frame and a second ground truth frame, and updating the neural network based at least in part on the first and second loss values.