Tiny DNN Speech Enhancement With Frequency Band-Wise Noise Mixup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern deep learning-based models for speech enhancement require significant computational resources and are impractical for deployment on resource-constrained devices, despite efforts to reduce computational overhead, resulting in unsatisfactory performance.
Innovation Solution
A novel training pipeline with a proper loss function and data augmentation technique, specifically frequency band-wise multiple noise mixup (FMN-Mixup), is applied to tiny DNN models to enhance speech intelligibility and perceptual quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If modern deep learning-based models are used for speech enhancement, then speech enhancement performance is significantly improved, but computational resources and hardware requirements increase significantly
Solution Approach 1:
The patent segments the frequency spectrum into multiple bands and applies different processing strategies to each band. The frequency band-wise mixup augmentation treats each frequency band independently, allowing the model to learn frequency-specific noise patterns and enhancement characteristics, thereby achieving effective speech enhancement with reduced computational complexity
Solution Approach 2:
The patent changes the data augmentation parameters by applying mixup augmentation at the frequency band level rather than globally. By generating mixup coefficients for each frequency band and applying them to combine noise components, the model achieves better performance with fewer training samples and reduced computational overhead
2Use of energy by moving object
If tiny DNN models are used to reduce computational overhead, then hardware requirements are reduced, but speech enhancement performance becomes unsatisfactory
Solution Approach 1:
The patent applies preliminary data augmentation during training by generating augmented samples through frequency band-wise mixup augmentation. This preliminary action creates diverse training data that helps the tiny DNN model learn robust noise patterns and enhancement characteristics, enabling the model to achieve satisfactory performance despite its limited computational capacity
Solution Approach 2:
The patent creates synthetic training data by copying and combining noise components from different frequency bands through mixup augmentation. This generates virtual training samples that expand the effective training dataset, allowing the tiny DNN to learn from diverse noise patterns without requiring additional computational resources during inference
3Reliability
If data augmentation is applied to increase training data diversity, then model performance improves, but training time and computational cost increase
Solution Approach 1:
The patent applies local quality by generating mixup coefficients specifically for each frequency band based on the noise characteristics and importance of that band. This localized approach allows the data augmentation to focus computational effort on the most critical frequency regions, improving model performance while minimizing unnecessary computation in less important bands
Data Source
AI summary
A system and a method are disclosed for data augmentation. A method includes obtaining a plurality of noisy spectrograms; extracting noise components from the plurality of noisy spectrograms; individually generating a mixup coefficient for each of the extracted noise components; applying the mixup coefficients to the extracted noise components; merging the extracted noise components; and combining the merged noise components with a clean spectrogram to provide an augmented sample.