Hardware-Aware Pruning of Conformer Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for speech enhancement are large and costly to train, making them challenging to implement in resource-constrained devices like mobile devices, where power and other resources are limited.
Innovation Solution
The method involves training a neural network using a series of pruning operations, where the pruning fraction increases gradually with each epoch, resulting in a significantly smaller neural network. This approach combines pruning with other compression techniques like parameter sharing, knowledge distillation, and input channel reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural network is used for speech enhancement, then speech quality is improved, but the model size and training cost increase
Solution Approach 1:
The patent extracts and removes redundant or less important neurons, weights, or filters from the neural network through pruning operations. This extraction process eliminates unnecessary components while preserving the essential functionality needed for speech enhancement, thereby reducing model size without significantly compromising speech quality.
Solution Approach 2:
The patent uses knowledge distillation where a smaller student network is trained to replicate the behavior of a larger teacher network. The student network copies the essential knowledge and patterns from the teacher network, achieving comparable speech enhancement performance with significantly reduced model size.
2Reliability
If a neural network is used for speech enhancement, then speech quality is improved, but the training cost increases
Solution Approach 1:
The patent performs pruning operations at multiple stages during the training process rather than training a complete large network first. By progressively removing less important parameters during training, the system avoids the computational waste of training and then discarding those parameters, thereby reducing overall training cost while maintaining speech quality.
Solution Approach 2:
The patent implements dynamic pruning where the network structure adapts during training based on the importance of different parameters. The pruning fraction and which parameters are removed change dynamically throughout training, allowing the system to optimize both speech quality and training efficiency by focusing computational resources on important parameters at each training stage.
3Reliability
If a neural network is used for speech enhancement, then speech quality is improved, but deployment in resource-constrained devices becomes difficult
Solution Approach 1:
The patent extracts and removes unnecessary computational components from the neural network through structured pruning, reducing the model size to fit within the memory constraints of mobile devices while preserving the core speech enhancement functionality needed for quality improvement.
Solution Approach 2:
The patent changes the parameters of the neural network by applying pruning fractions that control the density of the network. This parameter adjustment transforms the model from a dense large-scale network to a sparse compact network that can be deployed on resource-constrained devices while maintaining adequate speech enhancement performance.
Data Source
AI summary
A system and a method are disclosed for hardware-aware pruning of conformer networks. In some embodiments, the method includes: training a neural network, the training including: performing a first pruning operation, on the neural network, after a first training epoch, and performing a second pruning operation, on the neural network, after a second training epoch and after the first pruning operation, wherein each of the pruning operations results in a respective pruning fraction, the respective pruning fraction being a function of an index of a training epoch preceding the pruning operation.