Mixup Knowledge Distillation for Compact Speech Recognition Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks for speech recognition are computationally demanding and difficult to deploy on edge devices due to their large size, and existing knowledge distillation methods struggle to effectively transfer knowledge from complex teacher networks to smaller student networks, leading to poorer performance and processing speed.
Innovation Solution
A method using mix-up augmentation and soft labels is employed to train student neural networks by combining input vectors and their corresponding soft labels, allowing the student network to mimic the behavior of a teacher network, thereby reducing complexity and enhancing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a larger teacher neural network is used for speech recognition, then recognition accuracy is improved, but the model size and computational demand increase making it difficult to deploy on edge devices
Solution Approach 1:
The patent creates a simplified copy (student network) of the complex teacher network by training it to replicate the teacher's soft labels. The student network copies the essential recognition patterns from the teacher without needing to replicate its full complexity, enabling deployment on edge devices while maintaining recognition accuracy.
Solution Approach 2:
The patent transforms the training approach by changing from hard labels to soft labels, and introduces mixup augmentation that dynamically modifies input parameters. These parameter changes enable the student network to learn more efficiently from the teacher, achieving good performance with fewer parameters suitable for edge devices.
2Device complexity
If knowledge distillation is used to compress the neural network, then model size is reduced, but existing methods struggle to effectively transfer knowledge leading to poorer performance
Solution Approach 1:
The patent introduces soft labels as an intermediary between the teacher and student networks. Instead of direct hard label training, the soft labels mediate the knowledge transfer process, allowing the student network to learn nuanced patterns from the teacher more effectively, thus maintaining accuracy while achieving compression.
Solution Approach 2:
The patent applies mixup augmentation to create synthesized training examples before the knowledge distillation process. This preliminary action enriches the training data with diverse combinations of speech patterns, enabling the student network to learn more robust features from the teacher and improve performance after compression.
3Device complexity
If existing knowledge distillation methods are used, then model compression is achieved, but processing speed does not improve sufficiently for edge device deployment
Solution Approach 1:
The patent introduces dynamic mixup augmentation that randomly combines training examples during the knowledge distillation process. This dynamic approach allows the student network to learn more generalizable patterns that process faster on edge devices, improving processing speed while maintaining the compressed model size.
Data Source
AI summary
A method of training a student neural network is provided. The method includes feeding a data set including a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method further includes combining the two corresponding input vectors to form a synthesized data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthesized data vector into the student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.