Student DNN Output Distribution Matching for Resource-Constrained Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) with large numbers of parameters offer high accuracy in complex signal processing tasks but are resource-intensive, making them challenging to deploy on devices with limited computational and storage resources like smartphones and wearable devices, and reducing their size to address this issue often results in significant accuracy loss.
Innovation Solution
A 'teacher-student' learning approach where a larger, more accurate 'teacher' DNN trains a smaller 'student' DNN by minimizing the divergence between their output distributions using unlabeled data, allowing the student DNN to approximate the teacher's behavior without the need for labeled data, and iteratively optimizing the student's outputs to converge with the teacher's.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of parameters in DNN is increased to improve accuracy, then the accuracy in complex signal processing tasks is improved, but the computational and storage resources required increase
Solution Approach 1:
The patent creates a student DNN that copies the output distribution behavior of a teacher DNN with fewer parameters. By training the student network to reproduce the teacher's output probabilities rather than copying its internal structure, the system achieves high accuracy with reduced computational and storage requirements.
Solution Approach 2:
The patent changes the training objective from traditional parameter optimization to output distribution matching. By minimizing the divergence between student and teacher output distributions using unlabeled data, the system learns effective parameters for the student network that achieve high accuracy with fewer parameters.
2Device complexity
If the dimensions of DNN are reduced to decrease model size, then the model size is reduced, but the accuracy (e.g., word error rate) increases significantly
Solution Approach 1:
The student DNN with reduced dimensions learns to copy the output distribution of the larger teacher DNN. This approach allows the smaller model to maintain accuracy by reproducing the teacher's probabilistic outputs without requiring the same number of parameters or computational resources.
Solution Approach 2:
The teacher DNN is trained first on labeled data to establish a high-accuracy reference model. This preliminary training creates a target output distribution that the student network will learn to reproduce, enabling the student to achieve high accuracy despite its smaller size.
3Measurement precision
If labeled training data is required to train DNN, then the accuracy can be improved, but the availability and cost of training data increases
Solution Approach 1:
The teacher DNN serves as an intermediary that transforms unlabeled data into pseudo-labeled output distributions. By using the teacher's predictions as training targets for the student network, the system bypasses the need for expensive manual labeling while still achieving high accuracy through distribution matching.
Solution Approach 2:
The system uses the teacher DNN's own output on unlabeled data to train the student DNN. This self-service approach generates training signals automatically without external labeling, allowing both networks to improve accuracy while eliminating the need for large amounts of manually labeled training data.
Data Source
AI summary
Systems and methods are provided for generating a DNN classifier by “learning” a “student” DNN model from a larger more accurate “teacher” DNN model. The student DNN may be trained from un-labeled training data because its supervised signal is obtained by passing the un-labeled training data through the teacher DNN. In one embodiment, an iterative process is applied to train the student DNN by minimize the divergence of the output distributions from the teacher and student DNN models. For each iteration until convergence, the difference in the output distributions is used to update the student DNN model, and output distributions are determined again, using the unlabeled training data. The resulting trained student model may be suitable for providing accurate signal processing applications on devices having limited computational or storage resources such as mobile or wearable devices. In an embodiment, the teacher DNN model comprises an ensemble of DNN models.