Neural Network Knowledge Distillation for Edge Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with larger configurations have low computing capability, making them difficult to execute on edge systems like vehicles and IoT devices due to power consumption restrictions, and existing methods for downsizing, such as knowledge distillation, require significant data collection efforts.
Innovation Solution
A learning device that generates input data with minimal error for a student neural network by propagating data through a teacher neural network, allowing for training with reduced workload and smaller neural network configurations, while maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the configuration of the neural network is upsized to achieve high performance, then the performance is improved, but the computing capability deteriorates and power consumption increases
Solution Approach 1:
The patent divides the training process into two stages: first training a large-scale teacher neural network to achieve high performance, then using knowledge distillation to transfer knowledge to a smaller student neural network. This segmentation allows the system to benefit from both large-scale training and efficient deployment.
Solution Approach 2:
The patent creates a copy of the knowledge learned by the teacher neural network and transfers it to the student neural network through knowledge distillation. The student network learns from the teacher's output data (soft labels) rather than original training data, enabling the smaller network to achieve comparable performance.
2Productivity
If knowledge distillation is used to downsize the neural network configuration, then the computing capability is improved, but the workload for data collection increases
Solution Approach 1:
The patent performs preliminary action by first training the teacher neural network on the original training data to achieve high performance. This pre-trained teacher network then serves as the knowledge source for training the student network, eliminating the need to collect and prepare separate training data for the student network.
Solution Approach 2:
The patent introduces an intermediary mechanism where the teacher neural network acts as a mediator between the original training data and the student neural network. The teacher network processes the original training data and generates soft labels that are used to train the student network, reducing the direct data collection burden.
3Use of energy by moving object
If a smaller neural network configuration is used for edge systems, then the power consumption is reduced, but the performance deteriorates
Solution Approach 1:
The patent changes the parameters of the student neural network by training it with soft labels from the teacher network instead of hard labels from original data. This parameter change in the training process enables the smaller network to achieve performance comparable to the larger teacher network.
Solution Approach 2:
The patent implements feedback through the distillation loss function that compares the student network's output with the teacher network's soft labels. This feedback mechanism guides the student network to learn the teacher's knowledge, ensuring performance maintenance despite size reduction.
Data Source
AI summary
A learning device according to an embodiment includes one or more hardware processors configured to function as a generation unit, an inference unit, and a training unit. The generation unit generates input data with which an error between a value output from each of one or more target nodes and a preset aimed value is equal to or less than a preset value, the target nodes being in a target layer of a plurality of layers included in a first neural network. The inference unit causes the input data to propagate in a forward direction of the first neural network to generate output data. The training unit trains a second neural network differing from the first neural network by using training data including a set of the input data and the output data.