Knowledge Distillation Data Filtering by Teacher Uncertainty
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing knowledge distillation techniques using teacher neural networks for training student neural networks are hindered by noisy or inaccurate teacher outputs, particularly in unlabeled datasets, leading to suboptimal performance of the student networks.
Innovation Solution
A system filters training examples based on an uncertainty measure of the teacher neural network's outputs to generate a high-quality filtered dataset, using techniques like entropy or margin scores to determine the likelihood of incorrect labels, thereby improving the training process for the student neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If knowledge distillation uses teacher neural network outputs for training student neural networks, then the student network can be trained efficiently, but the training quality deteriorates due to noisy or inaccurate teacher outputs
Solution Approach 1:
The patent applies preliminary action by computing uncertainty measures for teacher network outputs before using them for student network training. The system calculates entropy or margin scores for each training example in advance, identifies high-uncertainty examples, and filters or down-weights them before the actual distillation training process. This preliminary filtering ensures that only high-confidence teacher outputs are used, resolving the contradiction between training efficiency and training quality.
2Quantity of substance
If all training examples from the original dataset are used, then the training data quantity is maximized, but the accuracy and diversity of training data deteriorates due to inclusion of noisy examples
Solution Approach 1:
The patent applies local quality by treating different training examples differently based on their individual uncertainty characteristics. Instead of uniformly processing all training examples, the system calculates uncertainty measures (entropy or margin scores) for each example and applies local filtering or down-weighting only to high-uncertainty examples. This allows the training dataset to maintain high quantity while improving accuracy by selectively removing or reducing the impact of noisy examples.
3Device complexity
If teacher outputs are used directly without filtering, then the distillation process is simple and fast, but the student network performance deteriorates due to learning from incorrect labels
Solution Approach 1:
The patent introduces an intermediary mechanism - the uncertainty measure computation and filtering step - between the teacher network output generation and the student network training process. This intermediary layer calculates entropy or margin scores for each teacher output, identifies unreliable predictions, and filters or down-weights them before passing data to the student network. This adds minimal complexity while significantly improving student network performance by preventing learning from incorrect labels.
Data Source
AI summary
Methods, systems, and apparatus for training a student neural network to perform a task. In one aspect, a method includes obtaining a training dataset including multiple training inputs, generating a filtered training dataset including a subset of the multiple training inputs, comprising, for each of the one or more training inputs, processing the training input using a teacher neural network to generate a respective teacher output for the task, determining an uncertainty measure for the training inputs that represents an uncertainty of the teacher neural network in generating the respective teacher output, and determining whether to filter out the training input based on the uncertainty measure. The method further includes training the student neural network using the filtered training dataset.