Meta Pseudo-Labels With Adaptive Teacher Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pseudo-labeling techniques for training neural networks suffer from confirmation bias when inaccurate pseudo-labels generated by a teacher neural network lead to suboptimal performance of the student neural network, potentially worse than the teacher.
Innovation Solution
Implement a system where the teacher neural network is adaptively updated based on the student's performance on labeled data, generating improved pseudo-labels through joint training with the student, leveraging unlabeled data for enhanced performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional pseudo-labeling techniques are used to train the student neural network, then the training process can utilize unlabeled data, but the student neural network performance deteriorates when the teacher generates inaccurate pseudo-labels due to confirmation bias
Solution Approach 1:
The patent implements a feedback mechanism where the student network's performance on labeled data is used to update the teacher network's parameters. The teacher objective function includes a term that measures the student's performance on labeled training inputs, and the teacher parameters are updated based on this feedback. This closed-loop feedback system allows the teacher to learn from the student's performance and generate better pseudo-labels over time, resolving the confirmation bias problem where inaccurate pseudo-labels would otherwise degrade student performance.
Solution Approach 2:
The patent makes the teacher network dynamic by allowing its parameters to be updated during the student training process. Instead of using a fixed teacher, the teacher parameters θT are continuously adapted based on the student's performance feedback. This dynamic adjustment enables the teacher to evolve and improve its pseudo-labeling capability throughout training, transforming the static pseudo-labeling approach into a dynamic, adaptive system that can overcome initial inaccuracies.
2Device complexity
If the teacher neural network is fixed during training, then the training process is simpler, but the quality of pseudo-labels remains static and cannot improve based on student performance
Solution Approach 1:
The patent introduces feedback from the student's performance on labeled data to the teacher's parameter updates. The teacher objective function includes a term that evaluates student performance, and this evaluation feeds back into updating the teacher parameters. This feedback loop enables the teacher to adapt and improve pseudo-label quality dynamically, resolving the contradiction between maintaining simplicity and improving accuracy.
Solution Approach 2:
The patent changes the parameters of the teacher network during the training process based on student performance feedback. The teacher parameters θT are updated using gradient descent with an objective function that includes terms related to student performance on labeled data. This parameter adaptation allows the teacher to improve its pseudo-labeling accuracy over time while maintaining a relatively simple training framework that builds upon conventional pseudo-labeling approaches.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using meta pseudo-labels. One of the methods includes training a student neural network using pseudo-labels generated by a teacher neural network that is being trained jointly with the student neural network.