Neural Network Training via Self-Supervised Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods struggle with efficiently training small networks for unsupervised visual representation learning, especially in resource-constrained environments, and often require post-processing steps like k-means clustering.
Innovation Solution
An unsupervised representation learning framework that integrates deep self-supervised representation learning and knowledge distillation within a single training process, using a teacher-student model architecture where the teacher model learns consistent clustering and the student model mimics the teacher's predictions, aided by a domain classifier and dropout-based multi-view generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If small neural networks are trained for unsupervised visual representation learning, then model size and computational resources are reduced, but training efficiency and performance achieve are insufficient
Solution Approach 1:
The patent introduces an intermediate representation learning framework that acts as a mediator between raw unsupervised data and final model training. The framework uses self-supervised pre-training on large-scale unsupervised data to create meaningful intermediate representations, which then serve as effective training data for smaller networks. This intermediary step enables small networks to achieve high performance without requiring extensive computational resources or complex training procedures.
2Ease of manufacture
If traditional unsupervised training methods are used, then no labeled data is required, but additional post-processing steps like k-means clustering are needed
Solution Approach 1:
The patent merges the representation learning objective with the clustering objective into a unified training framework. Instead of separately performing unsupervised representation learning and then applying k-means clustering, the method combines these operations by using cluster assignments as soft labels during the representation learning process itself. This integration eliminates the need for separate post-processing clustering steps while maintaining the benefits of unsupervised learning.
3Measurement precision
If offline distillation is used, then teacher model knowledge is transferred to student model, but training time and computational overhead increase
Solution Approach 1:
The patent performs preliminary self-supervised pre-training of the teacher model on large-scale unsupervised data before the distillation process. This preliminary action creates a well-prepared teacher model with high-quality representations that can be efficiently distilled. By preparing the teacher model in advance with meaningful unsupervised representations, the subsequent distillation process to student models becomes more efficient and requires less computational overhead compared to traditional offline distillation methods.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A processor-implemented method with neural network training includes: determining first backbone feature data corresponding to each input data by applying, to a first neural network model, two or more sets of the input data of the same scene, respectively; determining second backbone feature data corresponding to each input data by applying, to a second neural network model, the two or more sets of the input data, respectively; determining projection-based first embedded data and dropout-based first view data from the first backbone feature data; and determining projection-based second embedded data and dropout-based second view data from the second backbone feature data; and training either one or both of the first neural network model and the second neural network model based on a loss determined based on a combination of any two or more of the first embedded data, the first view data, the second embedded data, the second view data, and an embedded data clustering result.