Second Neural Network Transfer Using Joint Layer Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of transferring knowledge from an established neural network trained in a decentralized environment to a new neural network with a different architecture, where access to the original training data is restricted due to data privacy regulations and routine purging practices, is addressed.
Innovation Solution
A method involving constructing a joint neural network comprising parts of the first and second neural networks, using a layer loss function to train the second neural network without requiring complete access to the first network's training data, and employing self-supervised learning to enhance training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If knowledge transfer is performed using original training data from the first neural network, then transfer accuracy is improved, but data privacy compliance deteriorates
Solution Approach 1:
The invention extracts only the essential knowledge patterns and relationships from the first neural network's training process, rather than using the original training data itself. This is achieved by training a joint neural network on synthetic data that captures the underlying distributions and relationships, thereby transferring knowledge without accessing or exposing sensitive original data.
Solution Approach 2:
The invention introduces a joint neural network as an intermediary mechanism that facilitates knowledge transfer. This joint network is trained on synthetic data generated to match the statistical properties of the original training data, serving as a mediator that transfers knowledge patterns without requiring direct access to the sensitive original data.
2Object-affected harmful factors
If knowledge transfer is performed without original training data, then data privacy compliance is improved, but transfer accuracy deteriorates
Solution Approach 1:
The invention performs preliminary actions by generating synthetic training data that replicates the statistical properties, distributions, and relationships of the original training data before the knowledge transfer process. This synthetic data is created in advance to capture the essential patterns needed for accurate knowledge transfer, ensuring both privacy compliance and transfer accuracy.
Solution Approach 2:
The invention changes the parameters and properties of the training data by using synthetic data generation techniques. The synthetic data is designed to match key statistical parameters (distributions, correlations, relationships) of the original data while containing no actual sensitive information, thereby maintaining transfer accuracy without compromising privacy.
3Reliability
If data is collected from multiple decentralized sites, then model performance is improved, but system complexity deteriorates
Solution Approach 1:
The invention creates copies of the training data in the form of synthetic data that replicates the statistical properties and relationships of the original decentralized data. Instead of physically collecting and centralizing data from multiple sites, the system generates synthetic copies that capture the essential patterns, thereby improving model performance without the complexity of distributed data collection and management.
4Measurement precision
If extensive data collection is performed, then knowledge transfer quality is improved, but cost deteriorates
Solution Approach 1:
The invention uses synthetic data as a disposable alternative to extensive original training data. The synthetic data can be regenerated as needed and does not require long-term storage of large volumes of sensitive original data. This approach maintains knowledge transfer quality while significantly reducing data storage costs and eliminating the need for expensive data management infrastructure.
Data Source
AI summary
A computer-implemented method comprises: receiving a first neural network trained to map first input data to first output data; receiving a second neural network configured to map second input data to second output data, the second input data having a same structure as the first input data; determining a joint neural network including a first part of the first neural network and a second part of the second neural network; receiving first and second training data; training the joint neural network based on the first training data; training the second neural network based on the second training data and a second loss function, the second loss function including a layer loss function based on a comparison of values of a second layer of the second part in the second neural network and values of a corresponding layer in the trained joint neural network; and providing the second neural network.