Second Neural Network Training with a Joint Network and Layer Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of knowledge from a first neural network to a second neural network with a different architecture is hindered by data access restrictions, data privacy regulations, and the routine purging of training data, making it challenging and costly to scale and maintain decentralized AI models.
Innovation Solution
A method involving a joint neural network constructed from parts of the first and second neural networks, trained using a layer loss function and self-supervised learning, allows knowledge transfer without direct access to the first neural network's training data, leveraging similarities in network layers for efficient training of the second neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the first neural network is trained in a decentralized environment with data privacy regulations and routine data purging, then data security and privacy are improved, but knowledge transfer to the second neural network becomes difficult
Solution Approach 1:
The patent introduces a joint neural network as an intermediary that bridges the first and second neural networks. This joint network is trained on a combination of first training data and second training data, serving as a mediator that enables knowledge transfer without requiring direct access to the original training data of the first network. The joint network's weights are then used to initialize or guide the training of the second network, facilitating knowledge transfer while maintaining data privacy constraints.
2Adaptability or versatility
If the original training data is restricted or unavailable due to technical constraints, then data privacy compliance is improved, but model training and knowledge transfer become challenging
Solution Approach 1:
The patent performs preliminary training of a joint neural network using a combination of first training data and second training data before the actual deployment of the second neural network. This preliminary action creates a pre-trained joint network that encapsulates knowledge from both data sources, which can then be used to initialize or guide the second network's training, reducing the need for direct access to the restricted first training data during the actual model deployment phase.
Solution Approach 2:
The patent creates a joint neural network that copies and integrates knowledge from both the first training data and second training data. The joint network serves as a copied representation of the combined knowledge, which can then be transferred to the second neural network without requiring access to the original first training data, thus maintaining privacy compliance while enabling knowledge transfer.
3Quantity of substance
If data is gathered from multiple sites across various geographical locations, then model training data availability is improved, but time consumption and technical complexity increase
Solution Approach 1:
The patent merges the first training data and second training data into a unified training process for the joint neural network. By combining both data sources in a single training operation rather than separately gathering and processing them, the method reduces the time and technical complexity associated with data gathering from multiple sites, while still achieving comprehensive model training.
4Productivity
If data is stored at a central location for model training, then centralized training efficiency is improved, but storage costs and data access security risks increase
Solution Approach 1:
The joint neural network acts as an intermediary that consolidates knowledge from distributed data sources without requiring the physical centralization of the actual training data. By training the joint network on combined data and then using its weights to guide the second network, the system achieves centralized training efficiency while avoiding the high storage costs and security risks associated with centralizing the original training datasets.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The invention relates to providing a second neural network. It is based on receiving a first neural network trained to map first input data to first output data, and on receiving the second neural network configured to map second input data to second output data. The second input data has the same structure as the first input data. It is furthermore based on determining a joint neural network comprising a first part of the first neural network and a second part of the second neural network. It is furthermore based on receiving first training data and second training data. It is furthermore based on training the joint neural network based on the first training data, and on training the second neural network based on the second training data and a layer loss function. The layer loss function is based on a comparison of values of a second layer of the second part in the second neural network and values of a corresponding layer in the trained joint neural network. Furthermore, it is based on providing the second neural network.