Artificial Neural Network Knowledge Transfer via Synthetic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for transferring knowledge between artificial neural networks (ANNs) are limited, as they require identical architectures and are not flexible, especially when dealing with ANNs of different depths, and they do not efficiently handle the privacy constraints of sensitive data sets.
Innovation Solution
A method and system for generating training data by injecting random samples into a trained ANN to replicate input samples, reinjecting pseudo samples, and repeating this process to create a set of pseudo samples and pseudo labels, which can be used to train a new ANN, allowing knowledge transfer across different architectures and preserving data privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If transfer learning copies parameters from a trained ANN to a second ANN, then training time is reduced and hardware resources are saved, but the architectures of both ANNs must be identical which limits flexibility
Solution Approach 1:
The patent generates synthetic training data by copying and transforming output samples from a trained source ANN to create training inputs for a target ANN. This allows knowledge transfer without direct parameter copying, enabling architecture flexibility while maintaining training efficiency benefits.
Solution Approach 2:
The patent transforms the parameters by generating synthetic training data with modified inputs and corresponding outputs from the source ANN. This parameter transformation enables the target ANN to learn from the source ANN's knowledge while adapting to different architectures.
2Ease of manufacture
If the same training dataset is used to train multiple ANNs, then knowledge transfer is simplified, but hardware resources for data storage increase
Solution Approach 1:
The patent creates synthetic training data by copying and transforming samples from the source ANN's outputs rather than storing multiple copies of the original training dataset. This generates sufficient training data for multiple ANNs without proportionally increasing storage requirements.
Solution Approach 2:
The synthetic training data generation process creates a universal data source that can train multiple ANNs with different architectures. The same source ANN generates training data that serves multiple target models, reducing the need for separate training datasets.
3Adaptability or versatility
If random samples are reinjected multiple times into the trained ANN, then synthetic training data diversity increases, but processing time increases
Solution Approach 1:
The patent applies periodic reinjection of random samples through the trained ANN to generate diverse synthetic training data. By controlling the number of reinjection iterations, the system balances data diversity generation with processing time constraints.
Solution Approach 2:
The patent uses a limited number of reinjection iterations (partial action) rather than exhaustive sampling. This generates sufficient training data diversity without the excessive processing time that would result from unlimited reinjections.
Data Source
AI summary
A method of generating training data for transferring knowledge from a trained artificial neural network to a further artificial neural network, the method including: a) injecting a first sample into the trained artificial neural network; b) reinjecting a pseudo sample, generated based on a replicated sample present at the one or more outputs of the trained artificial neural network, into the trained artificial neural network in order to generate a new replicated sample; and c) repeating b) one or more times, wherein the training data for training the further artificial neural network includes at least two of the reinjected pseudo samples originating from the same first sample and corresponding output values generated by the trained artificial neural network.