Neural Network Mimicry via Random Probe Data Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training an accurate neural network is complex and time-consuming, and existing methods require access to the original training dataset to replicate or modify a pre-trained neural network, making it difficult to incorporate new information or remove old data without retraining from scratch.
Innovation Solution
A method to train a new neural network to mimic a pre-trained target network by probing it with random or semi-random input data, generating a random probe training dataset, and using this dataset to replicate the target network's behavior without access to the original training data, allowing for modification and improvement of the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional neural network training methods are used, then model accuracy can be achieved, but access to original training data is required which compromises data security and privacy
Solution Approach 1:
The patent creates a copy (student network) of the target network by training it on synthetically generated data that mimics the statistical properties of the original training data. This copy reproduces the target network's behavior and accuracy without requiring access to the original sensitive training data, thus resolving the contradiction between model accuracy and data security.
Solution Approach 2:
The patent introduces an intermediary process - synthetic data generation - that acts as a mediator between the target network and the student network. Instead of directly sharing original training data, the system generates intermediate synthetic data that preserves the essential statistical characteristics needed for training, thereby enabling knowledge transfer while maintaining data security.
2Adaptability or versatility
If the entire training dataset is retrained to incorporate new information or remove old data, then the neural network can be updated, but the process is time-consuming and requires access to the complete original dataset
Solution Approach 1:
The patent extracts only the necessary statistical properties and patterns from the training process that are needed to maintain and update the network. By identifying and extracting these essential characteristics, the system can update the student network with new information without requiring the entire original dataset, significantly reducing update time while maintaining adaptability.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and characterizing the training data distribution before actual training. This preliminary characterization creates a foundation that enables future updates to be performed more efficiently, as the statistical framework is already established and can be incrementally adjusted without complete retraining.
3Reliability
If a neural network is trained with billions of calculations to achieve high accuracy, then model performance is improved, but the training process takes hours, days or even weeks to complete
Solution Approach 1:
The patent applies partial action by training the student network on a synthesized subset of data that captures the essential statistical properties of the complete training dataset. Instead of requiring all billions of calculations on the full dataset, the synthetic data generation process creates a condensed representation that achieves similar training effectiveness with reduced computational burden and time.
Data Source
AI summary
A device, system, and method is provided to mimic a pre-trained target model without access to the pre-trained target model or its original training dataset. A set of random or semi-random input data may be sent to randomly probe the pre-trained target model at a remote device. A set of corresponding output data may be received from the remote device that is generated by applying the pre-trained target model to the set of random or semi-random input data. A random probe training dataset may be generated comprising the set of random or semi-random input data and corresponding output data generated by randomly probing the pre-trained target model. A new model may be trained with the random probe training dataset so that the new model generates substantially the same corresponding output data in response to said input data to mimic the pre-trained target model.