On-Device Neural Network Continual Learning via Synthetic Data Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
On-device learning of deep learning networks is hindered by storage constraints and the catastrophic forgetting problem when new training data differs from past data, especially in environments where data cannot be transmitted to servers due to privacy concerns or lack of communication networks.
Innovation Solution
A method for on-device continual learning that involves sampling new data to generate synthetic previous data, creating batches for current-learning, and using generative adversarial networks to update the neural network without storing past training data, thereby preventing catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If on-device learning is performed using new training data, then the device can learn autonomously without server communication, but the device forgets previously learned information (catastrophic forgetting)
Solution Approach 1:
The patent creates a copy of the previous training data distribution using a generative model (GAN). The generator network learns to synthesize training data that mimics the statistical properties of historical data, allowing the system to retain knowledge of past data distributions without storing the actual historical data. This copied representation prevents catastrophic forgetting by providing continuous exposure to synthetic versions of previous data patterns.
Solution Approach 2:
The system performs preliminary action by pre-training a generative model on historical data before the actual continual learning begins. This generator is trained beforehand to capture the essence of past data distributions, so when new data arrives, the system already has a prepared mechanism to synthesize and incorporate previous knowledge patterns, preventing information loss during the learning process.
2Reliability
If the device stores all past training data for continual learning, then catastrophic forgetting can be prevented, but the device runs out of storage space
Solution Approach 1:
Instead of storing actual historical training data, the system creates a compact generative model that copies the essential characteristics of past data. This generator network serves as a compressed representation that can synthesize unlimited amounts of synthetic training data, providing the same knowledge retention benefits as storing all original data but using minimal storage space.
Solution Approach 2:
The system transforms the storage requirement from storing vast amounts of raw training data to storing a much smaller set of model parameters that define the generative process. By changing the representation from data-centric to model-centric, the system achieves efficient storage while maintaining the ability to reproduce and learn from historical data patterns indefinitely.
3Measurement precision
If deep learning networks are trained on servers with large-scale data, then recognition performance improves, but personal data cannot be transmitted due to privacy concerns or lack of network connectivity
Solution Approach 1:
The patent extracts the learning capability from the server environment and places it directly on the edge device. By implementing the generative model and continual learning algorithms locally on the device, the system eliminates the need to transmit personal data to servers while maintaining the ability to achieve high recognition performance through autonomous local learning and adaptation.
Data Source
AI summary
A method for on-device continual learning of a neural network which analyzes input data is provided to be used for smartphones, drones, vessels, or a military purpose. The method includes steps of: a learning device, (a) sampling new data to have a preset first volume, instructing an original data generator network, which has been learned, to repeat outputting synthetic previous data corresponding to a k-dimension random vector and previous data having been used for learning the original data generator network, such that the synthetic previous data has a second volume, and generating a batch for a current-learning; and (b) instructing the neural network to generate output information corresponding to the batch. The method can be performed by generative adversarial networks (GANs), online learning, and the like. Also, the present disclosure has effects of saving resources such as storage, preventing catastrophic forgetting, and securing privacy.