Neural Network Further Training Using Data Subset Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in efficiently updating their training data without requiring extensive retraining or access to the complete original dataset, especially in applications like traffic sign recognition, where new data needs to be integrated without compromising existing knowledge.
Innovation Solution
A method for further training neural networks using a subset of previous training examples and new data, leveraging generative models to generate equivalent training examples, optimizing parameters to improve performance on new data without worsening performance on old data, and using gradient calculations to monitor and control changes, thus preventing 'catastrophic forgetting'.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complete retraining on the union set M∪B is performed, then the neural network can recognize new objects, but the computing effort and time required become excessively high
Solution Approach 1:
The training process is segmented into two phases: initial training on set M, and subsequent further training on batch B using a selected subset D. This segmentation allows the system to adapt to new traffic signs without performing complete retraining on the entire union set M∪B, significantly reducing training time while maintaining adaptability.
Solution Approach 2:
A subset D of previous training examples is pre-selected and stored before the further training process. This preliminary action enables the further training to proceed efficiently with a manageable subset of data, avoiding the need to process the complete original dataset M during updates.
2Adaptability or versatility
If complete retraining on the union set M∪B is performed, then the neural network can recognize new objects, but the computational resources and energy consumption increase significantly
Solution Approach 1:
The computational workload is segmented by separating the training into initial training on set M and further training on batch B with subset D. This segmentation reduces the computational energy required for updates, as the system only needs to process the smaller subset D rather than the complete union set M∪B.
Solution Approach 2:
The essential portion of the original training data is extracted into subset D, which is then used for further training. This extraction allows the system to maintain adaptability while minimizing computational energy consumption by working with a reduced dataset.
3Quantity of substance
If a small subset D of previous training examples is used, then memory space requirements are reduced, but access to the complete set M is not required
Solution Approach 1:
Different portions of the training data serve different functions: subset D is used for efficient further training and stored in limited onboard memory, while the complete set M remains available for comprehensive evaluation. This local quality assignment allows the system to operate within memory constraints while preserving the ability to maintain performance on all original classes.
Solution Approach 2:
The system incorporates feedback mechanisms to monitor performance on both subset D and the complete set M. This feedback ensures that further training on the smaller subset does not degrade performance on the complete dataset, thereby preserving existing knowledge while adapting to new traffic signs.
4Measurement precision
If further training is performed to improve recognition of new objects, then performance on batch B improves, but performance on previous examples may deteriorate due to catastrophic forgetting
Solution Approach 1:
The system uses feedback from performance evaluation on both subset D and batch B to guide the further training process. By monitoring accuracy on previous examples during further training, the system can adjust parameters to prevent catastrophic forgetting, ensuring that improvement on new traffic signs does not come at the cost of degradation on previous ones.
Solution Approach 2:
The training process is made dynamic by allowing the system to adaptively balance learning from new examples while retaining knowledge from previous examples. The further training on subset D is designed to be dynamic in nature, adjusting the influence of new versus old data to maintain overall performance.
Data Source
AI summary
A method for further training of a neural network for processing measurement data, which neural network has been pre-trained with training examples from a set M. In the method: a batch B of new training examples is provided; a subset D⊆M of the previous training examples is provided; the new training examples from batch B and the previous training examples from subset D are processed by the neural network into outputs respectively; the deviations of the outputs from the respective target outputs are evaluated using a predefined cost function; parameters characterizing the behavior of the neural network are optimized with the aim that, during further processing of previous and new training examples, the evaluation with the cost function is improved in regard to new training examples from batch B and is not made worse in regard to previous training examples from subset D.