ML Hyperparameter Search to Prevent Training Data Memorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems, particularly neural networks, tend to memorize training data rather than generalizing well, which is a concern for security-critical applications like automated driving due to the inability to effectively learn the underlying structure between input data and classifications.
Innovation Solution
Automated determination of architecture and optimization parameters through a parameter search method, optimizing hyperparameters to ensure the machine learning system generalizes better by selecting values that result in a specific distribution of margins, preventing memorization of training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the machine learning system is trained with many parameters to fit the training data, then the training accuracy is improved, but the generalization capability deteriorates due to memorization
Solution Approach 1:
The patent applies parameter changes by systematically varying hyperparameters (learning rate, batch size, network architecture parameters) during the training process to find optimal values that prevent memorization. The parameter search method explores different parameter configurations to identify settings that achieve both good training performance and generalization, directly addressing the contradiction between fitting training data and maintaining reliability on new data.
2Adaptability or versatility
If the network capacity is increased to learn complex patterns, then the learning capability is improved, but the risk of memorization increases
Solution Approach 1:
The patent implements dynamics by making the network architecture adaptive during training through techniques like dynamic filter selection and progressive unfreezing of layers. The system dynamically adjusts which parameters are trainable at different training stages, allowing the network to learn complex patterns when needed while preventing memorization by controlling capacity utilization. This dynamic approach enables the network to adapt its effective capacity based on the complexity of the task and the quality of training data.
Data Source
AI summary
A method for parameterizing a machine learning system, in particular a neural network, that is configured to ascertain in each case an associated class of a plurality of classes from input data. The machine learning system is trained once with correctly labeled training data, and once with not correctly labeled training data. Hyperparameters of the machine learning system are selected in such a way that the particular trained machine learning system may reproduce actual classifications of the correctly labeled training data better than actual classifications of the not correctly labeled training data.