Neural Network Training Data Sampling and Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods face challenges in ensuring reliability, particularly in safety-critical applications, due to issues like overfitting, variable training data sizes, and inconsistent data sampling, which can lead to reduced generalizability and performance.
Innovation Solution
The method involves training neural networks by drawing pairs from the entire training data set with replacement, allowing for augmentation and probabilistic gradient refinement, and using a scaling layer to adapt parameters based on layer depth or receptive field, ensuring reliable gradient direction and reducing overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pairs are drawn from the entire training data set with replacement, then reliability and consistency of training are improved, but training time and computational resources increase
Solution Approach 1:
The patent applies preliminary action by pre-shuffling the training data set once at the beginning of training, then systematically drawing pairs with replacement from this shuffled set. This preliminary shuffling ensures consistent randomization without requiring repeated shuffling operations during training, thereby improving reliability while controlling training time.
Solution Approach 2:
The patent uses copying by creating multiple copies of the training data through sampling with replacement. Instead of processing the entire data set sequentially multiple times, pairs are drawn with replacement, effectively creating virtual copies of training examples. This approach maintains reliability through consistent data distribution while reducing the computational burden of processing unique examples repeatedly.
2Adaptability or versatility
If data augmentation is applied to expand the training data set, then generalizability is improved, but training complexity and processing time increase
Solution Approach 1:
The patent applies partial action by selectively applying data augmentation only to certain training examples rather than all examples. The system determines which training examples benefit most from augmentation and applies transformations only to those cases. This selective approach improves generalizability for critical examples while avoiding the computational overhead of augmenting every single training example.
3Productivity
If epoch-based sampling without replacement is used, then training efficiency is improved, but mathematical convergence requirements are not met
Solution Approach 1:
The patent resolves the contradiction by performing preliminary shuffling of the training data set before training begins. This single shuffling operation satisfies the randomness requirement for convergence proofs, while the subsequent epoch-based sampling without replacement maintains training efficiency. The preliminary randomization ensures that each epoch starts with a different random ordering, providing the statistical properties needed for convergence while avoiding the inefficiency of repeated shuffling.
Data Source
AI summary
A computer-implemented method for training a neural network, which, in particular, is configured to classify physical measuring variables. The neural network is trained with the aid of a training data set. Pairs including an input signal and an associated desired output signal are drawn from the training data set for training. An adaptation of parameters of the neural network occurs as a function of an output signal of the neural network, when the input signal is supplied, and as a function of the desired output signal. The drawing of pairs always takes place from the entire training data set.