Neural Network Scaling Layer for Training Data Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods face challenges in efficiently classifying input signals due to issues with data augmentation, batch normalization, and the complexity of hyperparameter tuning, which can lead to overfitting and reduced generalizability.
Innovation Solution
The introduction of a scaling layer in the neural network that maps input signals onto a predeterminable value range, defined by a norm, simplifies the architecture search and training process by projecting inputs onto a ball with a fixed center and radius, thereby controlling the scale of input signals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data augmentation is used to expand the training data set, then the training data diversity is improved, but the statistical consistency of the training process deteriorates
Solution Approach 1:
The patent introduces a statistical model as an intermediary that captures the distributional characteristics of augmented data. This model serves as a mediator between the augmented training data and the learning algorithm, providing statistically consistent samples without requiring direct use of all augmented data, thus maintaining both diversity and statistical consistency
Solution Approach 2:
The patent transforms the training approach by changing from direct epoch-based sampling of augmented data to sampling from a statistical model with specific distributional parameters. This parameter-based approach maintains statistical consistency while preserving the diversity benefits of data augmentation
2Ease of operation
If epoch-based sampling is used for training, then the training process is simplified, but the mathematical convergence assumptions are violated
Solution Approach 1:
The patent introduces a statistical model as an intermediary layer between the simple epoch-based sampling process and the convergence requirements. This model ensures that even though sampling is done without replacement in epochs, the statistical properties needed for convergence are maintained through the model's distributional assumptions
Solution Approach 2:
The statistical model is designed to automatically handle the convergence requirements through its inherent distributional properties, without requiring complex adjustments to the training loop. The model self-corrects the statistical inconsistencies introduced by epoch-based sampling
3Reliability
If the number of training data points is increased to ensure convergence, then the convergence reliability is improved, but the training time is extended
Solution Approach 1:
The patent creates a statistical copy or representation of the training data distribution that can be sampled efficiently. Instead of using all N training points repeatedly, the statistical model captures the essential distributional characteristics, allowing convergence to be achieved with fewer actual data point evaluations while maintaining theoretical guarantees
4Measurement precision
If evaluation data set is used to determine convergence quality, then the training monitoring is improved, but the over-fitting risk increases
Solution Approach 1:
The patent uses the statistical model as an intermediary for monitoring training progress. Instead of directly evaluating on a separate test set that might lead to overfitting, the convergence is monitored through the statistical model's fit to the training data distribution, providing a more reliable indicator of generalization performance
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for providing an actuation signal (A) for actuating an actuator (10), wherein the actuation signal (A) is ascertained on the basis of an output signal (y) from a neural network (60), characterized in that the neural network (60) comprises a scaling layer (S 4), wherein the scaling layer maps an input signal (z 4) applied to the input of the scaling layer (S 4) to an output signal (z 5) applied to the output of the scaling layer (S 4) such that this mapping corresponds to a projection of the input signal (z 4) onto a prescribable range of values, wherein parameters (ρ, c) that characterize the mapping are prescribable.