Adaptive Standardization Block for Neural Network Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks rely on predefined standardization and rescaling values, which are not adapted to individual inputs, leading to poor performance when processing 'out-of-distribution' data, and require extensive training data and iterations.
Innovation Solution
A normalization block is introduced between neural network layers, dynamically generating adaptive standardization and rescaling values using trainable neural network layers, enabling the network to adapt to individual inputs and improve robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If predefined standardization and rescaling values are used in neural networks, then the network structure remains simple, but the performance deteriorates when processing out-of-distribution data
Solution Approach 1:
The patent implements dynamic standardization and rescaling by replacing fixed predefined values with trainable neural network layers that adaptively generate normalization parameters based on input data characteristics. This allows the normalization block to dynamically adjust its behavior to match the distribution of incoming data, improving reliability on out-of-distribution inputs while managing complexity through structured layer designs.
Solution Approach 2:
The invention changes the parameters from static predefined values to dynamic trainable parameters. The standardization and rescaling values are no longer fixed constants but are generated by neural network layers with learnable weights and biases, enabling the system to adapt parameters based on the specific characteristics of each input batch or individual sample.
2Reliability
If adaptive standardization values are generated using neural network layers, then the robustness to out-of-distribution inputs improves, but the training data requirements increase
Solution Approach 1:
The normalization block performs self-service by automatically adapting its standardization and rescaling parameters based on the statistical properties of the input data it receives. The trainable layers within the normalization block learn to generate appropriate normalization values from the data itself, reducing dependence on large external training datasets for separate normalization strategy optimization.
3Reliability
If adaptive standardization values are generated using neural network layers, then the robustness to out-of-distribution inputs improves, but the training iterations required increase
Solution Approach 1:
The patent merges the normalization functionality with the neural network layers by integrating trainable standardization and rescaling layers directly into the network architecture. This combination allows the normalization parameters to be learned jointly with the main network parameters in a unified training process, potentially reducing total training iterations compared to separate normalization training approaches.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing a network input using a neural network to generate a network output. The neural network includes a normalization block that is between a first neural network layer and a second neural network layer in the neural network. Processing the network input using the neural network comprises: receiving a first layer output from the first neural network layer; processing data derived from the first layer output using standardization neural network layers of the normalization block to generate one or more adaptive standardization values; standardizing the first layer output using the adaptive standardization values to generate a standardized first layer output; generating a normalization block output from the standardized first layer output; and providing the normalization block output as an input to the second neural network layer.