Neural Network Training with Grid-Based Equivariant Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art neural networks are over-parameterized and require vast amounts of data for training, necessitating a more efficient and sample-efficient training procedure.
Innovation Solution
A method for training an artificial neural network by organizing output channels into spatial grids and applying normalization functions that depend on spatial relationships and trainable distance parameters, using equivariant feature mapping to learn data symmetries without explicit knowledge of transformation representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If state-of-the-art neural networks are used with over-parameterization, then model capacity and representational power are improved, but data requirements and training complexity increase significantly
Solution Approach 1:
The patent segments the channel dimension into multiple groups, where each group is independently normalized. This segmentation allows the model to capture different statistical properties of different channel groups, effectively increasing model capacity without requiring proportional increases in training data
Solution Approach 2:
The patent introduces learnable parameters (gamma and beta) for each channel group that can be optimized during training. These parameters dynamically adjust the normalization behavior, enabling the model to adapt to different data distributions without requiring vast amounts of training data
2Reliability
If fixed groups of channels are normalized independently (Group Normalization), then training stability is improved, but sample efficiency and generalization accuracy are limited
Solution Approach 1:
The patent transforms the static group normalization approach into a dynamic one by introducing learnable parameters and attention mechanisms. The normalization behavior can adapt during training based on the input data, improving both sample efficiency and generalization while maintaining training stability
Solution Approach 2:
The patent incorporates feedback mechanisms where the normalization statistics from previous layers influence subsequent normalization operations. This feedback loop allows the model to learn from training examples more efficiently, improving sample efficiency without sacrificing training stability
3Measurement precision
If vast amounts of data are used for training, then model accuracy is improved, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary normalization operations on channel groups before the main processing stages. By pre-processing the data in this manner, the model converges faster during training, reducing the total training time required to achieve high generalization accuracy
Solution Approach 2:
The patent applies different normalization strategies to different channel groups based on their local characteristics. This localized approach allows the model to learn more efficiently from limited data, achieving high accuracy with reduced training time compared to uniform processing approaches
Data Source
Figure 1~2
Figure 3a~3c
AI summary
Device (100) and computer implemented method for training an artificial neural network (106), wherein the device (100) comprises an input (108) for data, an output (110) for an output signal, at least one processor (102) and memory (104) for an artificial neural network (106) that are adapted to execute the method providing a neural network layer for an equivariant feature mapping having a plurality of output channels, in particular a convolutional neural network layer, grouping (204) channels of the plurality of output channels of the neural network layer into a number of distinct groups, wherein the output channels of each individual distinct group are organized into an individual grid defining a spatial location of each of the output channels of the individual distinct group in the grid for the individual distinct group, providing (206) for each of the output channels of each individual distinct group of the number of distinct groups a distinct normalization function, wherein the distinct normalization function is defined depending on the spatial location of the output channel in the grid in that this output channel is organized and depending on tunable hyperparameters for the normalization function, determining (208) an output of the artificial neural network (106) depending on a result of each of the distinct normalization functions, training (210) the hyperparameters in a training of the artificial neural network (106).