Neural Network Layer With Percentile-Based Non-Linear Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional normalization techniques for deep neural networks are limited by their linear nature, which makes them susceptible to outliers and ineffective in handling skewed or multi-modal distributions, leading to instability in training and performance issues.
Innovation Solution
A non-linear normalization method using percentile estimators that are robust to outliers, allowing for better handling of skewed and heavy-tailed distributions by mapping empirical percentiles to a predefined probability distribution, such as the standard normal distribution, and incorporating smoothing operations to reduce noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If conventional linear normalization techniques are used, then training stability is improved through normalization of forward signal flow and backward gradient flow, but the system becomes susceptible to outliers and ineffective with skewed or multi-modal distributions
Solution Approach 1:
The patent changes the normalization parameters from linear statistics (mean and standard deviation) to non-linear percentile-based statistics. By using percentiles (e.g., 2nd and 98th percentiles) instead of mean and std, the system becomes robust to outliers while maintaining training stability. The non-linear transformation function maps these percentile-based statistics to normalize the data effectively.
Solution Approach 2:
The patent substitutes the conventional linear normalization mechanism with a non-linear normalization mechanism. Instead of using linear operations (subtracting mean and dividing by std), the system employs non-linear transformations based on percentile rankings and custom transformation functions, replacing the mechanical linear approach with a more sophisticated non-linear system.
2Manufacturing precision
If linear normalization mapping is applied, then values are transformed to have predefined mean and variance, but the system cannot handle skewed, heavy-tailed, or multi-modal distributions effectively
Solution Approach 1:
The patent changes from fixed linear parameters (mean=0, variance=1) to adaptive non-linear parameters based on percentile distributions. The system calculates percentiles from the data and uses these to create adaptive normalization that works with various distribution types including skewed, heavy-tailed, and multi-modal distributions.
Solution Approach 2:
The patent introduces dynamic adaptation to different distribution types. Instead of a static linear normalization, the system dynamically adjusts its normalization approach based on the actual data distribution characteristics, making it versatile across different distribution types while maintaining precision.
3Reliability
If non-linear percentile-based normalization is used, then robustness against outliers and skewed distributions is improved, but computational complexity increases
Solution Approach 1:
The patent uses a practical approximation by focusing on specific key percentiles (e.g., 2nd and 98th) rather than computing the complete non-linear transformation for all data points. This partial action approach maintains robustness against outliers while reducing computational complexity compared to full non-linear normalization.
Solution Approach 2:
The patent employs computationally efficient approximations that are easier to compute than full non-linear transformations. By using simplified percentile-based calculations and approximate transformation functions, the system achieves robustness with lower computational cost, effectively using simpler 'cheaper' computational objects.
Data Source
AI summary
A computer-implemented machine learning system. The machine learning system is configured to provide an output signal based on an input signal by forwarding the input signal through a plurality of layers of the machine learning system. At least one of the layers of the plurality of layers is configured to receive a layer input, which is based on the input signal, and to provide a layer output based on which the output signal is determined. The layer is configured to determine the layer output by means of a non-linear normalization of the layer input.