Smooth Piecewise Activation Functions for Reproducible Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks using Rectified Linear Unit (ReLU) activation functions suffer from irreproducibility due to non-continuous gradients, leading to significant prediction differences in models trained on identical data sets when the training order or examples are varied.
Innovation Solution
Implementing smooth, piecewise activation functions with continuous gradients and transition points, such as the Generalized Leaky Smooth Rectified Linear Unit (SmeLU), which mimic the behavior of ReLU while ensuring continuity and monotonicity, thereby improving reproducibility and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If ReLU activation functions are used in neural networks, then computational efficiency and simplicity are improved, but gradient continuity deteriorates leading to irreproducibility
Solution Approach 1:
The activation function is divided into multiple piecewise segments (first segment for negative inputs, second segment for positive inputs) with a transition region in between. This segmentation allows each segment to have simple linear behavior for computational efficiency while the transition region ensures gradient continuity for reliability.
Solution Approach 2:
A transition region is introduced as an intermediary between the two piecewise segments. This transition region contains intermediate points that smoothly connect the segments and ensure continuous gradients, acting as a mediator that resolves the conflict between simplicity and continuity.
2Reliability
If smooth piecewise activation functions with transition regions are implemented, then gradient continuity and reproducibility are improved, but computational complexity increases
Solution Approach 1:
The transition region, which adds computational complexity, is localized only where needed (at the boundary between positive and negative regions) rather than applying complex operations throughout the entire domain. This keeps the majority of computations simple while achieving gradient continuity only where necessary.
Solution Approach 2:
Instead of making the entire activation function complex, only a partial region (the transition region with intermediate points) is made complex to achieve gradient continuity, while the majority of the function remains simple linear segments.
3Stability of the object's composition
If piecewise segments are made continuous at transition points, then smoothness is improved, but the number of constraints on parameters increases
Solution Approach 1:
The patent transforms the problem of satisfying multiple continuity constraints into a parameter transformation process. By defining the activation function in terms of transformed parameters (scaling and shifting the basic SmeLU function), the constraints are automatically satisfied through the transformation rather than requiring direct solving of constraint equations.
Data Source
AI summary
Aspects of the present disclosure are directed to novel activation functions which enable improved reproducibility and accuracy tradeoffs in neural networks. In particular, the present disclosure provides a family of activation functions that, on one hand, are smooth with continuous gradient and optionally monotonic but, on the other hand, also mimic the mathematical behavior of a Rectified Linear Unit (ReLU). As examples, the activation functions described herein include a smooth rectified linear unit function and also a leaky version of such function. In various implementations, the proposed functions can provide both a complete stop region and a constant positive gradient (e.g., that can be 1) pass region like a ReLU, thereby matching accuracy performance of a ReLU. Additional implementations include a leaky version and/or functions that feature different constant gradients in the pass region.