Near-Linear Activation for Low-Loss Autoencoder Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional autoencoders (AEs) face challenges in anomaly detection due to overfitting when using linear activation functions, which fail to effectively identify critical features, and non-linear AEs filter out important features for anomaly detection, leading to higher reconstruction loss.
Innovation Solution
A near linear activation function is introduced, with a predominant linearly-sloped middle segment and small non-linearities at the boundaries, minimizing reconstruction loss and reducing overfitting, allowing for effective feature identification and anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If linear activation functions are used in autoencoders, then reconstruction loss is minimized, but overfitting occurs and critical features are not effectively identified
Solution Approach 1:
The patent applies local quality by making different parts of the activation function have different properties: the middle segment is linear to minimize reconstruction loss, while the boundary regions introduce non-linearities to reduce overfitting. This localized differentiation allows the activation function to simultaneously achieve good reconstruction and generalization performance.
Solution Approach 2:
The patent changes the parameters of the activation function by introducing a piece-wise structure with different linear segments and non-linear boundary regions. By adjusting the slope and curvature parameters in different regions, the function optimizes both reconstruction accuracy and resistance to overfitting.
2Reliability
If non-linear activation functions are used in autoencoders, then overfitting is reduced, but reconstruction loss increases and important features are filtered out
Solution Approach 1:
The patent applies partial action by introducing non-linearities only at the boundaries of the activation function rather than across the entire range. This partial non-linearity is sufficient to reduce overfitting while maintaining the linear behavior in the middle region that preserves reconstruction quality and important feature representation.
3Ease of manufacture
If conventional autoencoders are used for anomaly detection, then the system is simple to implement, but accuracy in identifying anomalies is reduced
Solution Approach 1:
The patent changes the activation function parameters to create a piece-wise structure with linear and non-linear regions. This parameter modification improves anomaly detection accuracy by better preserving critical features while maintaining a relatively simple autoencoder architecture that is still easy to implement.
Data Source
AI summary
Examples of the presently disclosed technology provide a “near linear” activation function for an autoencoder (AE). The “near linear” activation function may comprise a piecewise function comprising: (1) a linearly-sloped middle segment spanning a majority of a domain of the piece-wise near linear activation function; (2) a first end segment with a different slope than the linearly-sloped middle segment, wherein the first end segment commences at a lower boundary of the domain of the piece-wise near linear activation function and terminates at a first end of the linearly-sloped middle segment; and (3) a second end segment with a different slope than the linearly-sloped middle segment, wherein the second end segment commences at a second end of the linearly-sloped middle segment and terminates at an upper boundary of the domain of the piece-wise near linear activation function.