Neural Network Anomaly Detection via Gaussianization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection techniques using statistical hypothesis testing fail to detect anomalies in a population data set when individual outliers appear normal in isolation.
Innovation Solution
A neural network-based anomaly detector is trained using an auto-encoder and discriminator to identify anomalies by generating an internal representation of data sets and estimating divergence from a multivariate unit normal distribution, allowing for the detection of anomalies that individually appear normal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If statistical hypothesis testing is used to detect anomalies, then individual outliers can be identified, but anomalies that appear normal in isolation cannot be detected
Solution Approach 1:
The patent transforms the anomaly detection problem from individual data point analysis to population-level distribution analysis. By using a neural network to model the entire population distribution and comparing individual samples against this learned distribution, the system detects anomalies in a new dimensional space (population statistics rather than individual characteristics), enabling detection of anomalies that appear normal in isolation.
Solution Approach 2:
The patent replaces conventional statistical hypothesis testing with a neural network-based approach. The neural network learns complex population distributions and performs anomaly detection through learned representations rather than traditional statistical tests, substituting a more adaptable computational system for the limited mechanical statistical testing framework.
2Device complexity
If conventional statistical methods are used, then the detection process is simple, but detection accuracy for subtle anomalies is insufficient
Solution Approach 1:
The patent replaces simple statistical methods with a neural network system that, while more complex, provides superior detection accuracy. The neural network's ability to learn complex patterns and distributions compensates for the increased computational complexity, achieving high precision in detecting subtle anomalies that conventional methods miss.
Data Source
AI summary
Anomalies in a data set may be difficult to detect when individual items are not gross outliers from a population average. Disclosed is an anomaly detector that includes neural networks such as an auto-encoder and a discriminator. The auto-encoder and the discriminator may be trained on a training set that does not include anomalies. During training, an auto-encoder generates an internal representation from the training set, and reconstructs the training set from the internal representation. The training continues until data loss in the reconstructed training set is below a configurable threshold. The discriminator may be trained until the internal representation is constrained to a multivariable unit normal. Once trained, the auto-encoder and discriminator identify anomalies in the evaluation set. The identified anomalies in an evaluation set may be linked to transaction, security breach or population trends, but broadly, disclosed techniques can be used to identify anomalies in any suitable population.