Autoencoder Anomaly Detection Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the number of observed data pieces increases, the relationship between the normal data space and the projection data space becomes more complex, leading to a requirement for larger amounts of normal data in anomaly detection techniques that handle multiple types of data.
Innovation Solution
An anomaly detection apparatus with an autoencoder structure, including an input layer, hidden layers, and an output layer, learns parameters to reproduce input data in the output layer, using a dimensionality reduction approach to calculate the anomaly degree of test data based on the learned parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple types of data are input to improve detection comprehensiveness, then anomaly detection accuracy is improved, but the normal data space increases in combination requiring more normal data
Solution Approach 1:
The patent applies dimensionality reduction techniques to project high-dimensional multi-type data into a lower-dimensional space while preserving the essential correlation structures. This transformation maintains anomaly detection accuracy by retaining the fundamental relationships between different data types, but reduces the effective data space complexity that would otherwise require exponentially more normal training data to adequately cover.
2Measurement precision
If multiple types of data are input to improve detection comprehensiveness, then anomaly detection accuracy is improved, but device complexity increases
Solution Approach 1:
The patent segments the complex multi-type data processing into distinct functional modules: data collection from multiple sources, correlation learning component, dimensionality reduction component, and anomaly scoring component. Each module handles a specific aspect of the processing pipeline, making the overall system more manageable and easier to implement despite handling multiple data types.
Solution Approach 2:
The patent introduces a correlation learning component as an intermediary that captures the relationships between different data types before feeding them into the dimensionality reduction and anomaly detection pipeline. This intermediary layer simplifies the complexity by pre-processing and structuring the multi-type data relationships, making subsequent processing easier and more efficient.
Data Source
AI summary
An anomaly detection apparatus having a function of an autoencoder that includes an input layer, hidden layers, and an output layer, and that learns parameters such that data of the input layer is reproduced in the output layer, the anomaly detection apparatus including: input means that input normal data of a plurality of types; learning means that learn parameters such that normal data of the input layer is reproduced in the output layer by learning a feature across data types using data of a dimension which is smaller than a dimension of the normal data; and anomaly degree calculation means that input test data to the autoencoder using the parameters learned by the learning means, and calculate anomaly degree of the test data based on output data of the autoencoder and the test data.