Real-Time Anomaly Detection Using Two-Stage Data Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Real-time analysis of high-dimensional data generated by IoT networks is challenging due to its complexity and the difficulty in identifying useful data for anomaly detection, leading to low accuracy and inefficiency in identifying potential issues.
Innovation Solution
The method involves a two-stage dimension reduction process using t-distributed stochastic neighbor embedding (t-SNE) and Principle Component Analysis (PCA) to filter and cluster data, followed by statistical analysis to define a threshold group for distinguishing normal and abnormal data, which are then used to train classifiers or regression models for anomaly prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If high dimensional data is analyzed in real-time using traditional methods, then measurement precision may be maintained, but analysis speed and productivity deteriorate due to time-consuming complex processing
Solution Approach 1:
The patent segments the high dimensional data analysis process into distinct stages: data filtering to identify useful subsets, dimensionality reduction to transform data structure, and pattern recognition to detect anomalies. This segmentation enables real-time processing by breaking down the complex analysis into manageable, parallelizable operations that reduce overall computational burden while maintaining accuracy.
Solution Approach 2:
The patent extracts and removes irrelevant or redundant features from the high dimensional data set through filtering mechanisms and dimensionality reduction techniques. By taking out only the essential features that contribute to anomaly detection, the system achieves faster processing speeds without sacrificing measurement precision, directly resolving the contradiction between productivity and complexity.
2Measurement precision
If all data in the high dimensional data set is used for analysis, then measurement precision may be maintained, but loss of time increases due to processing unnecessary data
Solution Approach 1:
The patent applies extraction principles by identifying and isolating the useful subset of data from the complete high dimensional data set. Through filtering based on relevance criteria and dimensionality reduction, the system extracts only the essential features needed for accurate anomaly detection, eliminating processing of redundant data and thereby reducing time loss while preserving measurement precision.
Solution Approach 2:
The patent implements local quality by applying different processing strategies to different portions of the data. Rather than uniformly processing all data points, the system identifies regions of high relevance and focuses computational resources there, while applying simplified processing or filtering to less critical data portions. This localized approach maintains detection accuracy in critical areas while reducing overall processing time.
3Productivity
If dimensionality reduction is applied to high dimensional data, then productivity improves through faster processing, but measurement precision may deteriorate due to information loss
Solution Approach 1:
The patent employs parameter changes by transforming the data representation through dimensionality reduction techniques that change the parameter space. The system carefully selects and adjusts reduction parameters to preserve the most informative aspects of the data while reducing dimensionality. This controlled parameter transformation enables faster processing while maintaining measurement precision by retaining essential information in the reduced feature space.
Data Source
AI summary
A method of high dimensional data analysis in real-time comprising executing dimension-reducing an input historical data set under a t-SNE model and determining from the resulting dimension-reduced data set a recent; further dimension-reducing the recent group data set under a PCA model; statistical analyzing the further dimension-reduced data set to determine a threshold group for distinguishing abnormal data from normal ones in a real-time data stream. The method may further include training a classifier using the abnormal or normal data set for predicting anomaly in the real-time data source system. Alternatively, a discrepancy training data set is computed from one of the normal and abnormal data sets and be used to train one of independent normal and abnormal data regression models; with the other one trained by transfer learning based on the trained one. The trained regression models are then used to predict discrepancy values.