Multivariate Anomaly Detection Using Covariance-Based Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data anomaly detection methods, particularly in large data sets, face inefficiencies due to reliance on random or judgment-based sampling, leading to unreproducible and biased results.
Innovation Solution
A computer-implemented method and system that transforms input data sets into dimensionality-reduced forms using co-variance assessment, calculating anomaly scores for each data point to identify anomalies without requiring random or judgment-based sampling, ensuring reproducibility and reduced computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual review of all data entries is conducted, then anomaly detection accuracy is improved, but time consumption and computational resources increase significantly
Solution Approach 1:
The patent extracts only the most relevant features from the original data set through feature selection and dimensionality reduction techniques. By identifying and retaining only the features that contribute most to anomaly detection, the system maintains high detection accuracy while processing a reduced feature space, thereby reducing computational time and resources.
Solution Approach 2:
The patent segments the anomaly detection process into distinct stages: data preprocessing, feature selection, dimensionality reduction, anomaly scoring, and validation. This segmentation allows each stage to be optimized independently, with feature selection and dimensionality reduction specifically addressing the time consumption problem by reducing data complexity before the computationally intensive anomaly scoring stage.
2Loss of time
If random sampling or judgment-based sampling is used to reduce data volume, then processing time is reduced, but detection reliability and reproducibility deteriorate
Solution Approach 1:
The patent changes the parameter of data representation by transforming the original high-dimensional data into a lower-dimensional feature space through dimensionality reduction techniques. This parameter change maintains the essential anomaly-related information while reducing data volume, enabling efficient processing without relying on random sampling that compromises reliability.
Solution Approach 2:
The patent performs preliminary feature selection and dimensionality reduction before the actual anomaly detection process. By pre-processing the data to extract and retain only the most relevant features, the system prepares a condensed data representation that maintains detection reliability while enabling faster processing in the subsequent anomaly scoring stage.
3Productivity
If dimensionality reduction is applied to large data sets, then computational efficiency is improved, but data integrity may be compromised
Solution Approach 1:
The patent applies local quality by treating different features differently during dimensionality reduction. Instead of uniformly reducing all dimensions, the system identifies and retains features with higher importance weights that are more relevant to anomaly detection, while reducing or eliminating less important features. This selective approach maintains data integrity for critical features while achieving computational efficiency through reduction of redundant features.
Data Source
AI summary
Disclosed are systems, methods, and devices for data anomaly detection. A signal reflective of an input data set having a plurality of dimensions is received. Co-variance across said plurality of dimensions is assessed. Upon said assessing, at least a portion of the input data set is transformed into a dimensionality-reduced data set. For each given data point in the dimensionality-reduced data set, an anomaly score informative of whether said given data point is an anomaly is calculated.