Dictionary Based Projection for High-Dimensional Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for anomaly detection in high-dimensional big data lack efficiency and require significant computational resources, especially when dealing with unknown multidimensional data points, and often rely on domain expertise, signatures, rules, or semantics.
Innovation Solution
The use of dictionary-based projection (DBP) to construct a smaller dictionary from input data, which reduces computational requirements by determining whether a new data point is normal or anomalous, utilizing a Gaussian kernel for affinity matrix construction and iterative projection to identify anomalies without direct reference to the original data matrix.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection methods are used on high-dimensional big data, then detection accuracy can be maintained, but computational requirements and processing time increase significantly
Solution Approach 1:
The patent segments the high-dimensional data into multiple lower-dimensional subspaces using random projection. Each subspace is processed independently to compute local anomaly scores, which are then aggregated to form the final anomaly detection result. This segmentation approach maintains detection accuracy while significantly reducing computational complexity by avoiding direct processing of the full high-dimensional space.
Solution Approach 2:
The patent transforms the high-dimensional data into multiple lower-dimensional subspaces through random projection. This dimensionality change allows the system to process data more efficiently in each subspace while preserving the essential anomaly characteristics. The aggregation of scores from multiple subspaces compensates for the reduced dimensionality, maintaining overall detection accuracy.
2Measurement precision
If domain expertise and semantic understanding are incorporated into anomaly detection, then detection precision improves, but system complexity and implementation difficulty increase
Solution Approach 1:
The patent implements an unsupervised anomaly detection system that operates without requiring domain expertise, signatures, rules, or semantic understanding. The system automatically learns the normal data distribution from the input data and identifies anomalies based on deviations from this distribution. This self-service approach eliminates the need for manual configuration of domain-specific knowledge while maintaining effective anomaly detection capability.
3Loss of information
If the original data matrix is used for anomaly detection, then comprehensive analysis is possible, but memory usage and computational resources increase
Solution Approach 1:
The patent extracts essential information from the high-dimensional data by projecting it into multiple lower-dimensional subspaces. Instead of storing and processing the complete original data matrix in memory, the system processes projections that capture the essential anomaly characteristics. This extraction approach reduces memory requirements while preserving the information necessary for accurate anomaly detection through score aggregation.
Data Source
AI summary
Methods for anomaly detection using dictionary based projection (DBP), and system for implementing such methods. In an embodiment, a method comprises receiving input data including a plurality n of multidimensional data points (MDDPs) with dimension m, applying DBP iteratively to the input data to construct a dictionary D, receiving a newly arrived MDDP (NAMDDP), calculating a score S associated with the NAMDDP as a distance of the NAMDDP from dictionary D, and classifying the NAMDDP as normal or as an anomaly based on score S, wherein classification of the NAMDDP as an anomaly is indicative of detection of an unknown undesirable event.