Anomaly Detection via Low-Dimensional Feature Subsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems face challenges in reliably prioritizing and representing clusters of anomalies in high-dimensional datasets, often missing clusters that manifest atypicality on a small subset of features, leading to inefficient manual examination and potential missed detections.
Innovation Solution
A system that detects clusters of anomalous samples on low-dimensional feature subsets using parsimonious mixture modeling and Bayesian information criterion (BIC) for hypothesis testing, enabling joint detection and classification of anomalies, and an active learning framework for prioritizing and labeling clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard anomaly detection methods assess atypicality of individual samples on the full measured feature space, then individual anomalies may be detected, but clusters of anomalies manifesting on a small subset of features will be missed and detection efficiency deteriorates
Solution Approach 1:
The patent segments the high-dimensional feature space by identifying and focusing on specific low-dimensional feature subsets where anomalies manifest. Instead of analyzing all features uniformly, the system divides the feature space into relevant subspaces based on anomaly characteristics, enabling both accurate detection and efficient processing by concentrating computational resources on critical feature regions.
Solution Approach 2:
The patent extracts and isolates the salient low-dimensional feature subsets that characterize anomalous clusters from the full high-dimensional feature space. By extracting only the relevant features where anomalies manifest, the system achieves accurate cluster detection while improving efficiency by eliminating computation on irrelevant features.
2Loss of information
If individual anomalous samples are detected separately, then each anomaly can be identified, but the common statistical character of anomaly clusters cannot be inferred and manual examination burden increases
Solution Approach 1:
The patent merges individual anomaly detections by identifying samples that share common statistical characteristics across low-dimensional feature subsets. By combining individually detected anomalies into unified clusters based on their shared patterns, the system preserves complete anomaly information while automatically organizing it into meaningful groups, thereby eliminating the need for manual examination of each individual anomaly.
Solution Approach 2:
The patent implements feedback mechanisms where detection results from individual samples inform the clustering process, and cluster information反过来 refines individual anomaly assessment. This iterative feedback loop ensures that no anomaly information is lost while automatically prioritizing clusters for examination, significantly reducing manual review time through intelligent prioritization.
3Measurement precision
If high-dimensional feature vectors are used to measure each flow, then comprehensive anomaly detection is possible, but anomalies manifesting on a small subset of features evade detection and interpretability decreases
Solution Approach 1:
The patent applies local quality by assigning different measurement approaches to different feature subspaces. For low-dimensional feature subsets where anomalies manifest, the system applies precise local measurement and analysis. For other features, different or reduced measurement is applied. This localized approach maintains measurement precision for critical features while improving overall detectability by focusing computational effort where anomalies actually occur.
Solution Approach 2:
The patent transitions from analyzing high-dimensional feature vectors to focusing on low-dimensional feature subsets. By changing the dimensional perspective and projecting anomalies into lower-dimensional subspaces where they manifest, the system improves detectability without sacrificing measurement precision on the relevant features.
4Reliability
If all detected anomalies are presented to administrators for examination, then complete coverage is achieved, but the time-consuming nature of examining each anomaly individually reduces operational efficiency
Solution Approach 1:
The patent performs preliminary actions by automatically clustering and prioritizing anomalies before presenting them to administrators. The system pre-processes detected anomalies into organized clusters with assigned priorities based on their statistical significance and characteristics. This preliminary organization maintains complete detection coverage while dramatically improving operational efficiency by presenting administrators with pre-sorted, high-value targets for examination.
Data Source
AI summary
This patent concerns novel technology for detection of zero-day data classes for domains with high-dimensional mixed continuous/discrete feature spaces, including Internet traffic. Assume there is a known-class database available for learning a null hypothesis that a given new batch of unlabeled data does not contain any data from unknown/anomalous classes. A novel and effective generalization of previous parsimonious mixture and topic modeling methods is developed. The novel unsupervised anomaly detector (AD) acts on a new unlabeled batch of data to either identify the statistically significant anomalous classes latently present therein or reject the alternative hypothesis that the new batch contains any anomalous classes. The present AD invention can be applied in an on-line setting. Labeling (by a human expert or by other means) of anomalous clusters provides new supervised data that can be used to adapt an actively learned classifier whose objective is to discriminate all the classes.