Anomaly Detection Training Data Generation via LOF
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Anomaly detection in security applications faces challenges due to the lack of labeled samples, leading to high false positive and false negative rates, especially in multi-user environments where data distribution assumptions are unrealistic and high-dimensional data is involved.
Innovation Solution
A Local Outlier Factor (LOF) based method that automatically generates both benign and malicious training samples from unlabeled data by leveraging behavioral patterns from other users to estimate abnormal patterns for a target user, without relying on distribution patterns or manual labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If one-class classification approaches are used to detect anomalies by treating existing samples as benign, then the method can handle unique attacks without requiring labeled anomalous samples, but the false positive rate becomes excessively high
Solution Approach 1:
The patent introduces an intermediary mechanism that uses other users' behavior data as a mediator to generate synthetic anomalous samples. Instead of directly comparing target user behavior against a static benign profile (which causes high false positives), the system uses LOF algorithm to identify outliers in the broader user population context, creating a more nuanced anomaly detection approach that reduces false positives while maintaining detection capability for novel attacks
2Measurement precision
If the target user's own behavior samples are used as the only training data, then the model can capture user-specific patterns, but the model lacks anomalous samples for accurate classification
Solution Approach 1:
The patent applies copying by creating synthetic anomalous samples that replicate the structural characteristics of real anomalies without requiring actual anomalous data from the target user. The LOF algorithm identifies outliers in the aggregate user behavior data, and these synthesized samples are then used to train the classification model, effectively copying the anomaly structure needed for accurate detection while maintaining user-specific modeling through the behavior data collection mechanism
3Ease of manufacture
If distribution-based methods are used to generate artificial anomaly points near normal data points, then the method assumes anomalies are close to normal data, but this approach fails for high-dimensional data with continuous variables
Solution Approach 1:
The patent replaces distribution-based mechanical sample generation (which assumes specific data structures and fails on high-dimensional continuous data) with a statistical outlier detection approach using the LOF algorithm. This substitution allows the system to identify anomalies based on local density and neighborhood characteristics in the feature space, making it adaptable to high-dimensional data with continuous variables while still generating synthetic anomalous samples for training
Data Source
AI summary
A method of forming an anomaly detection monitor includes obtaining data samples of operations performed on an application by a plurality of users and detecting, by a processor, anomalous behavior associated with a target user of the plurality of users with respect to the application based on a portion of the data samples associated with the target user and a portion of the data samples associated with a second user of the plurality of users, different from the target user.