Anomaly Detection Using Artificial Outlier Injection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accumulating and labeling sufficient data points for anomaly detection in network traffic using unsupervised or semi-supervised machine learning algorithms is challenging, often resulting in a high number of false positives.
Innovation Solution
The system injects outlier data points into a training dataset for a K-means clustering algorithm to accurately define boundaries between normal and anomalous data points, using these outliers to determine a normalcy radius and classify new data points as normal or abnormal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional unsupervised or semi-supervised machine learning algorithms are used for anomaly detection, then the detection process can be performed with available data, but the number of false positives increases substantially
Solution Approach 1:
The system performs preliminary action by injecting artificial outlier data points into the training dataset before executing the K-means clustering algorithm. This pre-processing step prepares the data in advance to ensure that cluster boundaries are properly defined during the clustering process, thereby reducing false positives in anomaly detection without requiring additional computational resources during actual detection
Solution Approach 2:
The system changes the composition parameter of the training dataset by incorporating artificial outlier data points mixed with normal data points. This parameter modification allows the K-means algorithm to learn more accurate cluster boundaries during training, improving the precision of anomaly detection while maintaining reliability with available data
2Reliability
If more data points are accumulated and labeled for training, then the anomaly detection model may improve, but the process becomes more challenging and time-consuming
Solution Approach 1:
The system applies self-service by automatically generating artificial outlier data points through mathematical transformations of normal data points, eliminating the need for manual data accumulation and labeling. The algorithm autonomously creates diverse outlier samples by applying different transformation functions, thereby improving detection reliability without incurring time losses associated with manual data preparation
Solution Approach 2:
The system introduces artificial outlier data points as an intermediary element between normal data points and the anomaly detection process. These intermediary points serve as placeholders that guide the clustering algorithm to establish accurate boundaries, reducing the need for extensive manually labeled data while maintaining high detection reliability
3Adaptability or versatility
If the cluster boundary is defined to include all data points, then no data points are excluded, but the boundary becomes less precise in distinguishing normal from abnormal points
Solution Approach 1:
The system applies partial action by determining the normalcy radius to extend only to the farthest normal data point from the cluster center, deliberately excluding artificial outlier points. This partial inclusion strategy creates a precise boundary that captures all normal variations while rejecting anomalies, balancing adaptability with measurement precision in the anomaly detection process
Data Source
AI summary
The present disclosure is directed towards systems and methods for improving anomaly detection using injected outliers. A normalcy calculator of a device may include a set of outliers into a training dataset of data points. The normalcy calculator, using a K-means clustering algorithm applied on the training dataset, identify at least a first cluster of data points. The normalcy calculator of the device may determine a region with a center and an outer radius that covers at least a spatial extent of the first cluster of data points. The normalcy calculator may determine a first normalcy radius for the first cluster by reducing the region around the center until a point at which all artificial outliers are excluded from a region defined by the first normalcy radius. An outlier detector of the device may use the region defined by the first normalcy radius to determine whether a new data point is normal or abnormal.