Anomaly Detection Forest Subspace Nodes Reduce False Positives
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection algorithms, such as Isolation Forest, face challenges in accurately identifying anomalies without labeled malicious data, leading to high false positives and underweighting unexpected data points, especially when trained on datasets without anomalies.
Innovation Solution
The proposed method introduces Anomaly Detection Forests, which utilize subspace selection nodes and anomaly-catching nodes to create AD trees, allowing for better isolation of unseen data and reducing false positives by bisecting feature samples based on predetermined thresholds and depth limits, and calculating anomaly scores for improved detection accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the Isolation Forest algorithm is trained on a dataset without anomalies, then the training process completes successfully, but the algorithm produces high false positives because each leaf in the trees will have at least one corresponding training sample
Solution Approach 1:
The patent changes the fundamental parameter of how leaves are created in isolation trees. Instead of requiring each leaf to contain at least one training sample, the invention allows leaves to be created when a predetermined number of training samples are assigned to an internal node, enabling leaves to potentially contain zero training samples. This parameter change directly reduces false positives while maintaining reliable anomaly detection.
2Adaptability or versatility
If the Isolation Forest algorithm isolates training samples to create leaves, then the training data is fully utilized, but unexpected data points (unseen data) are not given sufficient weight and cannot produce shorter paths
Solution Approach 1:
The patent applies preliminary action by establishing predetermined thresholds and depth limits before the training process begins. These pre-set parameters guide the tree construction to create leaves that can accommodate unseen data points, rather than strictly adapting to the training data distribution. This allows the model to be more adaptable to new, unexpected data while maintaining precise anomaly scoring through the predetermined criteria.
Data Source
AI summary
The present disclosure relates to a method and an apparatus for training a model for detecting anomalies in network data traffic between devices in a first part of a network and devices in a second part of the network. The method comprises collecting feature samples of network data traffic at a monitoring point between a first and a second part of the network, and training the model for detecting anomalies on the collected feature samples using a plurality of anomaly detection, AD, trees. The training comprises creating the plurality of AD trees using respective subsets of the collected feature samples, at least some of the AD tree comprising subspace selection nodes and anomaly-catching nodes to a predetermined AD tree depth limit. Each subspace selection node is arranged to bisect a set of feature samples reaching the subspace selection node to at least one anomaly-catching node when a number of feature samples leaving the subspace selection node for the at least one anomaly-catching node is below a predetermined threshold. The disclosure also relates to a method and an apparatus for anomalies in network data traffic using said model.