Hierarchical Binary Tree Anomaly Detection for Unbalanced Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Isolation Forest algorithm fails to effectively detect anomalies in unbalanced data distributions, as it randomly determines features and thresholds for binary tree structures, leading to misclassification of minority data groups as outliers.
Innovation Solution
The proposed solution involves creating a global binary tree structure to divide data into balanced groups, followed by a local binary tree structure for each group, using the depth from the root node to the leaf node as a score to calculate anomaly levels, thereby addressing the issue of unbalanced data distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Isolation Forest algorithm uses random features and thresholds to create a binary tree structure, then the algorithm is simple and fast to execute, but it misclassifies minority data groups as outliers when data distribution is unbalanced
Solution Approach 1:
The patent divides the data processing into two distinct phases: first creating a global binary tree structure for all data points, then creating separate local binary tree structures for each group obtained from the global division. This segmentation allows the algorithm to handle different data groups with their own characteristics, improving anomaly detection accuracy for unbalanced data while maintaining computational efficiency through hierarchical processing.
Solution Approach 2:
The patent applies different tree structures to different data groups: a global binary tree structure for overall data division and local binary tree structures for each specific group. This local quality approach ensures that each data group is evaluated with a structure optimized for its characteristics, preventing misclassification of minority groups while maintaining the simplicity and speed of the overall algorithm.
2Device complexity
If a single binary tree structure is used for all data, then the device complexity is low, but the anomaly detection reliability deteriorates when data distribution is unbalanced
Solution Approach 1:
The patent implements a nested structure where local binary tree structures are created within each group obtained from the global binary tree structure. This nesting allows the system to maintain a simple global structure while incorporating more complex local structures where needed, improving reliability for unbalanced data without excessive increase in overall complexity.
Solution Approach 2:
The patent adds a hierarchical dimension to the tree structure by introducing both global and local levels. This dimensional change transforms the single-level tree structure into a multi-level hierarchical structure, enabling the system to handle unbalanced data distributions effectively while maintaining manageable complexity through organized hierarchy.
Data Source
AI summary
An anomaly detection apparatus according to an embodiment of the present disclosure includes: a global tree structure creation unit configured to create a global tree structure for dividing a plurality of data pieces into a plurality of groups, a local tree structure creation unit configured to create a local tree structure for further dividing the data pieces divided into the plurality of groups for each of the plurality of groups, and a score calculation unit configured to calculate a score indicating an anomaly level of the plurality of data pieces using a depth from a root node to a leaf node of the local tree structure.