Decision Tree Anomaly Detection for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated anomaly detection methods become inefficient as data set size increases, requiring exhaustive searches and significant processing time, making it difficult to identify anomalies in large datasets.
Innovation Solution
A computerized method using a decision tree to partition data sets, calculating z-scores for subsets to identify anomalous data, and providing these subsets for display, which maintains efficiency even with large data sets by limiting processing to only relevant subsets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional automated anomaly detection methods are used to search through all possible subsets of data, then comprehensive anomaly detection is achieved, but processing time increases significantly as data set size increases
Solution Approach 1:
The patent applies segmentation by dividing the large data set into smaller subsets using a decision tree structure. The decision tree partitions the data based on feature values, creating manageable segments that can be processed independently. This allows the system to detect anomalies in each subset without needing to exhaustively search through all possible combinations of the entire data set, thereby reducing processing time while maintaining detection effectiveness.
Solution Approach 2:
The patent implements preliminary action by pre-processing the data through decision tree partitioning before anomaly detection. The decision tree structure is built in advance to organize data into logical subsets based on feature relationships. This preliminary organization enables subsequent anomaly detection to focus only on relevant subsets rather than searching through all data combinations, significantly reducing the time required for comprehensive anomaly detection.
2Reliability
If exhaustive search of all possible m-tuples of n columns is performed to find surprising subsets, then complete anomaly identification is achieved, but computing efficiency decreases considerably
Solution Approach 1:
The patent applies local quality by focusing computational resources on specific subsets of data that are most likely to contain anomalies. Instead of uniformly processing all possible m-tuples of n columns, the decision tree identifies and prioritizes particular regions of the data space based on feature values and relationships. This localized approach maintains anomaly detection accuracy by concentrating analysis where anomalies are most likely to occur while improving computing efficiency by avoiding unnecessary processing of low-probability regions.
Solution Approach 2:
The patent implements partial action by performing anomaly detection on selected subsets rather than exhaustively analyzing all possible data combinations. The decision tree structure enables the system to identify and process only the most promising subsets that contain potential anomalies, rather than conducting complete searches through all m-tuples of n columns. This partial processing approach achieves sufficient anomaly detection accuracy while dramatically improving computing efficiency.
3Reliability
If all data is processed to ensure no anomalies are missed, then detection completeness is maximized, but processing complexity increases beyond human capacity
Solution Approach 1:
The patent applies segmentation by organizing data into structured subsets through decision tree partitioning. This creates a hierarchical view of the data that maintains completeness while reducing complexity. Each node in the decision tree represents a manageable subset defined by specific feature conditions, allowing the system to process and verify anomalies in an organized manner rather than dealing with the overwhelming complexity of complete data processing.
Solution Approach 2:
The patent introduces the decision tree structure as an intermediary between the raw data and the anomaly detection process. This intermediary organizes and pre-processes the data into meaningful subsets before anomaly analysis begins. The decision tree acts as a mediator that reduces processing complexity by filtering and structuring data in advance, enabling comprehensive anomaly detection without requiring direct processing of all raw data combinations.
Data Source
AI summary
Examples for detecting anomalies in a dataset are provided herein. A decision tree is trained using the data set and partitions of the data set produced by the trained decision tree are identified. Further, subsets of data based at least on the partitions of the data set are identified and z-scores are computed for the subsets of data. Based at least on the subsets of data, a subset of data with a highest z-score is identified as an anomalous subset of data, and the anomalous subset of data is provided for display.