Anomaly Detection via Adversarial Shuffling and Gradient Boosting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Predictive models are only valid when the variables of unseen data are drawn from the same distribution as the variables in the historic or training data, and existing anomaly detection methods fail to accurately identify anomalies in high-dimensional spaces and complex interactions across data fields.
Innovation Solution
The system generates an adversarial data set by shuffling each column of the true data set and builds a series of models to distinguish it from the adversarial data set, allowing for the detection of anomalous data points by iteratively removing the most important features, using gradient boosted models with decision trees to identify predictor variables and values that cause distribution shifts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional anomaly detection methods (DBScan, Isolation Forests) are used, then outliers can be identified through clustering, but these methods fail to accurately identify anomalies in high-dimensional spaces and cannot capture complex interactions across data fields
Solution Approach 1:
The patent segments the anomaly detection process into multiple specialized components: (1) distribution verification module that checks if data follows expected distributions, (2) interaction term analysis module that detects complex field interactions, (3) sanity check module that validates data合理性. This segmentation allows each component to focus on specific aspects of anomaly detection, improving accuracy without requiring a single complex model to handle all scenarios.
Solution Approach 2:
The patent introduces interaction terms as intermediary variables that capture complex relationships between data fields. By adding these intermediary features that represent pairwise interactions, the model can detect anomalies arising from complex field interactions without requiring extremely complex model architectures, thus improving detection capability while managing complexity.
2Loss of information
If statistical measures (KL divergence, covariate shift analysis) are used to evaluate distribution changes, then probabilistic approaches can identify differences, but these methods cannot identify which specific values in which columns are driving the differences
Solution Approach 1:
The patent transforms the anomaly detection problem from evaluating overall distribution differences to analyzing individual column and interaction term contributions. By decomposing the anomaly score into contributions from each data column and their interactions, the system provides granular information about which specific values and fields are driving anomalies, making the detection process more interpretable and actionable.
Solution Approach 2:
The patent performs preliminary analysis by calculating distribution statistics and interaction term contributions before final anomaly classification. This preliminary action involves computing expected distributions, variance explanations, and interaction effects in advance, which enables the system to quickly identify specific anomaly sources without requiring complex post-hoc analysis.
3Ease of operation
If manual checking and rules-based validation are performed, then data quality can be maintained with simple sanity checks, but these methods do not fully diagnose actual issues with data or identify complex interactions across fields
Solution Approach 1:
The patent implements self-service anomaly detection where the system automatically performs distribution verification, interaction term analysis, and anomaly identification without requiring manual rule configuration. The model learns expected data distributions and interactions from training data and autonomously detects deviations, eliminating the need for manual sanity check rules while improving detection reliability through automated statistical analysis.
Data Source
AI summary
Systems, methods, and apparatuses for detecting and identifying anomalous data in an input data set are provided.