Dynamic Peer Group Analysis for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection techniques in large-scale datasets struggle to reduce false anomaly detections and detect anomalies early, often mislabeling normal behaviors as anomalies and requiring threshold met conditions for detection.
Innovation Solution
A method for dynamic peer group analysis that involves receiving data on entities with static and dynamic characteristics and risk scores, generating a relational graph, determining peer groups, calculating risk score averages and standard deviations, and identifying systematic changes to differentiate between false and true anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection algorithms are used on large-scale unlabeled data, then anomaly detection can be performed, but false anomaly detections increase and anomaly detection timing is delayed
Solution Approach 1:
The patent segments the anomaly detection process into two distinct phases: a training phase where peer groups are formed using normal behavior patterns, and a detection phase where anomalies are identified by comparing current behavior against peer group averages. This segmentation allows the system to learn normal patterns once and then efficiently detect deviations without reprocessing entire datasets, reducing both false detections and detection time.
Solution Approach 2:
The patent performs preliminary action by pre-calculating and storing peer group average risk scores during a training phase before actual anomaly detection is needed. This preliminary computation of baseline behavior patterns allows the system to quickly compare against established norms during detection, enabling early anomaly identification without requiring real-time complex analysis.
2Reliability
If anomaly detection thresholds are set high to reduce false alarms, then false anomaly detections decrease, but anomaly detection capability at early stages is lost
Solution Approach 1:
The patent applies dynamics by making the anomaly detection threshold adaptive rather than fixed. The system dynamically compares individual entity risk scores against the average risk score of their peer group, allowing the threshold to automatically adjust based on the specific peer group's behavior patterns. This dynamic comparison enables early anomaly detection while maintaining reliability, as the threshold adapts to normal variations within each peer group context.
Solution Approach 2:
The patent implements local quality by evaluating anomalies within the context of specific peer groups rather than using a single global threshold. Each peer group has its own average risk score calculated from its members' normal behavior, allowing the system to apply appropriate thresholds locally for each group. This local evaluation enables early detection of true anomalies while reducing false alarms, as each group's baseline is tailored to its specific characteristics.
Data Source
AI summary
Provided are systems, methods, and computer program products for dynamic peer group analysis for systematic changes in large scale data. Data associated with a plurality of entities is received and a relational graph is generated based on the data. A target entity is selected and a peer group for the target entity is determined based on the relational graph. An average and a standard deviation of the risk scores of the peer group are calculated and used to determine whether a systematic change in the behavior of the peer group has occurred. Whether a change in behavior of the target entity is a false anomaly or a true anomaly is determined based on whether a systematic change in the behavior of the peer group has occurred. An action is performed based on whether the change in behavior of the target entity is a false anomaly or a true anomaly.