Power-Law Anomaly Detection for Unknown Event Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems rely on prior knowledge of specific characteristics to detect anomalies, making them less effective against new or unknown types of anomalies.
Innovation Solution
Anomaly detection system utilizing power law profiles, such as Benford's Law and Zipf's Law, to analyze data groupings and produce goodness of fit values, enabling detection of anomalies without prior knowledge of their characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If anomaly detection systems use prior knowledge of specific characteristics to detect anomalies, then detection accuracy for known anomalies is improved, but the system becomes less effective against new or unknown types of anomalies
Solution Approach 1:
The patent changes the detection parameter from specific anomaly characteristics to general power law conformance. Instead of looking for known anomaly signatures, the system monitors whether system data follows power law distributions, which is a fundamental property of complex systems. This parameter change enables detection of both known and unknown anomalies while maintaining adaptability.
Solution Approach 2:
The patent applies power law analysis universally across different types of system data and anomaly types. The power law conformance check serves as a universal detection mechanism that works for various anomaly types (malware, network attacks, system failures) without requiring anomaly-specific knowledge, thus achieving multi-functionality in anomaly detection.
2Reliability
If anomaly detection systems continuously update knowledge bases to keep up with current anomalies, then detection of known anomalies is improved, but system complexity and resource requirements increase
Solution Approach 1:
The system uses the system's own operational data to detect anomalies by checking power law conformance. The power law properties serve as an inherent reference that doesn't require external knowledge bases or continuous updates. The system essentially monitors itself against fundamental statistical properties, eliminating the need for complex update mechanisms.
Solution Approach 2:
The patent establishes power law conformance as a preliminary condition for normal system operation. By pre-defining what normal behavior looks like through power law distributions, the system eliminates the need for continuous knowledge base updates. The power law characteristics are established once and serve as a permanent reference for anomaly detection.
3Measurement precision
If anomaly detection systems analyze complete data groups, then detection accuracy is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent segments the data analysis process into multiple passes: first checking power law conformance of complete data groups for overall anomaly detection, then potentially analyzing specific subsets or individual data points only when anomalies are suspected. This segmentation allows efficient processing of large datasets while maintaining detection accuracy by focusing detailed analysis only where needed.
Data Source
AI summary
An anomaly detection system is described for detecting an anomalous event associated with source data. The anomaly detection system comprises a data grouping creator arranged to produce a plurality of data groupings of data elements from a data group of data elements. The data group is obtained from the source data, the data groupings including data elements from the data group and less data elements than the data group, and the source data conforming substantially to a natural power law when an anomalous event is not present. The system also includes a power law goodness of fit tester arranged, for each of at least some of the data group and data groupings, to compare a power law profile obtained from the data group or data grouping with a reference power law profile, and produce a goodness of fit value indicative of a similarity between the power law profile obtained from the data group or data grouping and the reference power law profile. The system also includes a decision system arranged to receive the produced goodness of fit values and output an indication as to whether the produced goodness of fit values are indicative of an anomalous event in the computing system.