Anomaly Detection Using Data Distribution Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security systems in computing environments lack accuracy and automation in detecting outliers and anomalies in data distributions, making them ineffective in identifying unauthorized or malicious activities.
Innovation Solution
A system and method that classify and cluster datapoints to distinguish between natural and unnatural modes in data distributions, using techniques like k-means, Jenks Natural Breaks Optimization, or Kernel Density Estimation, to identify outliers and initiate appropriate actions such as notifications or security protocols.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If threshold-based outlier detection is used, then implementation is simple, but detection accuracy is low
Solution Approach 1:
The patent transforms the detection approach by changing from fixed threshold parameters to dynamic statistical parameters (mean, standard deviation, interquartile range) that adapt to the actual data distribution, thereby improving detection accuracy while maintaining implementation feasibility through standardized statistical methods
Solution Approach 2:
The patent replaces the mechanical threshold-based detection mechanism with a statistical model-based mechanism that uses probability distributions and variance analysis to identify outliers, substituting a rigid mechanical approach with a more flexible statistical approach that improves precision
2Productivity
If automated outlier detection is implemented, then detection efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements automated detection systems that self-adjust to data characteristics by automatically calculating statistical parameters and adapting detection thresholds based on the inherent properties of the monitored data, reducing the need for manual configuration and lowering operational complexity
Solution Approach 2:
The patent performs preliminary statistical analysis to establish baseline parameters and detection models before actual outlier detection begins, pre-processing the data to create adaptive thresholds that simplify subsequent automated detection operations and reduce real-time computational complexity
3Reliability
If existing security systems are deployed, then basic monitoring is provided, but accuracy in identifying malicious activities is insufficient
Solution Approach 1:
The patent transitions from static threshold-based monitoring to dynamic adaptive monitoring that continuously adjusts detection parameters based on evolving data patterns and statistical characteristics, enabling the system to maintain reliability while significantly improving accuracy in identifying malicious activities through adaptive threshold adjustment
Data Source
AI summary
A system, non-transitory computer readable medium and method for detecting anomalous activity based on a data distribution is disclosed. The system includes a processor, and a memory coupled to the processor. The memory stores computer executable instructions that, when executed by the processor, cause the processor to obtain a plurality of datapoints defining a data distribution. The processor classifies the plurality of datapoints as isolated or non-isolated datapoints, and clusters at least some of the non-isolated datapoints into at least one mode segment. A reference mode from the at least one mode segment is identified, and at least one outlier being either an isolated or non-isolated datapoint separated from the reference mode by a predetermined number of isolated datapoints representing a flat portion is identified. An action is initiated based on the identified at least one outlier.