Iterative Clustering Algorithm for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current clustering algorithms, such as K-Mean, face limitations when dealing with large datasets, requiring predefined cluster numbers and being sensitive to anomalies, which hinders their effectiveness in real-world applications like cyber-attack detection and data with unknown cluster structures.
Innovation Solution
A method involving iterative clustering processes that divide sessions into chunks, generate chunk-based clusters, and iteratively refine them to produce group-based clusters, capable of handling any number of sessions and detecting anomalies without predefining cluster numbers or initial outlier removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If K-Mean clustering algorithm is used, then clustering can be performed on datasets, but computational complexity increases rapidly as the number of objects increases, making it impractical for millions of objects
Solution Approach 1:
The patent divides the large dataset into multiple chunks or blocks, each containing a portion of the total objects. The clustering algorithm is applied independently to each chunk, reducing the computational burden per iteration. This segmentation allows the algorithm to scale to millions of objects by processing them in manageable portions rather than requiring all objects to be in memory simultaneously.
Solution Approach 2:
The patent introduces a hierarchical dimension to the clustering process by organizing results into chunk-level clusters and then performing higher-level clustering on cluster representatives. This two-level hierarchy transforms the single-pass clustering problem into a multi-stage process that reduces overall computational complexity while maintaining clustering quality.
2Ease of operation
If traditional clustering algorithms are used, then clustering can be performed, but the number of clusters must be predefined, which is not known in cases like entity activity flows
Solution Approach 1:
The patent implements a dynamic clustering approach where the number of clusters is not fixed in advance but emerges during the clustering process. The algorithm adaptively determines the appropriate number of clusters based on the data distribution and similarity metrics, allowing it to handle cases where the true number of clusters is unknown, such as entity activity flow analysis.
3Measurement precision
If standard clustering algorithms are used, then clustering can be performed on raw data, but anomalies in the data degrade the accuracy of the outcome by becoming clusters themselves
Solution Approach 1:
The patent performs preliminary anomaly detection and handling before the main clustering process. By identifying and treating anomalies in advance—either by removing them, separating them into a distinct category, or weighting them differently—the algorithm prevents anomalies from forming spurious clusters and degrading the overall clustering accuracy.
Data Source
AI summary
A method for searching for abnormal sessions, the method may include (a) obtaining session metadata for each of session of a group of sessions; wherein a session metadata of a session is indicative of at least one session feature that represents activities of the session; (b) forming multiple chunks, whereas each chunk comprises session metadata regarding a portion of the group of sessions; (c) for each chunk, generating chunk-based clusters by applying an iterative clustering process on data points that represent session metadata of the chunk; (d) generating group-based clusters, based on the chunk-based clusters; (e) determining, based at least on the group-based clusters, user profiles and abnormal sessions.