Quasi-Supervised Clustering for Dynamic User Behavior Profiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing User and Entity Behavioral Analytics (UEBA) systems struggle with accurately profiling online user activity due to the inability to determine the number of distinct patterns of activity in advance, limiting the effectiveness of clustering methods.
Innovation Solution
A quasi-supervised clustering system, UBPAS, automatically determines the number of distinct clusters using sparse similarity matrices and tri-point arbitration to mimic supervised approaches, allowing for dynamic inference of optimal cluster numbers without pre-specification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If pre-specification of cluster number is used in clustering analysis, then computational simplicity is improved, but accuracy of user activity profiling deteriorates
Solution Approach 1:
The clustering system automatically determines the optimal number of clusters through self-organization and iterative refinement without requiring external specification. The algorithm performs clustering with an initial number of clusters, evaluates results using validation metrics, and adjusts the cluster count automatically to achieve accurate user activity profiling without manual intervention.
Solution Approach 2:
The system dynamically adjusts the number of clusters based on the characteristics of the user activity data. Rather than using a fixed pre-specified number, the clustering algorithm adapts the cluster count during the analysis process, allowing the system to respond to the actual structure and complexity of the data being analyzed.
2Measurement precision
If automatic determination of cluster number is implemented, then accuracy of user activity profiling is improved, but device complexity increases
Solution Approach 1:
The system incorporates feedback mechanisms through validation metrics that evaluate clustering quality at different cluster counts. The algorithm uses these feedback signals to automatically adjust the number of clusters, comparing clustering results against established criteria and refining the cluster count to optimize accuracy while managing system complexity through iterative improvement.
Data Source
AI summary
Systems, methods, and other embodiments associated with quasi-supervised clustering for activity pattern characterization and anomalous activity detection are described. In one embodiment, a method generates a first sparse similarity matrix for nearest neighbors of a plurality of data points. The data points each characterize a pattern of activity associated with an account. The method generates a second sparse similarity matrix for random neighbors of the plurality of data points. The method recursively clusters the plurality of data points based on the first sparse similarity matrix. The method quasi-supervises the recursive clustering based on the second sparse similarity matrix to stop the iterative clustering when the data points are split into N clusters. The value of N is not pre-determined. The method detects that the individual data point has changed clusters, indicating anomalous activity. And, the method generates an electronic alert that the anomalous activity is associated with the account.