Categorical Data Clustering Through Iterative Block Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing clustering methods for large volumes of categorical data, particularly in cyber security and enterprise networks, are inefficient and resource-intensive, leading to inaccurate threat classification and increased security risks.
Innovation Solution
A clustering process that partitions data into subsets, maps data points onto multi-dimensional categorical vectors, iteratively clusters within blocks, and updates clusters over time using hyperspheres, allowing for efficient and scalable processing of large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional clustering methods are used to process large volumes of categorical data, then clustering accuracy can be maintained, but computational resources and processing time increase substantially
Solution Approach 1:
The patent segments the large dataset into multiple blocks or partitions, processing each block separately through distributed computing nodes. This segmentation allows the system to handle large volumes of categorical data efficiently by dividing the computational workload across multiple processors while maintaining clustering accuracy through coordinated processing of all blocks.
2Measurement precision
If manual review processes are used to assess threat levels, then classification accuracy can be maintained, but human resources and processing time are substantially consumed
Solution Approach 1:
The patent implements automated threat classification systems that perform self-assessment of threat levels using machine learning models and clustering algorithms. The system automatically evaluates clustered data points, assigns threat classifications, and takes mitigation actions without requiring manual human review, thereby maintaining accuracy while dramatically reducing time loss.
3Reliability
If comprehensive threat screening is performed on all emails, then security reliability is improved, but computational resources and processing speed are reduced
Solution Approach 1:
The patent applies preliminary clustering and filtering actions to categorize emails before comprehensive threat screening. By pre-processing the data stream to identify and group similar emails, the system can then apply targeted screening only to relevant clusters, maintaining high security reliability while improving overall screening throughput and reducing computational resource consumption.
Data Source
AI summary
An approach for clustering large sets of categorical data involves iteratively ordering the data points, partitioning the data into blocks based on the ordering, and clustering the data points within each block, where different iterations use different orderings and, thus, different partitionings. In some embodiments, the data points are represented by multi-dimensional categorical vectors, and the orderings are based on permutations of the categorical dimensions. The iterative clustering may be repeated for multiple successive time windows to track the clusters. Various applications of the disclosed clustering approach, including for cyber security, are also described.