Hierarchical Clustering via Random Intersection Leaves
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional clustering algorithms struggle with high-dimensional big data sets, as distance-based metrics become meaningless in infinite dimensions, making it difficult to identify similarity and cluster structure, especially in fields like genomics where data sets contain millions of points in tens of thousands of dimensions.
Innovation Solution
A hierarchical cluster analyzer using an information-based metric, specifically random intersection leaves (RIL) and stochastic partitioning, generates an association matrix representing data points as a weighted graph, allowing for global clustering and revealing topological structure independently of dimensionality, without requiring the number of clusters as input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If distance-based metrics (e.g., Euclidean distance) are used to quantify similarity of data points, then clustering can be performed in low-dimensional spaces, but the distance measure becomes meaningless and fails to identify similarity in high-dimensional spaces
Solution Approach 1:
The patent changes the fundamental parameter used for measuring similarity from distance-based metrics to information-theoretic metrics (mutual information). This parameter change allows the system to maintain measurement precision across varying dimensionalities, as information-based measures do not suffer from the curse of dimensionality that plagues distance-based approaches in high-dimensional spaces.
Solution Approach 2:
The patent substitutes the mechanical/geometric approach of distance measurement with an information-theoretic approach using mutual information. This substitution replaces the geometric intuition that breaks down in high dimensions with a statistical measure of dependence that remains meaningful regardless of dimensionality.
2Productivity
If traditional clustering algorithms are used on big data sets, then computational simplicity is maintained, but the algorithms fail to identify cluster structure in high-dimensional data
Solution Approach 1:
The patent segments the high-dimensional data processing task into multiple stages: first computing pairwise mutual information between data points, then constructing a similarity graph, and finally applying clustering algorithms. This segmentation allows the complex problem to be broken into manageable computational steps that can be parallelized and optimized.
Solution Approach 2:
The patent introduces an association matrix as an intermediary structure that captures pairwise relationships between data points based on mutual information. This matrix serves as a compressed representation of the data that preserves cluster structure while reducing the complexity of subsequent clustering operations.
3Adaptability or versatility
If the number of clusters is not specified as input, then the method becomes more versatile and adaptable to unknown data structures, but traditional algorithms require this parameter to function
Solution Approach 1:
The patent implements self-service by allowing the clustering algorithm to automatically determine the number of clusters through spectral analysis of the association matrix. The algorithm computes eigenvalues and uses their distribution to infer the appropriate number of clusters, eliminating the need for manual specification while maintaining operational simplicity.
Data Source
AI summary
A hierarchical cluster analyzer identifies clusters in a big data set by identifying topological structure without distance-based metrics. The hierarchical cluster analyzer stochastically partitions the big data set to create pseudo-partitions of the big data set. The stochastic partitioning may be implemented with a random forest classifier that uses ensemble techniques to reduce variance and prevent overfitting. The hierarchical cluster analyzer implements random intersection leaves (RIL), a data mining technique that grows an intersection tree by intersecting candidate sets generated from the pseudo-partitions. The hierarchical cluster analyzer updates an association matrix according to co-occurrences of data points within each leaf node of the intersection tree. These co-occurring data points exhibit a high degree of similarity, which is recorded in the association matrix. A hierarchy of clusters may then be formed by finding community structure in the association matrix.