Two-Stage Variable Clustering for Large Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current variable clustering methods, such as PROC VARCLUS, are inefficient for large data sets with over 3,000 variables, requiring significant calculation resources and not scaling well, making them inappropriate for data mining tasks.
Innovation Solution
A two-stage variable clustering system that computes a similarity matrix to create global clusters, followed by a latent variable clustering technique to generate sub-clusters, reducing the number of variables and creating a multi-level hierarchy for improved efficiency and interpretability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If PROC VARCLUS algorithm is used for variable clustering, then interpretability of principal components is improved, but computational efficiency deteriorates for large data sets
Solution Approach 1:
The patent divides the variable clustering process into two stages: first computing a similarity matrix and creating initial clusters, then performing latent variable clustering on the clustered variables. This segmentation reduces the computational burden compared to applying PROC VARCLUS directly to all variables, while maintaining interpretability through the two-stage clustering approach.
Solution Approach 2:
The patent performs preliminary clustering by computing a similarity matrix and creating initial variable clusters before applying the full latent variable clustering. This preliminary action reduces the dimensionality of the problem space, making the subsequent clustering computation more efficient while preserving the interpretability benefits of variable structure methods.
2Reliability
If variable structure method with eigenvalue decomposition is used, then clustering quality is improved, but computational cost increases
Solution Approach 1:
The patent segments the variable set into clusters based on a similarity matrix before performing eigenvalue decomposition and latent variable clustering. This segmentation reduces the number of variables that require computationally expensive eigenvalue decomposition, thereby lowering computational cost while maintaining clustering quality through the structured approach.
Solution Approach 2:
The patent performs preliminary similarity computation and variable clustering before applying eigenvalue decomposition. This preliminary action identifies groups of related variables, allowing the computationally intensive eigenvalue decomposition to be applied more efficiently to clustered variables rather than all variables, thus reducing overall computational cost while preserving clustering quality.
3Productivity
If common clustering techniques are applied to distance matrix, then computational efficiency is improved, but clustering performance deteriorates
Solution Approach 1:
The patent segments the clustering process into two distinct stages: initial clustering based on a similarity matrix, and subsequent latent variable clustering. This segmentation allows the use of efficient similarity computations in the first stage while applying more sophisticated latent variable methods in the second stage, thereby achieving both computational efficiency and improved clustering performance compared to common clustering techniques alone.
Solution Approach 2:
The patent performs preliminary clustering using efficient similarity matrix computation and initial cluster formation before applying latent variable clustering. This preliminary action provides a computationally efficient starting point that groups variables meaningfully, and the subsequent latent variable clustering refines these groups to achieve better overall clustering performance while maintaining computational efficiency.
Data Source
AI summary
Computer-implemented systems and methods are provided for creating a cluster structure from a data set containing input variables. Global clusters are created within a first stage, by computing a similarity matrix from the data set. A global cluster structure and sub-cluster structure are created within a second stage, where the global cluster structure and the sub-cluster structure are created using a latent variable clustering technique and the cluster structure output is generated by combining the created global cluster structure and the created sub-cluster structure.