Clustering Data Points Using Iterative Similarity Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional clustering solutions fail to separate the computation of similarity functions from the clustering process, often requiring data points to be embedded in normed linear spaces, which is not feasible for nonlinear similarity functions, and necessitate pre-specifying the number of clusters, limiting their effectiveness.
Innovation Solution
A computer-implemented method and system that selects data points, evaluates similarity functions, and iteratively groups them into clusters, allowing for the estimation and updating of similarity functions to determine cluster membership, enabling the identification of clusters without pre-specifying their number and accommodating nonlinear relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional clustering solutions treat computation of similarity function and clustering process as one process, then the process is simpler, but the flexibility and accuracy are reduced because practitioners cannot separate similarity function computation from clustering and must embed data in normed linear spaces
Solution Approach 1:
The patent divides the clustering process into two separate modules: a similarity function computation module that computes pairwise similarity values between data points, and a clustering module that uses these similarity values to form clusters. This segmentation allows practitioners to independently select and optimize the similarity function (including nonlinear functions like decision trees) without being constrained by the clustering algorithm, and vice versa, thereby resolving the contradiction between flexibility and complexity.
2Measurement precision
If the number of clusters is specified in advance, then the clustering process is more straightforward, but the accuracy is reduced when the actual number of clusters is unknown
Solution Approach 1:
The patent implements a feedback mechanism where the clustering module continuously evaluates the quality of formed clusters based on the similarity function values and iteratively adjusts the number of clusters. The system monitors clustering quality metrics and uses this feedback to determine when to stop forming new clusters, eliminating the need for pre-specification while maintaining ease of operation through automated quality assessment.
3Adaptability or versatility
If data points are required to be embedded in normed linear spaces, then the mathematical framework is simpler, but the applicability is reduced for nonlinear similarity functions
Solution Approach 1:
The patent introduces a similarity matrix as an intermediary representation that captures pairwise similarity relationships between data points without requiring embedding in normed linear spaces. This similarity matrix serves as a mediator between the raw data and the clustering process, allowing nonlinear similarity functions to be applied directly while maintaining mathematical tractability through the matrix structure, thus resolving the contradiction between applicability and complexity.
Data Source
AI summary
Systems and methods for clustering a group of data points based on a measure of similarity between each pair of data points in the group are provided. A pairwise similarity function can be estimated for each pair of data points in the group. A clustering algorithm can be executed to create clusters and associate data points with the clusters using the pairwise similarity function. The algorithm can be iterated multiple times until a stopping condition is reached in order to reduce variance in the output of the algorithm. The pairwise similarity function for each pair of data points can be updated between iterations of the algorithm and the results of each iteration can be aggregated. The data in each data point associated with a cluster can be consolidated into a consolidated data point.