Data Clustering via Heuristic Similarity Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accessing and labeling large and complex datasets for machine learning becomes challenging as the number of unlabeled data grows, making it difficult to train effective models without efficient clustering and labeling methods.
Innovation Solution
A method involving a computing device that receives feature vectors, applies rules to generate heuristics, and uses random forest similarity learning to create matrices for graph-based clustering, allowing for efficient labeling of data clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data sets grow in size and complexity, then the amount of available training data increases, but accessing and labeling the data becomes more challenging
Solution Approach 1:
The patent segments the large unlabeled dataset into multiple clusters based on feature similarity. Each cluster represents a subset of data with comparable characteristics, making it more manageable. The system generates heuristics to identify related values of feature vectors and creates similarity matrices to group data points into distinct clusters, which can then be labeled more efficiently than processing the entire dataset at once.
2Measurement precision
If manual labeling of all data is performed, then labeling accuracy is maintained, but time and resources required increase significantly
Solution Approach 1:
The system performs preliminary clustering of the unlabeled dataset before the actual labeling process. By pre-grouping data points into clusters based on feature similarity and generated heuristics, the system prepares the data in advance so that labelers only need to label representative samples from each cluster rather than every individual data point. This preliminary organization significantly reduces the time and resources required while maintaining labeling accuracy through the representative sampling approach.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for clustering data are disclosed. In one aspect, a method includes the actions of receiving feature vectors. The actions further include accessing rules that each relate one or more values of the feature vectors to a respective label of a plurality of labels. The actions further include, based on the rules, generating heuristics that each identify related values of the feature vectors. The actions further include, for each of the heuristics, generating a matrix that reflects a similarity of the feature vectors. The actions further include, based on the matrices that each reflects a respective similarity of the feature vectors, generating clusters that each include a subset of the feature vectors. The actions further include, for each cluster, determining a label of the plurality of labels.