Graph-Based Data Quality Assessment for Unsupervised ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unsupervised machine learning algorithms face significant challenges in handling large datasets, such as those in gigabytes or petabytes, due to high computational loads, which affects their performance and efficiency in identifying representative subsets of data for effective modeling.
Innovation Solution
The approach involves converting unlabeled data into a graph structure, where nodes represent data points and weighted edges represent similarity, and computing metrics to identify dissimilar data points, allowing for the removal of less valuable data points to create a subset that maintains the original dataset's distribution, thereby improving algorithm performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If unsupervised machine learning algorithms process large datasets to identify representative subsets, then model effectiveness is improved, but computational load increases significantly
Solution Approach 1:
The patent segments the large dataset into smaller manageable subsets by converting data points into a graph structure and applying clustering algorithms. This allows the system to process and analyze representative subsets rather than the entire dataset, reducing computational load while maintaining model effectiveness.
Solution Approach 2:
The patent extracts key representative data points from the large dataset by computing metrics such as node centrality, betweenness, and clustering coefficients in the graph structure. These extracted representative points capture the essential characteristics of the full dataset, enabling effective unsupervised learning with reduced computational requirements.
2Productivity
If the dataset is reduced to a subset for processing, then computational efficiency is improved, but data representativeness may be compromised
Solution Approach 1:
The patent employs feedback mechanisms by computing multiple graph metrics (node centrality, betweenness, clustering coefficients) and iteratively refining the selection of representative data points. This feedback loop ensures that the extracted subset accurately reflects the distribution and characteristics of the original large dataset, maintaining data representativeness while improving computational efficiency.
Solution Approach 2:
The patent changes parameters by transforming the dataset into a graph structure with various attributes (nodes, edges, weights, centrality measures). This parameter transformation enables the system to identify and extract representative subsets that preserve the statistical properties and distribution of the original data, achieving both efficiency and representativeness.
Data Source
AI summary
Techniques for qualitatively assessing unlabeled data in an unsupervised machine learning environment are disclosed. In one example, a method comprises the following steps. A dataset of unlabeled data points is converted into a graph structure. Nodes of the graph structure represent the unlabeled data points in the dataset and weighted edges between at least a portion of the nodes represent similarity between the unlabeled data points represented by the nodes. A metric is computed for each node of the graph structure. A value generated by the metric for a given node represents a measure of dissimilarity between the corresponding unlabeled data point of the given node and one or more other unlabeled data points of one or more other nodes. A subset of the dataset is generated by removing one or more unlabeled data points from the dataset based on one or more values of the computed metric.