Hybrid Unsupervised Supervised Data Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current unsupervised learning algorithms face challenges in effectively categorizing unstructured and unlabelled data, leading to misclassification, noise in clusters, and a rigorous process for re-categorizing updated datasets, which is resource-intensive and time-consuming.
Innovation Solution
A method that generates a structured and unlabelled training dataset, uses an unsupervised learning algorithm to cluster data, labels selected clusters, and then employs a supervised learning algorithm to create a trained model that assigns labels to datapoints, with the option to retrain the model using a pre-trained supervised model and unsupervised algorithm for updated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If unsupervised learning algorithms are used to categorize unstructured and unlabelled data, then the process is less complex and can be performed in real-time, but misclassification and noise in clusters occur
Solution Approach 1:
The patent combines unsupervised learning algorithms with supervised learning algorithms to create a hybrid system. The unsupervised algorithm performs initial clustering on unlabelled data, and then a supervised learning component refines these clusters by learning from labelled examples, thereby reducing misclassification and noise while maintaining the real-time processing capability of unsupervised methods
Solution Approach 2:
The patent introduces a labelled data set as an intermediary between the unsupervised learning algorithm and the final categorization output. The supervised learning algorithm uses this intermediate labelled data to learn mapping relationships, which then guide the unsupervised clustering process to produce more accurate and reliable categories
2Ease of operation
If unsupervised learning algorithms are used for categorization, then unlabelled data can be processed easily, but re-categorizing updated datasets is rigorous and resource-intensive
Solution Approach 1:
The patent performs preliminary supervised learning training on a labelled data set before processing updated unlabelled data. This pre-trained model captures the underlying patterns and relationships, so when new unlabelled data arrives, the system can quickly apply the learned knowledge without needing to re-categorize from scratch, significantly reducing re-categorization time and resource requirements
3Speed
If unsupervised learning algorithms are used, then real-time processing is achieved, but new distinct patterns in updated data may be missed
Solution Approach 1:
The patent implements a feedback mechanism where the supervised learning component continuously learns from labelled examples and updates its understanding of data patterns. This feedback loop allows the system to adapt to new distinct patterns in updated unlabelled data while maintaining real-time processing speed, as the supervised model refines its predictions based on accumulated knowledge rather than reprocessing all data
Data Source
AI summary
Embodiments provide methods and systems for processing unstructured and unlabelled data. A method includes generating, by a processor, a structured and unlabelled training dataset from an unstructured and unlabelled dataset. The method includes categorizing the structured and unlabelled training dataset into a plurality of clusters by executing an unsupervised algorithm. Each cluster of a selected set of clusters from the plurality of clusters is labelled with an applicable label from a set of labels. The method includes executing a supervised algorithm to generate a trained supervised model using a labelled training dataset including the set of labels and an input dataset generated from plurality of datapoints present in each cluster of the selected set of clusters. The method includes generating a Labelled Data1 (LD1) by executing the trained supervised model configured to assign applicable label from the set of labels to each datapoint of the structured and unlabelled training dataset.