Semi-Supervised Clustering for Unstructured Data Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning-based classification systems for objects in unstructured data, such as images in videos, perform poorly when faced with variations in physical and environmental conditions different from their training samples, leading to unstable performance in real-world applications.
Innovation Solution
A semi-supervised method that extracts feature vectors from unstructured data, clusters them based on similarity, and iteratively removes outlier vectors to create a more robust training dataset, allowing for retraining of the classification system to improve accuracy across varying conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing machine learning based classification systems are trained using a training dataset under ideal conditions, then high classification accuracy is obtained in experimental conditions, but performance becomes unstable under real conditions with varying physical and environmental conditions
Solution Approach 1:
The system performs preliminary clustering and outlier detection on the training dataset to identify and remove feature vectors that represent atypical or erroneous conditions before training the classification model. This preliminary action ensures that only high-quality, representative feature vectors are used for training, improving both accuracy and reliability when deployed in real-world conditions with varying physical and environmental parameters.
Solution Approach 2:
The system implements an iterative feedback mechanism where the classification model is trained, evaluated, and then used to identify outliers in the training data. The identified outliers are removed and the model is retrained, creating a feedback loop that continuously improves performance stability. This feedback process allows the system to adapt to the actual distribution of real-world data and become more robust to variations in lighting, camera angles, and environmental conditions.
2Measurement precision
If feature vectors are clustered based on similarity with multiple iterations and outliers are removed, then a more robust training dataset is created for improved accuracy, but the processing time and computational complexity increase
Solution Approach 1:
The system performs a limited number of clustering iterations (e.g., 3-5 iterations) rather than exhaustive iterations, removing only the most significant outliers in each pass. This partial action approach achieves sufficient improvement in classification accuracy without the prohibitive computational cost of exhaustive outlier removal, balancing processing time with performance gains.
Solution Approach 2:
The system extracts and removes only the most critical outlier feature vectors from the training dataset rather than attempting to process or analyze all feature vectors equally. By identifying and removing a small subset of high-impact outliers through clustering, the system achieves robust training dataset creation with reduced computational overhead compared to comprehensive data processing approaches.
Data Source
AI summary
System and method for classifying data objects occurring in an unstructured dataset, comprising: extracting feature vectors from the unstructured dataset, each feature vector representing an occurrence of a data object in the unstructured dataset; classifying the feature vectors into feature vector sets that each correspond to a respective object class from a plurality of object classes; for each feature vector set: performing multiple iterations of a clustering operation, each iteration including clustering feature vectors from the feature vector set into clusters of similar feature vectors and identifying outlier feature vectors, wherein for at least one iteration after a first iteration of the clustering operation, outlier feature vectors identified in a previous iteration are excluded from the clustering operation; and outputting a key cluster for the feature vector set from a final iteration of the multiple iterations, the key cluster including a greater number of similar feature vectors than any of the other clusters of the final iteration; and assembling a dataset that includes at least the feature vectors from the key clusters of the feature vector sets.