Sparse Data Machine Learning Training Dataset Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning training systems face challenges in adequately detecting specific classes with low representation in training datasets, due to the use of datasets with multiple classes, which results in low performance for certain classes.
Innovation Solution
A system and method that involve applying linear and non-linear dimension reduction processes to generate reduced feature sets, clustering records based on these sets, and generating a training dataset by labeling records based on cluster purity scores, allowing for the training of a machine learning model using a supervised process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If training datasets include multiple classes, then the system can identify different classes of data, but the performance for specific classes with low representation deteriorates
Solution Approach 1:
The patent segments the training process into two distinct phases: unsupervised pre-training on all available data to learn general features, followed by supervised fine-tuning on the specific target class. This segmentation allows the model to first build general knowledge without class imbalance issues, then specialize on the target class with high precision, resolving the contradiction between multi-class capability and single-class accuracy.
Solution Approach 2:
The patent applies preliminary unsupervised pre-training on the complete dataset before performing supervised training on the target class. This preliminary action enables the model to learn robust feature representations from all data first, which then serves as a strong foundation for subsequent specialized training, improving performance on low-representation classes.
2Measurement precision
If data tagging is performed manually to ensure accurate labeling, then the quality of training data improves, but the time and resource consumption increases significantly
Solution Approach 1:
The patent performs preliminary unsupervised clustering and pseudo-labeling on the entire dataset before manual verification. This preliminary action generates high-quality initial labels automatically, reducing the amount of data that requires time-consuming manual tagging while maintaining overall labeling accuracy.
Solution Approach 2:
The system uses self-supervised learning where the model generates its own training labels through unsupervised pre-training and pseudo-labeling. This self-service approach creates a large pool of pre-labeled data that can be quickly verified or corrected, dramatically reducing the time investment required for manual data preparation compared to traditional fully manual labeling.
Data Source
AI summary
Systems and methods for generating training datasets for use in machine learning are disclosed. A plurality of data records are received. Each record in the plurality of records includes a set of features. A first reduced dimension feature set is generated by applying a linear dimension reduction process to the set of features and a second reduced dimension feature set is generated by applying a non-linear dimension reduction process to the first reduced dimension feature set. The set of records is clustered based on the second reduced dimension feature set and a training dataset is generated by labeling each record in the plurality of records based on a cluster associated with each record. A machine learning model is trained by applying a supervised training process based on the training dataset.