Pretrained Vector Space Clustering for Supervised Learning Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current supervised learning methods face challenges in efficiently labeling data for training models, particularly in detecting relationships between phrases in sentences, where supervised learning is essential but labor-intensive and prone to suboptimal data set curation, leading to model performance issues with confident wrong classifications and out-of-distribution cases.
Innovation Solution
The method involves selecting input candidates from a corpus, mapping them to a pretrained vector space, clustering, and labeling, followed by partitioning into training, development, and test sets, using a fine-tuned model to improve model performance and reduce confident wrong cases, with the use of a bipartite graph to assign confidence scores and handle out-of-distribution inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If supervised learning is used to detect relationships between phrases in sentences, then model performance can be achieved, but labor-intensive labeling and suboptimal data set curation lead to confident wrong classifications and out-of-distribution cases
Solution Approach 1:
The patent applies preliminary action by pre-training a model on the entire corpus before selective labeling. The model is pre-trained to learn representations from all available data, then used to identify and label only the most informative candidates. This preliminary pre-training phase enables the system to achieve reliable model performance while significantly reducing the time required for manual labeling, as the model can make informed decisions about which data points require human annotation.
2Productivity
If supervised learning is used with limited labeled data, then labeling effort is reduced, but model performance deteriorates with confident wrong cases and out-of-distribution failures
Solution Approach 1:
The patent introduces an intermediary mechanism - a pre-trained model that acts as a bridge between the entire corpus and the final supervised model. This intermediary model is trained on all available data (labeled and unlabeled) and then used to select and prioritize candidates for human labeling. It also serves as an intermediary for generating confidence scores and identifying out-of-distribution cases during evaluation. This intermediary approach enables high labeling efficiency while maintaining model reliability by ensuring that the limited labeled data is used most effectively.
3Adaptability or versatility
If the entire corpus is used for training, then model robustness improves, but data set curation complexity increases leading to suboptimal performance
Solution Approach 1:
The patent applies segmentation by dividing the corpus into distinct functional components: (1) the entire corpus used for pre-training to ensure robustness, (2) a selected subset of informative candidates identified by the pre-trained model for human labeling, and (3) separate evaluation sets for assessing model performance. This segmentation allows the system to leverage the entire corpus for learning while managing data set curation complexity through automated candidate selection and structured partitioning, ultimately achieving both model robustness and simplified curation processes.
Data Source
AI summary
In one aspect, a method is disclosed that includes selecting a first plurality of input candidates from a corpus of data; mapping the first plurality of input candidates onto a pretrained vector space of a pretrained model; clustering the first plurality of input candidates in the pretrained vector space; adding the first plurality of input candidates to a plurality of queues for labelling; and labelling the first plurality of input candidates.