Classification Model Training Using Confidence-Scored Auto-Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine-learning systems to detect and recognize images is labor-intensive, time-consuming, and requires large amounts of high-quality training data, which is often expensive to produce.
Innovation Solution
Utilizing a combination of multimodal large language models (LLMs) and domain-specific models to generate annotated training data with confidence scores, which are then used to train second machine-learned models by modifying their parameters based on these scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation of training data is used, then data quality is high, but the process is labor-intensive and time-consuming
Solution Approach 1:
The system uses machine-learned models to automatically annotate training data without human intervention. The models generate labels and confidence scores autonomously, transforming the manual annotation process into an automated self-service system that maintains high data quality while dramatically increasing labeling speed
Solution Approach 2:
The system incorporates confidence scores from machine-learned models as feedback mechanisms. Samples with high confidence scores are used to train new models, while low confidence samples can be reviewed or used to refine existing models, creating a feedback loop that improves data quality and model performance iteratively
2Measurement precision
If large amounts of training data are collected, then model accuracy improves, but production cost and time increase
Solution Approach 1:
The system performs preliminary model training on a subset of data to create initial annotators. These preliminary models then rapidly annotate large portions of the training data, allowing the system to scale data collection without proportionally increasing training time for the main model
Solution Approach 2:
The system creates multiple machine-learned models that copy and replicate the annotation capabilities of highly accurate models. These copied models can independently generate training data at scale, effectively multiplying the productivity of the original model without requiring proportional increases in human annotation time
3Productivity
If existing machine-learned models are used for annotation, then labeling speed increases, but data quality may decrease
Solution Approach 1:
The system changes the parameter of model confidence by using confidence scores to filter and select high-quality annotations. By adjusting the confidence threshold parameter, the system can control the balance between labeling speed and data quality, ensuring only high-confidence annotations are used for training
Solution Approach 2:
The system uses partial annotation by machine-learned models rather than requiring complete manual verification. By using models to annotate the majority of data with high confidence, and only reserving manual review for edge cases, the system achieves both high productivity and maintained data quality through selective application of human expertise
Data Source
AI summary
Methods, systems, devices, and non-transitory computer readable media for training machine-learning models are provided. The disclosed technology can include receiving input samples associated with classification concepts. Based on inputting the input samples into a first plurality of machine-learned models, classification outputs comprising labels and confidence scores can be generated. The first plurality of machine-learned models can comprise one or more multimodal large language models (LLMs) and one or more domain-specific models. Annotated input samples comprising the input samples, the classification outputs, and identifiers that identify each of the first plurality of machine-learned models that generated each of the classification outputs can be generated. Furthermore, based on the annotated input samples, one or more second machine-learned models can be trained. The training can comprise modifying parameters of the one or more second machine-learned models based on the confidence scores.