Video Relabeling Scores for Accurate ML Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for labeling data for a machine learning model are time-consuming, prone to errors, and inefficient, leading to resource wastage and generation of erroneous models due to inaccurate labels.
Innovation Solution
A video system that determines when to relabel data, and relabeling data, for a machine learning model, by identifying mislabeled data, correcting it, and retraining the model with correctly labeled data, using a video system that calculates event severity scores, user feedback scores, reviewer mistrust scores, and reviewer bias scores to generate new labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If manual labeling techniques are used for machine learning data, then labeling can be performed with simple processes, but time consumption increases and error rates increase
Solution Approach 1:
The patent introduces an automated labeling system that acts as an intermediary between raw data and the machine learning model. This system uses pre-trained models and algorithms to generate labels automatically, reducing direct human involvement while maintaining labeling quality. The automated system serves as a mediator that processes data through structured pipelines including confidence scoring and verification mechanisms.
Solution Approach 2:
The patent replaces manual mechanical labeling processes with automated computational systems. Instead of human experts manually annotating data, the system uses machine learning models, confidence thresholds, and automated verification algorithms to generate and validate labels. This substitution dramatically reduces time consumption while maintaining or improving labeling accuracy.
2Ease of manufacture
If manual labeling techniques are used for machine learning data, then labeling can be performed with simple processes, but error rates increase leading to erroneous models
Solution Approach 1:
The patent implements feedback mechanisms where the automated labeling system continuously monitors and adjusts its performance. Confidence scores are calculated for each label, and labels below certain thresholds are flagged for review or re-labeling. The system learns from verification results and adjusts its labeling strategies, creating a closed-loop feedback system that improves reliability while maintaining automation.
Solution Approach 2:
The patent applies preliminary actions by using pre-trained models and existing knowledge bases to generate initial labels before final verification. This preliminary labeling stage allows the system to establish a baseline that can be quickly verified or corrected, reducing the need for complete manual re-labeling and improving overall accuracy through multi-stage validation.
3Reliability
If comprehensive data labeling is performed to improve model accuracy, then model performance improves, but computing resources are wasted on already accurate labels
Solution Approach 1:
The patent applies partial action by selectively labeling only the portion of data that requires it. Instead of comprehensively relabeling entire datasets, the system uses confidence scoring to identify specific data points or categories that need attention. This targeted approach ensures model accuracy is improved where needed while avoiding unnecessary computation on already accurate labels.
Solution Approach 2:
The patent dynamically adjusts labeling parameters such as confidence thresholds, sampling rates, and verification depths based on the current state of the model and data quality metrics. When model accuracy is sufficient, the system reduces labeling intensity; when accuracy drops below thresholds, it intensifies labeling efforts. This adaptive parameter adjustment optimizes resource usage while maintaining model performance.
4Reliability
If frequent model retraining is performed to maintain accuracy, then model performance is maintained, but productivity decreases due to repeated processing
Solution Approach 1:
The patent implements periodic action by scheduling model retraining based on performance degradation thresholds rather than continuously. The system monitors model accuracy over time and triggers retraining only when performance drops below specified thresholds or when significant data drift is detected. This periodic approach maintains model accuracy while avoiding unnecessary retraining cycles that would reduce productivity.
Data Source
AI summary
A device may receive video data identifying videos, and may process the video data with a machine learning model, to determine classifications. The device may generate labels for the videos, and may calculate event severity scores and event severity labels. The device may calculate event severity incoherence scores, and may calculate user feedback scores of users associated with the device. The device may determine reviewer mistrust scores, and may calculate time review scores. The device may calculate reviewer bias scores, and may determine relabeling scores for the videos based on the event severity incoherence scores, the user feedback scores, the reviewer mistrust scores, the time review scores, and the reviewer bias scores. The device may generate new labels for one or more of the videos based on the relabeling scores, and may retrain the machine learning model, with the new labels, to generate a retrained machine learning model.