Auto Labeling Verification Using Uncertainty Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional auto labeling methods are labor-intensive and costly, requiring a large number of skilled inspectors to verify labels, which slows down the process and increases costs due to low throughput and difficulty in acquiring enough skilled labor to match the throughput of automatic labeling devices.
Innovation Solution
A method using uncertainty scores for auto-labeled labels to automatically verify and correct labels, involving a learning device that inputs unlabeled images into object detection and convolution networks, applies pooling and deconvolution operations, and uses classifiers to generate uncertainty scores, allowing for selective verification and re-training of classifiers based on labeled test images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large number of skilled inspectors are hired to verify auto-labeled images, then labeling accuracy is improved, but cost and time requirements increase
Solution Approach 1:
The system uses uncertainty scores generated by the auto-labeling device itself to identify which images require human verification. This self-assessment mechanism allows the system to automatically filter and prioritize images that need inspection, reducing the burden on inspectors and improving overall efficiency without sacrificing accuracy
Solution Approach 2:
Instead of having all inspectors verify all images, the system applies partial verification only to images with high uncertainty scores. This selective verification approach maintains labeling accuracy for critical cases while significantly reducing the total time and resources required for the verification process
2Productivity
If more inspectors are hired to match the throughput of automatic labeling devices, then productivity is improved, but cost increases
Solution Approach 1:
The auto-labeling device performs self-verification using uncertainty scores to identify images requiring human review. This reduces the volume of work for inspectors and allows fewer inspectors to handle the same throughput, directly addressing the contradiction between productivity and inspector quantity
Solution Approach 2:
The verification workload is segmented into two categories: images with high uncertainty scores that require inspector verification, and images with low uncertainty scores that can be automatically accepted. This segmentation allows the system to maintain high throughput while minimizing the number of inspectors needed
3Device complexity
If conventional auto labeling is used without uncertainty-based verification, then process simplicity is maintained, but labeling accuracy degrades
Solution Approach 1:
The system incorporates uncertainty scores as feedback from the auto-labeling process to guide verification decisions. This feedback mechanism allows the system to automatically identify and prioritize images that need human review, improving accuracy without significantly increasing process complexity
Solution Approach 2:
The verification process is applied partially only to images with high uncertainty scores rather than all images. This selective approach maintains process simplicity for the majority of images while improving accuracy for the critical subset that requires verification
Data Source
AI summary
A method for training an auto labeling device performing automatic verification using uncertainty of labels is provided. The method includes steps of: a learning device (a) (i) inputting unlabeled training images into (i-1) an object detection network to generate bounding boxes for training and (i-2) a convolution network to generate feature maps for training, and (ii) allowing an ROI pooling layer to generate pooled feature maps for training and inputting the pooled feature maps for training into a deconvolution network to generate segmentation masks for training; and (b) (i) inputting the pooled feature maps for training into at least one of (i-1) a first classifier to generate first per-pixel class scores for training and first mask uncertainty scores for training and (i-2) a second classifier to generate second per-pixel class scores for training and second mask uncertainty scores for training and (ii) training the first classifier or the second classifier.