Data Collection System Using Confidence Scores to Select Target Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face inefficiencies in data collection, where early data collection is effective but later data collection yields less impactful information, hindering performance improvement over time.
Innovation Solution
A data collection system that selects target data with lower confidence levels and actively collects training data to increase confidence, using domain adversarial neural networks (DANN) to calculate a usefulness score and adjust camera parameters for optimal data acquisition, thereby improving model accuracy in a short learning time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data collection continues over time to improve model accuracy, then model accuracy improves, but the effectiveness of data collection diminishes as learning progresses
Solution Approach 1:
The system performs preliminary identification of low-confidence target data before full model training. By pre-selecting which data points would be most beneficial for training (those with low confidence scores), the system prepares the optimal training dataset in advance, avoiding wasted time collecting ineffective data during the training process.
Solution Approach 2:
The system uses confidence scores as feedback to iteratively improve data collection. After each training iteration, the model's confidence scores on unlabeled data are evaluated, and this feedback guides the selection of new target data for labeling. This closed-loop feedback mechanism ensures that data collection continuously adapts to the model's current state, maximizing the effectiveness of each data collection cycle.
2Reliability
If labeled data is collected to improve model performance, then model accuracy improves, but collection cost increases
Solution Approach 1:
Instead of uniformly collecting and labeling all data, the system applies local quality by selectively identifying specific data points (those with low confidence scores) that would benefit most from labeling. This localized approach to data selection ensures that labeling resources are concentrated on the most valuable data points rather than being wasted on data that the model already handles confidently.
Solution Approach 2:
The system changes the parameter of data selection from random or uniform sampling to confidence-score-based selection. By using the confidence score as a selection parameter, the system transforms the data collection strategy to prioritize data points that will have the greatest impact on model improvement, thereby reducing the total amount of labeled data needed.
3Adaptability or versatility
If random data collection is used, then data diversity is maintained, but data effectiveness for model improvement decreases
Solution Approach 1:
The system replaces random data collection with feedback-driven selection. The confidence score serves as feedback that indicates which data points are most effective for training. This feedback mechanism ensures that data collection is neither purely random nor completely deterministic, but rather adaptively selects diverse data points based on their potential to improve the model.
Solution Approach 2:
The system performs preliminary evaluation of data effectiveness using confidence scores before actual data collection. This preliminary action identifies which data points would be most beneficial, allowing the system to maintain diversity while ensuring effectiveness by pre-filtering based on the model's current confidence levels.
Data Source
AI summary
A computer-readable storage medium storing a data collection program for causing a computer, which is configured to collect training data used for training a machine learning model, to perform processing. In an example, the processing includes: selecting a target data having a confidence level lower than a predetermined value, the confidence level corresponding to a confidence for an output from the machine learning model when collected data is input into the machine learning model; and collecting, for a target object related to the selected target data, the training data such that the confidence is high.