Difficulty-Based Training Data Selection for Uncertain ML Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning modules for safety-critical applications struggle with variable uncertainties due to poor quality of measurement data, inherent object recognition difficulties, and adversarial manipulations, leading to suboptimal training data sets and overconfidence in classifications.
Innovation Solution
A method to augment training data sets by using a difficulty function to assess the uncertainty of samples, selecting candidate samples based on predetermined criteria, and incorporating them into the training set, along with an encoder-decoder architecture to create a working space that enhances the machine learning module's ability to handle ambiguous situations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large number of training samples are used to improve classification accuracy, then the model can handle more situations, but the training efficiency decreases and redundant information increases
Solution Approach 1:
The patent changes the parameter of training data selection from random or uniform sampling to difficulty-based sampling. By introducing a difficulty function that evaluates samples based on uncertainty metrics, the system prioritizes hard samples (those with high uncertainty) over easy samples, thereby improving training efficiency and accuracy without requiring excessively large data sets
Solution Approach 2:
The patent applies local quality by treating different training samples with different levels of importance based on their difficulty. Instead of uniform treatment, hard samples (ambiguous, occluded, or adversarial cases) are selected with higher probability for training, while easy samples are selected less frequently, optimizing the training process
2Productivity
If training data includes only clear and unambiguous samples, then training converges faster, but the model fails to handle uncertain or ambiguous situations in real applications
Solution Approach 1:
The patent converts the harmful effect of ambiguous and uncertain samples (which slow down training and cause confusion) into a benefit by deliberately selecting these hard samples for training. The difficulty function identifies samples with high uncertainty, and these same samples are prioritized for training, enabling the model to learn from challenging cases and improve its ability to handle ambiguous situations
Solution Approach 2:
The patent performs preliminary action by pre-evaluating all training samples using the difficulty function before training begins. This pre-assessment categorizes samples by difficulty level, allowing the training process to systematically prioritize hard samples that need more attention, ensuring the model is exposed to challenging cases from the outset
3Ease of operation
If the model is trained to be highly confident in its outputs, then decision-making is simplified, but the model becomes overconfident and unreliable in uncertain situations
Solution Approach 1:
The patent implements feedback by using uncertainty estimates to guide the training process. The difficulty function provides feedback about which samples are hard to classify, and this information is fed back into the training selection process. Additionally, the model's uncertainty outputs are used to adjust training priorities, creating a closed-loop system that continuously improves confidence calibration
Data Source
Figure 1
Figure 2~3(h)
Figure 4~5
AI summary
A method (100) for obtaining and/or augmenting a training data set (11*) for a machine learning module (1), comprising: • providing (110) samples (11a) of training input data annotated with labels (13a) with respect to a given problem; • obtaining (120), from the samples (11a) of training input data and the associated labels (13a), a difficulty function (14) providing a measure for the difficulty (14a) of evaluating samples (11a) with respect to the given problem; • obtaining (130) at least one candidate sample (15) of training input data, and/or a representation (25) thereof in the working space (20); • calculating (140), by means of the difficulty function (14), a measure for the difficulty (14a) of evaluating this candidate sample (15), and/or the representation (25) thereof, with respect to the given problem; and in response to this difficulty meeting a predetermined criterion (150), including (160) the candidate sample (15) in the training data set (11*).