Calibration Model for Label Inconsistencies in Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly those using medical data, face accuracy issues due to individual differences in label assignment by users, leading to low reliability and inferior performance.
Innovation Solution
An apparatus and method that utilize a calibration model to calibrate individual characteristics in label assignment, generating a calibration parameter to correct label inconsistencies, thereby improving the accuracy of a target model by using this calibration data during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If labels are manually assigned by multiple users to prepare large numbers of training data, then the quantity of training data increases, but individual differences in label assignment reduce reliability and accuracy
Solution Approach 1:
A calibration model is introduced as an intermediary between user-assigned labels and the target model training. The calibration model learns the relationship between different users' labeling patterns and generates calibrated labels that reduce individual differences. This mediator transforms unreliable user labels into more reliable calibration labels without reducing the quantity of training data.
Solution Approach 2:
The system changes the parameters of labels by applying calibration parameters generated from the calibration model. These calibration parameters adjust the original user-assigned labels to compensate for individual differences, transforming the label characteristics while preserving the training data quantity.
2Reliability
If labels are assigned by a single person or few people to maintain consistency, then reliability of labels improves, but the quantity of training data becomes insufficient
Solution Approach 1:
The calibration model serves as an intermediary that enables multiple users to contribute labels while maintaining reliability. By calibrating labels from multiple users through the calibration model, the system achieves both increased data quantity and maintained reliability, resolving the trade-off between these two parameters.
3Measurement precision
If calibration models are trained for each user to calibrate individual characteristics, then accuracy of target model improves, but device complexity increases
Solution Approach 1:
The calibration model is designed with multi-functionality to handle multiple users' labeling patterns. Instead of requiring separate complex calibration systems for each user, a universal calibration model learns from and calibrates labels from multiple users simultaneously, reducing overall system complexity while maintaining accuracy.
4Measurement precision
If calibration data is generated and used to train target model, then accuracy of machine learning improves, but loss of time increases due to additional calibration process
Solution Approach 1:
The calibration model is trained in advance using user-assigned labels before the target model training begins. This preliminary calibration action prepares calibrated labels that can be directly used for target model training, reducing the time penalty during the main training process by performing calibration work beforehand.
Data Source
AI summary
An apparatus of machine learning includes processing circuitry. The processing circuitry uses a first calibration model that receives, as input, first processing data and a first processing label assigned by a user to the first processing data and outputs calibration data relating to calibration of individual characteristics in label assignment by the first user, and trains a target model based on at least the first processing data and the calibration data or a calibrated label having individual characteristics calibrated using the calibration data.