Entity Matching Confidence Calibration with Regression-Based Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ML systems struggle with selecting appropriate confidence thresholds for predictions, leading to inaccurate task execution due to similar accuracy values across varying confidence levels, resulting in incorrect task automation.
Innovation Solution
A machine learning system that determines confidence-to-accuracy relationships through a training pipeline, using regression models to define confidence thresholds for selective prediction filtering during inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If confidence thresholds are configured based on target accuracy, then accurate predictions are maximized, but predictions with low confidence are incorrectly deemed acceptable
Solution Approach 1:
The confidence range is segmented into multiple bins (e.g., 0.0-0.1, 0.1-0.2, ..., 0.9-1.0) to create granular accuracy measurements. This segmentation allows the system to identify that different confidence levels within the same accuracy threshold have different reliability characteristics, enabling more precise confidence threshold selection.
Solution Approach 2:
The system calculates accuracy for each confidence bin individually, which is a partial action approach. Instead of calculating a single overall accuracy, it performs multiple partial accuracy calculations across different confidence ranges, allowing identification of the specific confidence level where accuracy first meets the target threshold.
2Reliability
If multiple confidence values have the same accuracy, then target accuracy is achieved, but it becomes difficult to select an appropriate confidence threshold
Solution Approach 1:
The system pre-calculates accuracy for each confidence bin during the training phase and stores this information. This preliminary action allows the system to quickly reference pre-computed accuracy data during inference, making confidence threshold selection straightforward by identifying the lowest confidence bin that meets the target accuracy requirement.
Solution Approach 2:
The system provides feedback by comparing the target accuracy against the accuracy of each confidence bin. This feedback mechanism enables automatic identification of the appropriate confidence threshold by finding the first bin where the accuracy meets or exceeds the target, eliminating manual trial-and-error threshold selection.
3Productivity
If confidence thresholds are set too low, then more predictions are accepted, but incorrect predictions increase
Solution Approach 1:
The system dynamically adjusts the confidence threshold based on the relationship between confidence bins and their corresponding accuracies. Rather than using a fixed or arbitrarily low threshold, the system determines the optimal threshold by identifying the confidence level where accuracy first meets the target, balancing productivity and reliability adaptively.
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving a first set of predictions generated by a ML model during execution of a training pipeline to train the ML model, each prediction in the first set of predictions being associated with a confidence, determining a set of confidence bins based on confidences of the first set of predictions, for each confidence bin in the set of confidence bins, providing an accuracy, processing the set of confidence bins and accuracies through a regression model to provide one or more regressions, each regression representing a confidence-to-accuracy relationship, defining a set of confidence thresholds based on at least one regression of the one or more regressions, and during an inference phase, applying the set of confidence thresholds to selectively filter predictions from a second set of predictions generated by the ML model.