ML Fairness Scoring With Confidence-Based Bias Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for computing the disparate impact ratio of machine learning models do not consider confidence values of scored record predictions, leading to inaccurate results due to treating records with varying confidence levels uniformly.
Innovation Solution
An approach that identifies low confidence prediction records, perturbs them into different fairness groups, and re-scores these records to generate perturbed scores, changing predictions if the perturbed confidence value exceeds a threshold and differs from the original prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If confidence values are not considered in computing disparate impact ratio, then the computation is simpler and faster, but the measurement precision of data bias assessment deteriorates
Solution Approach 1:
The patent introduces confidence values as an additional parameter to the disparate impact ratio computation. Instead of simply counting favorable outcomes, the method weights these outcomes by their confidence values, transforming the computation from a simple ratio to a confidence-weighted ratio that more accurately reflects model reliability across fairness groups.
Solution Approach 2:
The patent segments the computation process into distinct steps: identifying low-confidence predictions, perturbing fairness group attributes for these specific records, re-scoring perturbed records, and adjusting the disparate impact ratio based on confidence-weighted favorable outcomes. This segmentation allows the complex computation to be managed systematically while maintaining accuracy.
2Measurement precision
If low confidence predictions are adjusted through perturbation and re-scoring, then the measurement precision of disparate impact ratio improves, but the loss of time in computation increases
Solution Approach 1:
The patent applies perturbation and re-scoring only to a subset of records - specifically those with low confidence predictions - rather than processing all records uniformly. This partial action approach focuses computational resources on the records that most impact measurement accuracy while avoiding unnecessary computation on high-confidence records.
Solution Approach 2:
The patent applies different processing quality to different records based on their confidence levels. High-confidence records receive standard processing while low-confidence records undergo additional perturbation and re-scoring steps. This local quality differentiation optimizes the balance between computation time and measurement precision by applying intensive processing only where needed.
Data Source
AI summary
An approach is provided in which the approach receives scored records that include a selected scored record comprising a first fairness group attribute, a first prediction, and a first confidence value corresponding to the first prediction. The approach perturbs the selected scored record to a second fairness group attribute in response to determining that the first confidence value is below a confidence threshold. The approach scores the perturbed record to generate a second prediction and a second confidence value corresponding to the second prediction. The approach modifies the selected scored record by changing the first prediction to the second prediction in response to determining that the second prediction is different from the first prediction and that the second confidence value is higher than the confidence threshold.