Binary ML Classifier Fairness Using Surrogate Class Variables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fairness evaluation methods for machine learning classification models require access to private and sensitive data, which is often sparse, unavailable, or illegal to collect, making it difficult to assess model fairness for protected attributes such as race or gender.
Innovation Solution
The use of surrogate class variables and probabilistic methods to infer fairness metrics, allowing evaluation without individual-level protected attribute data, by leveraging group-level probability information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing fairness methodologies collect and store private data for each individual to evaluate model fairness, then measurement precision of fairness metrics is improved, but data availability and legality deteriorate
Solution Approach 1:
The patent introduces a proxy variable (surrogate) as an intermediary to represent the protected attribute. Instead of directly using private protected attribute data (e.g., race, gender), the method uses a proxy variable that is correlated with the protected attribute but does not reveal it. This allows fairness evaluation to proceed without collecting sensitive individual data, resolving the contradiction between measurement precision and data availability.
Solution Approach 2:
The patent creates a copy of the protected attribute information through the proxy variable. The proxy variable captures statistical correlations with the protected attribute at the group level, providing sufficient information for fairness evaluation without being the actual protected attribute data. This copying approach enables fairness metrics to be calculated while avoiding direct collection of sensitive information.
2Reliability
If individual-level protected attribute data is required for fairness evaluation, then reliability of fairness assessment is improved, but ease of operation deteriorates
Solution Approach 1:
The proxy variable serves as a mediator that enables reliability to be maintained while ease of operation improves. The proxy variable is correlated with the protected attribute, allowing reliable fairness assessment, but it can be collected from public or easily accessible data sources rather than requiring sensitive individual data collection, thus improving ease of operation.
Solution Approach 2:
The patent changes the parameter from individual-level protected attribute data to group-level proxy variable data. This parameter change maintains the reliability of fairness evaluation by preserving statistical correlations while dramatically improving ease of operation, as group-level data is typically more readily available and less sensitive than individual-level data.
3Measurement precision
If private data collection is required to assess model fairness, then measurement precision is improved, but device complexity and legal compliance deteriorate
Solution Approach 1:
The proxy variable acts as a simplifying intermediary that reduces device complexity. Instead of building complex systems to collect, store, and manage sensitive individual protected attribute data, the method uses simpler group-level proxy variables that can be obtained from public datasets or existing records, thereby reducing system complexity while maintaining measurement precision.
4Ease of operation
If group-level surrogate data is used instead of individual-level protected attribute data, then ease of operation is improved, but measurement precision may deteriorate
Solution Approach 1:
The patent addresses the precision concern by carefully selecting proxy variables that have strong statistical correlations with the protected attribute. The intermediary proxy variable is chosen to preserve the essential statistical relationships needed for accurate fairness evaluation, thereby maintaining measurement precision while achieving improved ease of operation through data accessibility.
Solution Approach 2:
The method changes from individual-level to group-level parameters, which improves ease of operation. To maintain measurement precision, the patent employs statistical methods that account for the aggregation effects, ensuring that fairness metrics calculated from group-level proxy data remain accurate and reliable.
Data Source
AI summary
Methods and apparatuses for evaluating probabilistic fairness of machine learning (ML) classification models include a server that generates a first input data set, including assigning a class membership label to each of a plurality of participants based upon a probability of class membership derived from a surrogate class variable. The server generates a second input data set, including assigning a class membership label to each of the plurality of participants based upon ground truth class values. The server executes a binary classification model on the first input data set to generate inferred fairness metrics for the binary classification model. The server executes the binary classification model on the second input data set to generate actual fairness metrics for the binary classification model. The server determines a disparity in one or more fairness metrics for the binary classification model based upon a comparison of the inferred fairness metrics to the actual fairness metrics.