Inverse Propensity Weighting for Biased Feedback in ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning assistants, such as chatbots, face biased learning due to selection bias when user feedback is non-randomly missing, leading to incomplete training data and inaccurate model performance, especially when presenting single answers or insufficient feedback is obtained.
Innovation Solution
The implementation of a method that performs option exploration and applies a second layer of feedback intervention to ensure additional feedback is obtained, using multiplicative inverse propensity weighting to mitigate bias by computing and applying an overall cost as a weight to samples, thereby addressing biased feedback and improving model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single answers are presented to users, then device complexity is reduced, but user feedback is insufficient leading to biased learning
Solution Approach 1:
The system applies inverse propensity weighting that compensates for missing feedback by giving higher weights to underrepresented feedback samples. This partial action approach allows the system to work with incomplete feedback while still achieving unbiased learning by emphasizing the partial feedback that is available rather than requiring complete feedback for all samples.
Solution Approach 2:
The system changes the parameter of feedback collection by introducing inverse propensity weights that transform the distribution of feedback samples. By modifying how feedback is weighted rather than how it is collected, the system maintains simple single-answer presentation while achieving balanced representation of all options through parameter transformation of the feedback data.
2Loss of information
If option exploration is performed, then user feedback is obtained, but selection bias occurs in propensity weighting
Solution Approach 1:
The system implements a feedback mechanism where inverse propensity weights are calculated based on the observed feedback distribution and applied to correct selection bias. The weights are derived from the propensity scores that represent the probability of each option being selected, and this feedback loop continuously adjusts the weighting to compensate for biased feedback patterns.
Solution Approach 2:
The system applies counterweighting through inverse propensity scores that balance out the selection bias. Options that are over-selected receive lower weights while under-selected options receive higher weights, creating a counterbalancing effect that eliminates the bias introduced by non-random feedback patterns in the exploration process.
3Measurement precision
If propensity weighting is applied, then bias is reduced, but computational complexity increases
Solution Approach 1:
The system performs preliminary computation of inverse propensity weights during the training phase before actual inference. By pre-calculating and storing the propensity weights, the system avoids repeated complex calculations during model updates, reducing the computational burden while maintaining the bias-correction benefits of propensity weighting.
Data Source
AI summary
Option exploration of one or more candidate options is performed in response to a user interaction. A multiplicative inverse of a propensity of showing each of the one or more candidate options to a user is computed and a multiplicative inverse of a propensity of each of the one or more candidate options being able to receive feedback is computed. An overall cost is computed by multiplying the multiplicative inverse of the propensity of showing each of the one or more candidate options by the multiplicative inverse of propensity of each of the one or more candidate options being able to receive feedback. The overall cost is applied as a weight to a corresponding sample.