Position Bias Correction in Ranking Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information-retrieval systems face the challenge of position bias, where items placed in favorable positions receive more interaction, leading to inaccurate relevance estimates and potential revenue loss for advertisers.
Innovation Solution
A method is proposed to quantify and correct for position effects using machine-learning models trained with historical data, leveraging past A/B testing results to estimate position bias without disrupting live systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If items are placed in favorable positions within a display feed, then user interaction increases, but relevance estimation accuracy deteriorates
Solution Approach 1:
The patent introduces propensity scores as an intermediary variable that mediates between item position and user interaction. By modeling the probability of user interaction conditional on item characteristics and position, the system separates the position effect from the true relevance signal, allowing accurate relevance estimation while maintaining favorable item placement for user engagement
Solution Approach 2:
The patent transforms the relevance estimation problem by changing the parameter space to include position-adjusted interaction metrics. By incorporating position as an explicit parameter in the propensity score model and using inverse probability weighting, the system adjusts observed interactions to account for position bias, thereby recovering unbiased relevance estimates from biased observation data
2Measurement precision
If propensity score methods are used to address position bias, then some bias correction is achieved, but unobserved confounding is not adequately addressed
Solution Approach 1:
The patent implements a feedback mechanism where the propensity score model is iteratively refined using observed user interactions. By continuously updating the model parameters based on actual user behavior data and comparing predicted versus observed interaction rates across different positions, the system progressively reduces the impact of unobserved confounding factors and improves the accuracy of position bias correction
Solution Approach 2:
The patent performs preliminary stratification of items and users based on observed characteristics before conducting relevance estimation. By pre-segmenting the data into strata with similar propensity scores and within-strata analysis, the system reduces the impact of unobserved confounders that vary across different user-item contexts, thereby improving the reliability of bias correction
3Measurement precision
If randomized experiments are used to identify position bias, then accurate bias measurement is achieved, but cost and feasibility increase
Solution Approach 1:
The patent creates a virtual randomized experiment by synthesizing counterfactual interaction data through the propensity score model. Instead of conducting actual randomized controlled trials that would require complex experiment infrastructure and user allocation, the system copies the essential feature of randomization through statistical modeling, generating unbiased relevance estimates from existing observational data as if from a randomized experiment
Data Source
AI summary
Methods, systems, and computer programs are presented for eliminating bias while training an ML model using training data that includes past experimental data. One method includes accessing experiment results, for A/B testing of a first model, that comprise information regarding engagement with a first set of items presented to users, each item being presented within an ordered list of results. A position bias is calculated for positions within the ordered list of results where the items were presented. A machine-learning program is trained to obtain a second model using a training set comprising values for features that include the calculated position bias. The method includes detecting a second set of items to be ranked for presentation to a first user, and calculates, using the second model, a relevance score for the second set of items, which are ranked based on the respective relevance score and presented on a display.