Document Retrieval System Positional Bias Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document retrieval systems face challenges in accurately computing document relevance scores due to positional biases, where user selections are influenced by the position of documents in search results rather than their true relevance, leading to skewed relevance scores.
Innovation Solution
A supervised machine learning module uses user clicks and other runtime selections to learn and distinguish true document relevance from positional biases, employing a neural network to model the probability of document selection based on both document features and position, allowing for the separation of true relevance from positional effects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If user clicks are used as implicit proxies for explicit relevance specifications, then the time and expense of manual relevance specification is reduced, but the accuracy of relevance scores deteriorates due to positional bias influencing user selections
Solution Approach 1:
The patent extracts and separates the positional bias component from the user click data. By modeling the probability of selection as a function of both relevance and position, the system isolates the positional bias effect and removes it from the relevance calculation, leaving only the true relevance signal.
Solution Approach 2:
The patent introduces a probabilistic model as an intermediary between raw user clicks and final relevance scores. This model acts as a mediator that adjusts the observed click data by accounting for positional effects, transforming biased observations into unbiased relevance estimates.
2Ease of operation
If documents are ranked by position in search results, then user selection behavior is influenced by position, but this positional influence makes it difficult to determine true document relevance
Solution Approach 1:
The patent uses feedback from observed user selection behavior across multiple positions to learn and correct for positional bias. By continuously monitoring which documents are selected at which positions and adjusting the model accordingly, the system compensates for the distorting effect of presentation order on relevance measurement.
3Device complexity
If a simple relevance calculation method is used, then computational complexity is reduced, but the ability to account for positional bias and compute accurate relevance scores is compromised
Solution Approach 1:
The patent changes the parameters of the relevance calculation by introducing position as an additional variable and modeling the relationship between position, relevance, and user selection probability. This parameter expansion allows the system to account for positional bias while maintaining a computationally tractable probabilistic framework.
Data Source
AI summary
A document retrieval system tracks user selections of documents from query search results and uses the selections as proxies for manual user labeling of document relevance. The system trains a model representing the significance of different document features when calculating true document relevance for users. To factor in positional biases inherent in user selections in search results, the system learns positional bias values for different search result positions, such that the positional bias values are accounted for when computing document feature features that are used to compute true document relevance.