Rating Normalization for Online Content Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
User-specific biases in rating content items on online systems lead to inaccurate quality scores, causing content items to be either under- or over-presented to users, which can result in reduced interaction with interested users and increased interaction with uninterested users.
Innovation Solution
The online system generates a distribution of ratings from users within a specific time interval and normalizes ratings for individual users to mitigate biases, using the normalized ratings to determine quality scores for content items, ensuring more accurate content selection and presentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If user-specific ratings are collected from a group of users, then the quantity of ratings data increases, but user-specific biases cause measurement precision to deteriorate
Solution Approach 1:
The patent introduces an intermediary normalization process that mediates between raw user ratings and final quality scores. A normalization module transforms individual user ratings by comparing them against a distribution of ratings from the entire user group, effectively filtering out user-specific biases while preserving genuine content quality signals. This intermediary step allows the system to utilize data from many users without suffering from their individual biases.
2Productivity
If content items are selected based on ratings from biased users, then content selection speed increases, but reliability of content selection deteriorates
Solution Approach 1:
The patent applies preliminary normalization to ratings before they are used for content selection. By pre-processing ratings to remove bias effects beforehand, the system ensures that subsequent content selection operations use already-corrected data. This preliminary action maintains fast selection speeds while improving reliability, as the normalization is performed once rather than repeatedly during selection operations.
3Measurement precision
If normalized ratings are generated to offset user bias, then measurement precision of quality scores improves, but device complexity increases
Solution Approach 1:
The patent changes the parameter representation of ratings by transforming them into a normalized scale based on their position within the rating distribution. Instead of using raw rating values directly, the system converts them to normalized scores that reflect their relative standing among all user ratings. This parameter transformation simplifies the processing by converting a complex bias problem into a straightforward statistical normalization operation.
Data Source
AI summary
An online system generates a model for determining quality scores for content items by presenting content items to a group of users who provide ratings for the content items. Based on ratings received from various users of the group, the online system generates the model. To mitigate bias from ratings received from different users of the group, the online system generates a distribution of ratings received from users of the group within a time interval and generates a cumulative distribution function by integrating the distribution. For a user of the group, the online system generates a user distribution of rankings received from the user. Additionally, for a rating received from the user, the online system determines a percentage of ratings received from the user less than the rating. Based on the cumulative distribution function and the percentage, the online system generates a normalized rating corresponding to the rating from user.