Biased Content Sampling for Quality Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online systems face challenges in efficiently identifying and evaluating low-quality content due to class imbalance, where most user-provided content complies with policies, making it difficult to sample sufficient low-quality content for effective review process evaluation.
Innovation Solution
The online system employs a biased sampling process using risk scores and impression counts to increase the proportion of low-quality content in the sampled subset, allowing for more accurate quality performance metrics and reducing human review time by prioritizing content with higher impact on user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If random sampling is used to evaluate the review process, then the sampling process is simple and fast, but the proportion of low-quality content in the sampled subset is too low to provide accurate quality performance metrics
Solution Approach 1:
The patent changes the sampling parameter from uniform random sampling to biased sampling based on risk scores. Each content item is assigned a sampling probability proportional to its risk score, which transforms the sampling distribution to enrich low-quality content in the evaluated subset while maintaining computational efficiency.
Solution Approach 2:
The patent segments the content population into different risk categories using the machine learning model's risk scores. By dividing content into high-risk, medium-risk, and low-risk segments, the system can apply differential sampling probabilities to each segment, ensuring adequate representation of low-quality content without requiring evaluation of all content items.
2Measurement precision
If a large number of content items are sampled to identify sufficient low-quality content, then the accuracy of quality performance metrics improves, but the human review time and computational resources increase significantly
Solution Approach 1:
The patent changes the sampling parameter from uniform probability to risk-score-proportional probability. This transformation allows the system to achieve the same measurement precision with a much smaller sample size, because the sampling distribution is optimized to concentrate on high-risk content items that are more likely to be low-quality.
Solution Approach 2:
The patent applies partial action by sampling only a subset of content items with probability proportional to their risk scores, rather than evaluating all content items. This partial sampling approach achieves sufficient accuracy for quality performance metrics while significantly reducing human review time and computational resources.
3Measurement precision
If the review process evaluates all user-provided content, then the quality performance metrics would be most accurate, but the computational cost and human review time become prohibitively high
Solution Approach 1:
The patent changes the evaluation parameter from comprehensive evaluation of all content to risk-score-proportional sampling. This parameter transformation enables the system to achieve adequate measurement precision with a fraction of the computational resources required for full evaluation.
Solution Approach 2:
The patent implements partial evaluation by selecting a representative subset of content items for human review based on risk scores. This partial action approach provides sufficient accuracy for quality performance metrics while dramatically reducing computational cost and human review time compared to evaluating all content.
Data Source
AI summary
An online system obtains risk scores determined by a machine learning model for a content item provided by a user of an online system for display to users of the online system, where the risk scores indicate the likelihood of content items violating a content policy. The online system uses the risk scores to determine sampling weights used to select content items for inclusion in a sampled subset of content items. The sampling weights are determined from risk score counts indicating the relative frequency of the obtained risk scores and impression counts indicating the number of times content items have been presented to the users of the online system. The online system presents the selected content items for evaluation by a human reviewer using a quality review interface. Using the results of the quality review, the online system determines quality performance metrics of the machine learning model.