Human Rater Pool for Social Network Content Relevance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Social networking systems face challenges in providing relevant content to users as machine-based ranking systems may overlook human-centric factors, leading to suboptimal content interaction and engagement.

Innovation Solution

A quality-controlled and representative pool of human raters is established to provide content ratings, with consistency and representativeness scores ensuring that the feed ranking model is improved to better align with user preferences, thereby enhancing content relevance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If machine-based ranking systems are used to determine content relevance, then automation and efficiency are improved, but human-centric factors and relevance accuracy deteriorate

Engineering Contradiction:

Improveautomation of content rankingVSAvoidcontent relevance accuracy

Core Design Contradiction:

Extent of automationVSMeasurement precision

Solution Approach 1:

Human raters are introduced as intermediaries between the machine-based ranking system and the final content presentation. These raters evaluate content items and provide feedback that is used to refine and adjust the automated ranking model, thereby incorporating human-centric factors while maintaining automation efficiency

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements a feedback loop where human rater evaluations are collected, analyzed, and used to continuously improve the machine-based ranking model. This feedback mechanism allows the system to learn from human judgments and adjust its algorithms to better align with human preferences and relevance criteria

Inventive Principle:

Principle #23Feedback

2Measurement precision

If a large pool of human raters is established to improve relevance accuracy, then measurement precision is improved, but system complexity and operational difficulty worsen

Engineering Contradiction:

Improvecontent relevance accuracyVSAvoidrater pool management complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system dynamically adjusts parameters such as the number of raters assigned to specific content items, the weighting of individual rater opinions, and the selection criteria for raters based on content characteristics. This allows flexible optimization of rater pool utilization without requiring manual management of every rater assignment

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The human rater pool serves multiple functions: evaluating content relevance, providing feedback for model improvement, and representing diverse user perspectives. The same rater pool is used across different content types and ranking scenarios, reducing the need for separate evaluation systems

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11580482B1Content relevance in a social networking system using population-representative human rater pool

Publication Date: 2023.02.14 META PLATFORMS INC

AI summary

A social networking system builds a quality controlled and desired population-representative pool of human raters to provide ratings on content items to improve a feed ranking model used for providing its users with more relevant content. The system identifies a pool of candidate human raters for providing ratings on a feed of content items. For each candidate human rater of the pool of candidate human raters, the system presents a feed of content items based on a feed ranking model, obtains ratings on the feed of content items, and determines a score representing the consistency of the obtained ratings, the representativeness of the pool of human raters, or the relevance of the content provided by the ranking model. The system uses the computed scores to modify the ranking model used to present content to its users for improving the relevance of the presented content.