AI Sentiment Detection for Offensive Content Filtering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing recommendation systems fail to effectively detect and filter out offensive content due to outdated manual lists, reliance on keywords, and inability to adapt to changing user sentiments, leading to negative user experiences.

Innovation Solution

Implementing artificial intelligence and machine learning techniques to identify negative user sentiment cues, build models that predict offensive content, and filter out recommendations based on user-specific and segment-specific data, using keyword-based and event analysis models to inhibit potentially offensive items.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual lists of known offensive keywords are used to filter recommendations, then the system can block some offensive content, but the lists become outdated as user perceptions change and fail to detect new offensive patterns

Engineering Contradiction:

Improveoffensive content detection accuracyVSAvoidadaptation to changing user sentiments

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system implements feedback loops where user interactions with recommendations (such as reporting offensive content, skipping recommendations, or expressing negative sentiment) are continuously collected and used to retrain and update the machine learning models. This allows the system to adapt to changing user perceptions and identify new offensive patterns without relying on static manual lists.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent replaces the mechanical system of manual keyword list creation and maintenance with an automated machine learning-based detection system. The AI model analyzes user behavior patterns, sentiment data, and interaction metrics to dynamically identify offensive content, eliminating the need for periodic manual updates of keyword lists.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If keyword-based filtering is used to identify offensive content, then the system can quickly block known offensive terms, but it relies solely on keywords and misses contextual nuances and intentional misspellings

Engineering Contradiction:

Improveoffensive content filtering speedVSAvoidoffensive content detection precision

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system introduces machine learning models as an intermediary layer between the recommendation generation and the user. These models analyze not only keywords but also contextual information, user behavior patterns, and sentiment data to make more accurate determinations about offensive content, thereby improving detection precision while maintaining filtering speed.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent combines multiple detection approaches into a composite system that integrates keyword-based filtering with machine learning analysis of user behavior, sentiment data, and contextual information. This multi-component approach leverages the speed of keyword matching while adding the precision of AI-based pattern recognition to detect intentional misspellings and contextual nuances.

Inventive Principle:

Principle #40Composite materials

3Ease of operation

If recommendation services are implemented to provide personalized recommendations, then user experience is improved through relevant content, but users may receive offensive or obnoxious recommendations that cause them to quit the service

Engineering Contradiction:

Improveuser experience qualityVSAvoidnegative user sentiment from offensive content

Core Design Contradiction:

Ease of operationVSObject-affected harmful factors

Solution Approach 1:

The system applies preliminary anti-action by proactively identifying and blocking offensive recommendations before they reach the user. The machine learning models predict which recommendations might generate negative sentiment based on user profiles, historical data, and content analysis, preventing offensive content from being presented in the first place.

Inventive Principle:

Principle #9Preliminary anti-action

Solution Approach 2:

The system continuously monitors user responses to recommendations and uses this feedback to refine its filtering capabilities. When users express negative sentiment or report offensive content, the system learns from these interactions to improve future filtering, creating a self-improving mechanism that protects user experience while maintaining personalized recommendation quality.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10410125B1Artificial intelligence based identification of negative user sentiment in event data

Publication Date: 2019.09.10 AMAZON TECH INC

AI summary

A recommendation system uses artificial intelligence to identify, based on negative sentiment cues from users, item attributes, such as keywords, that users may find offensive or undesirable. The negative sentiment cues may be explicit (e.g., a user selects an option not to view a particular recommendation again), implicit (e.g., a user does not interact with recommendations relating to an attribute), or both. The system may use a computer model generated based on these identified attributes to filter or modify recommendations to a user or group of users. For instance, if a particular keyword is identified as highly offensive to a group of users, items associated with the keyword may be filtered from item recommendations presented to the group of users. If an attribute is identified as moderately offensive to a user, items associated with the attribute may be down-weighted in item recommendations presented to the user.