Bias Document Detection Using Low-Prevalence Sentiment Concepts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional natural language processing systems struggle to identify bias concepts or documents that are not frequently mentioned but are important for understanding sentiment, often treating them as less relevant.
Innovation Solution
A method and system that utilize Natural Language Processing techniques to identify prevalent concepts in documents, categorize them as high or low prevalence, and specifically detect low prevalence concepts with negative sentiment by applying a pre-trained model to determine bias concepts or documents, using a concept window for sentiment analysis and comparing with bias keywords.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional NLP techniques use bag of words representation and frequency-based relevance, then common concepts are easily identified, but low prevalence bias concepts are missed or treated as irrelevant
Solution Approach 1:
The patent segments concepts into high prevalence and low prevalence categories based on frequency thresholds. This segmentation allows the system to apply different processing strategies: conventional frequency-based methods for high prevalence concepts and specialized sentiment analysis with concept windows for low prevalence concepts, thereby preventing loss of bias-related information while maintaining efficiency for common terms.
Solution Approach 2:
The patent applies local quality by using concept windows of specific sizes (e.g., 5-10 terms) around low prevalence concepts to capture local contextual sentiment. This localized approach enables precise sentiment analysis for rare terms without requiring global reprocessing of entire documents, thus preserving bias concept information while maintaining computational efficiency.
2Productivity
If the system focuses on frequently mentioned concepts, then processing efficiency is high, but bias concepts with low frequency are overlooked
Solution Approach 1:
The patent segments the document processing task into two streams: rapid frequency-based filtering for high prevalence concepts and targeted sentiment analysis for low prevalence concepts. This segmentation maintains high overall processing speed while ensuring reliable bias identification through specialized handling of rare but potentially biased terms.
Solution Approach 2:
The patent applies partial action by focusing computational resources on analyzing low prevalence concepts only when they meet specific sentiment criteria. Instead of analyzing all concepts equally, the system performs detailed sentiment analysis only on low prevalence concepts that show potential bias indicators, thus maintaining productivity while improving bias identification reliability.
3Device complexity
If conventional techniques treat all concepts equally by frequency, then simple classification is achieved, but nuanced sentiment analysis of rare concepts is lost
Solution Approach 1:
The patent segments sentiment analysis into two levels: simple frequency-based sentiment for high prevalence concepts and nuanced context-aware sentiment analysis using concept windows for low prevalence concepts. This segmented approach achieves precise sentiment detection for rare terms while maintaining manageable overall system complexity through selective application of advanced methods.
Data Source
AI summary
Present disclosure describes techniques for determining document bias in a document set. The techniques include the step of receiving a document set comprising a plurality of documents, applying a Natural Language Processing (NLP) technique to the document set to identify one or more prevalent concepts in each document of the plurality of documents, and categorizing one or more concepts as a high prevalence concept and a low prevalence concept based on one or more parameters. The techniques further include the step of identifying one or more low prevalence concepts with negative sentiment and comparing the identified concepts with a plurality of keyword to determine at least one bias document/concept in the document set.