Text Mining Semantic Relatedness Social Media Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems are inadequate in handling the complexities of 'big data' from unstructured sources like social media, particularly in identifying and filtering out off-topic postings, due to their reliance on structured data and inability to process unordered, untagged, or very short social media content efficiently.
Innovation Solution
A computer-implemented method using a text mining server that determines semantic relatedness by establishing connections with social networking platforms, extracting non-redundant concepts from reference and test documents, computing co-occurrence frequencies and relatedness scores, and aggregating these to assess document similarity, thereby filtering out off-topic content without manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data processing systems use structured data processing and syntax parsing to analyze social media content, then measurement precision of semantic meaning is improved, but device complexity and processing time increase significantly
Solution Approach 1:
The patent extracts only the essential co-occurrence frequency information from social media postings, bypassing complex syntax parsing and semantic role labeling. By extracting only the frequency with which words appear together in the same posting, the system achieves semantic analysis without the computational burden of conventional NLP methods.
Solution Approach 2:
The patent creates a simplified representation of semantic relationships by copying and counting word co-occurrences directly from the raw text data. Instead of copying complex parsed syntax trees or semantic structures, the system directly copies word pairs and their frequencies, maintaining semantic information while dramatically reducing processing complexity.
2Measurement precision
If conventional systems apply semantic role labeling and syntax parsing to social media postings, then semantic analysis accuracy is improved, but processing speed and productivity deteriorate
Solution Approach 1:
The patent extracts only the necessary co-occurrence frequency data from social media postings, eliminating the need for time-consuming syntax parsing and semantic role labeling. This extraction approach maintains sufficient semantic accuracy for identifying off-topic postings while dramatically improving processing speed.
Solution Approach 2:
The patent applies partial action by performing only the essential co-occurrence counting needed for semantic analysis, without applying the full suite of conventional NLP techniques. This partial approach achieves the necessary analytical depth for filtering off-topic content while avoiding the productivity loss associated with complete syntax parsing and semantic role labeling.
3Quantity of substance
If conventional data processing methods are used on large volumes of unstructured social media data, then comprehensive analysis coverage is improved, but CPU consumption and energy usage increase
Solution Approach 1:
The patent extracts only the essential co-occurrence frequency information from large volumes of unstructured social media data, bypassing the CPU-intensive processes of syntax parsing and semantic role labeling. This extraction method enables comprehensive processing of big data volumes while maintaining low CPU consumption.
Solution Approach 2:
The patent uses a simple, computationally inexpensive co-occurrence counting method that can be rapidly applied to large datasets. This disposable-like approach processes each posting independently through a lightweight algorithm, enabling high-volume processing without the sustained high CPU usage required by complex semantic analysis methods.
4Extent of automation
If automated text mining is applied to identify off-topic postings, then ease of operation and automation level are improved, but device complexity increases
Solution Approach 1:
The patent extracts only the essential co-occurrence frequency metrics needed for automated off-topic detection, avoiding the complexity of full semantic parsing systems. This extraction-based approach achieves high automation levels while keeping the system architecture simple and manageable.
Data Source
AI summary
Described herein is an approach for automatically determining the semantic relatedness of documents to semantic concepts. A first text mining analysis extracts a set of reference concepts from reference documents. A second text mining analysis extracts a set of test concepts from test documents that include a mixture of new concepts and reference concepts. An extended co-occurrence matrix is computed that indicates a frequency of co-occurrence (RCCF) of each new and each reference concept in the test documents with all other new and reference concepts. The extended co-occurrence matrix is used for computing a new concept relatedness score (NCRS) for the new concepts. A document similarity score (DSS) is computed for each of the test documents by aggregating, inter alia, the NCRS of each new concept with the RCCF of each reference concept. The DSS represents the semantic relatedness of the test document to the totality of the reference concepts.