Forward-Reverse Similarity Clustering for Sparse Short Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document clustering methods struggle with short text documents due to data sparsity, leading to unbalanced and ineffective clustering results.
Innovation Solution
A short text similarity based clustering system that receives short text documents, preprocesses the data using natural language processing techniques, determines forward and reverse similarity values, generates a similarity matrix, and clusters the documents based on this matrix.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional document clustering methods are used on short text documents, then the clustering process can be performed, but the results are unbalanced and ineffective due to data sparsity
Solution Approach 1:
The patent transforms the clustering approach by introducing a new dimension of analysis through forward and reverse similarity calculations. Instead of relying solely on traditional text feature similarity, the system creates a bidirectional similarity matrix that captures contextual relationships, effectively adding a new dimensional perspective to the clustering problem that works well with short text data
Solution Approach 2:
The patent changes the parameters used for similarity measurement by implementing forward similarity (how much document A contains information from document B) and reverse similarity (how much document B contains information from document A). This parameter transformation allows the system to effectively measure similarity even when traditional text features are sparse
2Stability of the object's composition
If short text documents are clustered using traditional methods, then clustering can be performed, but the clusters are unbalanced
Solution Approach 1:
The patent implements a feedback mechanism through iterative clustering where the system calculates forward and reverse similarities, performs clustering, evaluates cluster quality, and refines the clustering in subsequent iterations. This feedback loop allows the system to progressively improve cluster balance by adjusting document assignments based on bidirectional similarity measurements
Data Source
AI summary
Methods and systems for receiving a plurality of documents including short text data and determining a plurality of forward similarity values based on the short text data in each of the plurality of documents, determining a plurality of reverse similarity values based on the short text data in each of the plurality of documents, generating a forward and reverse similarity matrix based on the plurality of forward similarity values and the plurality of reverse similarity values, and generating a plurality of short text similarity based clusters to group the short text data of the plurality of documents based on the forward and reverse similarity matrix.