Hybrid Text Clustering via Decision-Driven Context Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text clustering algorithms fail to effectively handle mixed datasets containing both short and long text documents, as they are either unsuitable for numerical datasets or produce misleading results when dealing with non-linear data, and lack a solution for determining feature importance based on context patterns.
Innovation Solution
A Decision Driven Hybrid Text Clustering (DDHTC) method that computes a decision score using moment values and statistical coefficients to differentiate between short and long documents, sorting them based on context patterns, and then clustering each set separately using n-tuple word distributions and conditional probability equations to produce a final set of clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional clustering algorithms (K-Means, Fuzzy C-Means) are used on text documents, then numerical and structured datasets can be clustered effectively, but text documents with non-linear nature and mixed lengths cannot be handled properly
Solution Approach 1:
The patent replaces traditional mechanical clustering algorithms (K-Means, Fuzzy C-Means) with a probabilistic model-based approach. Instead of using distance-based mechanical clustering, the invention uses probability distributions to model document clusters, transforming the clustering mechanism from geometric to statistical, thereby achieving better results on non-linear text data with mixed lengths
Solution Approach 2:
The patent changes the parameter space by introducing probability distribution parameters (mean vectors, covariance matrices) to represent clusters. Instead of using fixed distance metrics, the system models each cluster as a probability distribution, allowing flexible handling of documents with varying lengths and non-linear structures through parameter optimization
2Adaptability or versatility
If topic modeling algorithms are used for short text documents, then short text clustering can be achieved, but long text documents cannot be handled effectively
Solution Approach 1:
The patent creates a universal clustering framework that handles both short and long text documents using the same probability distribution-based approach. The system is designed to be multi-functional, accommodating documents of any length by modeling clusters as probability distributions rather than relying on length-specific algorithms, thereby achieving versatility across different document types
3Device complexity
If documents are clustered without separating short and long texts, then the clustering process is simpler, but the clustering quality deteriorates due to heterogeneous data
Solution Approach 1:
The patent segments the document set into multiple clusters based on their probability distribution characteristics rather than document length. The segmentation is performed in the probability space where documents of different lengths can be properly separated into distinct clusters, improving clustering quality without requiring pre-separation by length
4Productivity
If feature vectors are created without context-based feature determination, then the processing is faster, but the clustering accuracy decreases due to lack of contextual understanding
Solution Approach 1:
The patent performs preliminary action by pre-processing text documents to extract meaningful features and representing them as probability distributions before clustering. This preliminary transformation of text to probability space captures contextual information in advance, enabling accurate clustering without requiring complex contextual analysis during the actual clustering process, thus maintaining both speed and accuracy
Data Source
AI summary
The present invention discloses a method and a system for clustering of short and long text documents. The documents are input through an input module and a pre-processing module overtakes the documents from the input module. The pre-processing module refines the documents and removes unwanted text from the documents. Then a decision driven hybrid text clustering algorithm is applied via different modules to achieve clustering of the documents. Firstly, a context module computes a moment value of a feature signifying at least one feature importance value of the feature for the documents. The moment value is used by a decision module to calculate a decision score. Based on the decision score the documents are split into two sets. A clustering module then forms clusters of the two sets of documents based on n-tuple word distribution. Finally, a convergence module congregates the clusters in a final set of documents.