Suffix Tree Clustering with Term Weighting for Document Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document clustering algorithms, such as the Suffix Tree Clustering (STC) algorithm, suffer from generating large, poor-quality clusters due to the lack of effective quality measures for evaluating clusters and assessing the importance of phrases in a suffix tree model, leading to diminished confidence in clustering results.
Innovation Solution
The new suffix tree similarity measure (NSTM) incorporates term weighting schemes and sequence order of words to generate feature vectors for documents, allowing for improved clustering by using the Group-average Agglomerative Hierarchical Clustering (GAHC) algorithm, which maps nodes of a suffix tree into a multi-dimensional space and applies tf-idf weighting to assess node importance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Suffix Tree Clustering (STC) algorithm is used for document clustering, then the clustering process can be automated and efficiency is improved, but the quality of clusters deteriorates due to the generation of large, poor-quality clusters
Solution Approach 1:
The patent changes the parameters used in suffix tree clustering by introducing term weighting schemes (tf-idf) and sequence order considerations. This transforms the basic STC algorithm into a more sophisticated approach that evaluates phrase importance and positional relationships, thereby improving cluster quality while maintaining automated efficiency
Solution Approach 2:
The patent replaces the simple mechanical counting approach of traditional STC with a more refined system that incorporates statistical weighting (tf-idf) and sequence analysis. This substitution transforms the clustering mechanism from a crude frequency-based system to one that captures semantic importance and structural relationships
2Device complexity
If traditional vector document models are used for clustering, then the computational complexity is reduced, but the sequence order information of words is lost leading to diminished clustering accuracy
Solution Approach 1:
The patent segments the document into ordered sequences of words and phrases, maintaining the structural integrity of the text. By preserving sequence order information in the suffix tree representation, the method captures n-gram patterns and phrase structures that are critical for accurate clustering while managing computational complexity through efficient tree-based operations
3Manufacturing precision
If manual sorting is used for document classification, then the quality of information sorting is improved, but the workload becomes monumentally complex and difficult to achieve as the community grows
Solution Approach 1:
The patent implements self-service by enabling automated document clustering that evaluates and organizes information without human intervention. The system uses sophisticated algorithms incorporating term weighting, sequence order, and suffix tree structures to autonomously perform the sorting task that would otherwise require manual expertise, thereby maintaining high quality while eliminating the scaling problem of manual workload
Data Source
AI summary
The subject innovation provides for systems and methods to facilitate weighted suffix tree clustering. Conventional suffix tree cluster models can be augmented by incorporating quality measures to facilitate improved performance. Further the quality measure can be employed in determining cluster labels that show improvements in accuracy over conventional means. Additionally “stopnodes” can be defined to facilitate traversing suffix tree models efficiently. Quality measurements can be determined based in part on weighting factors applied to terms in a vector model, said terms being mapped from a suffix tree model.