Latent Semantic Density Analysis for Document Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document clustering methods, such as hierarchical and K-means clustering, face challenges in determining the number of clusters and are sensitive to initial conditions, leading to inconsistent results and inefficiencies, especially when dealing with large datasets and discrete data types.
Innovation Solution
The proposed solution involves generating a latent semantic mapping (LSM) space where document vectors are grouped based on a hypersphere diameter, allowing for the identification of clusters without pre-specifying the number of clusters, and using a density analysis to ensure cluster quality, with the ability to iteratively adjust the hypersphere diameter to refine cluster identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If hierarchical clustering methods are used, then the clustering process is simple to implement, but the decisions to join elements are based solely on local distance without considering the data as a whole, leading to potential mistakes in overall clustering
Solution Approach 1:
The patent segments the clustering process into two distinct phases: (1) generating candidate clusters using hierarchical clustering with single linkage, and (2) selecting final clusters using a global criterion that considers the entire dataset. This segmentation allows the simple hierarchical method to generate candidates while a more rigorous global evaluation ensures accuracy in the final selection.
Solution Approach 2:
The patent introduces an intermediary selection process that acts as a bridge between the simple hierarchical clustering and the final clustering result. This intermediary step uses a global criterion (maximizing the number of covered documents) to evaluate and select from the candidate clusters generated by hierarchical clustering, thereby improving reliability without sacrificing the simplicity of the initial clustering generation.
2Speed
If K-means clustering is used, then the algorithm converges quickly for good initial choices, but the number of clusters K must be specified prior to performing the algorithm, which is not known in advance for most document collections
Solution Approach 1:
The patent enables the clustering algorithm to determine the number of clusters automatically through a self-service mechanism. By using the global criterion to select clusters from candidates and stopping when no more significant clusters can be formed, the algorithm determines K itself rather than requiring it to be specified in advance, thus achieving adaptability to unknown cluster numbers.
Solution Approach 2:
The patent makes the number of clusters dynamic rather than static. Instead of fixing K beforehand, the algorithm dynamically determines the number of clusters based on the data characteristics and the global criterion, allowing the clustering structure to adapt to the inherent organization of the document collection.
3Shape
If SOM method is used, then representative points are placed in a prespecified geometrical configuration, but it requires the choice of an underlying geometry and assumes K is specified a priori
Solution Approach 1:
The patent extracts the geometric configuration requirement from the clustering process. By using hierarchical clustering with single linkage to generate candidates and a global criterion for selection, the method eliminates the need for prespecified geometrical configurations and the associated complexity of choosing underlying geometries, while still achieving meaningful cluster representations.
4Ease of operation
If traditional clustering techniques are used, then they operate on continuous data, but document clustering data is inherently discrete
Solution Approach 1:
The patent changes the fundamental parameter of data type handling by using single linkage hierarchical clustering, which is particularly suitable for discrete data. The method uses a global criterion based on document coverage counts rather than continuous distance metrics, making the clustering process inherently appropriate for discrete document data while maintaining operational simplicity.
Data Source
AI summary
According to one embodiment, a latent semantic mapping (LSM) space is generated from a collection of a plurality of documents, where the LSM space includes a plurality of document vectors, each representing one of the documents in the collection. For each of the document vectors considered as a centroid document vector, a group of document vectors is identified in the LSM space that are within a predetermined hypersphere diameter from the centroid document vector. As a result, multiple groups of document vectors are formed. The predetermined hypersphere diameter represents a predetermined closeness measure among the document vectors in the LSM space. Thereafter, a group from the plurality of groups is designated as a cluster of document vectors, where the designated group contains a maximum number of document vectors among the plurality of groups.