Document Clustering via Cohesion Term Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document clustering methods using algorithms like k-means and Expectation Maximization often produce categories that are difficult for humans to interpret, as they rely on artificial feature spaces and do not effectively create understandable concepts from unstructured data.
Innovation Solution
A method that identifies cohesion terms in documents, calculates a cohesion score based on cosine differences, sorts terms by these scores, and creates categories using the most cohesive terms, refining them through a k-means algorithm to ensure categories are easily understandable and named with a single word or phrase.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If standard clustering algorithms (k-means, EM) are used to categorize documents, then automatic categorization is achieved, but the resulting categories are difficult for humans to interpret
Solution Approach 1:
The patent extracts and identifies cohesive terms from the document collection that naturally occur in the data. These terms are then used to define categories, separating the category definition process from standard clustering algorithms. By taking out the most representative terms and using them as category labels, the system achieves both automation and human interpretability.
Solution Approach 2:
The patent introduces cohesive terms as intermediaries between the documents and the categories. Instead of directly clustering documents into abstract categories, the system uses cohesive terms as mediators that bridge the gap between raw document data and meaningful categories, making the categories more interpretable while maintaining automated processing.
2Device complexity
If artificial feature spaces are used for clustering, then mathematical clustering can be performed, but the categories do not construct easily understood concepts
Solution Approach 1:
The patent segments the document collection into groups based on cohesive terms that naturally occur in the data. Instead of using a monolithic artificial feature space, the system segments the information into meaningful term-based categories, preserving conceptual understanding while enabling mathematical processing through cosine similarity calculations.
Solution Approach 2:
The patent changes the parameter space from abstract mathematical features to concrete cohesive terms that appear in the documents. By transforming the clustering parameters from artificial feature vectors to meaningful terms, the system maintains mathematical tractability while producing conceptually understandable categories.
3Productivity
If existing clustering techniques are applied to unstructured data, then document grouping is achieved, but cluster naming becomes problematic
Solution Approach 1:
The patent enables the data itself to name the categories through automated identification of cohesive terms. The system uses the documents' own language and structure to generate category names, eliminating the need for manual naming or complex post-processing. The cohesive terms self-service as both clustering units and category labels.
Data Source
AI summary
A method and a storage medium, that includes instructions for causing a computer to implement the method, for document categorization is presented. The method includes identifying terms occurring in a collection of documents, and determining a cohesion score for each of the terms. The cohesion score is a function of a cosine difference between each of the documents containing the term and a centroid of all the documents containing the term. The method further includes sorting the terms based on the cohesion scores. The method also includes creating categories based on the cohesion scores of the terms, wherein each of the categories includes only documents (i) containing a selected one of the terms and (ii) that have not already been assigned to a category. The method still further includes moving each of the documents to a category of a nearest centroid, thereby refining the categories.