Document Clustering via Topic Purity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing large volumes of electronic documents due to the inefficiency of existing clustering algorithms, which often result in poorly formed clusters and require time-consuming manual labeling.
Innovation Solution
A computer-implemented method that calculates a topic purity score for each cluster, combines clusters with low scores into new clusters, and assigns labels based on a higher-level parent category, automating the clustering and labeling process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If known clustering algorithms are used to organize documents, then documents can be grouped into clusters, but the clusters are poorly formed and contain many unrelated documents
Solution Approach 1:
The patent segments the clustering process into multiple hierarchical levels. Instead of creating one large cluster at a time, the system divides documents into smaller sub-clusters at lower hierarchical levels, then progressively merges these sub-clusters into larger clusters at higher levels. This segmentation allows for better quality control at each level while reducing the computational burden of processing all documents simultaneously.
Solution Approach 2:
The patent introduces a hierarchical dimension to the clustering process. Clusters are organized into multiple levels where lower-level clusters are composed of even lower-level sub-clusters. This dimensional transformation from flat clustering to hierarchical clustering enables the system to manage complexity better and improve cluster quality without proportionally increasing implementation time.
2Measurement precision
If manual inspection is used to label clusters, then accurate labels can be assigned to clusters, but the process is very time consuming and expensive
Solution Approach 1:
The patent implements self-service labeling through automated topic purity calculation. Instead of requiring manual inspection to determine cluster labels, the system automatically calculates topic purity scores by analyzing the semantic content of documents within each cluster. This self-service approach maintains high labeling accuracy while eliminating the time and cost associated with manual human inspection.
Solution Approach 2:
The patent replaces the mechanical process of manual human inspection with an automated computational system. The topic purity calculation mechanism uses natural language processing and semantic analysis algorithms to automatically evaluate and label clusters, substituting human labor with machine-based measurement that achieves comparable or superior accuracy without the associated time and cost constraints.
3Productivity
If clusters are created without validation, then the clustering process is faster, but the clusters lack topic purity and contain unrelated documents
Solution Approach 1:
The patent applies preliminary validation actions during the clustering process itself. Before finalizing cluster assignments, the system calculates topic purity scores as an intermediate step and uses these scores to validate whether documents should remain in their current clusters or be redistributed. This preliminary validation ensures topic purity is maintained without requiring a separate post-processing validation stage that would slow down the overall process.
Data Source
AI summary
A computer-implemented method executes instructions stored on a computer-readable medium. The method includes accessing a hierarchy of clusters, wherein each cluster includes at least one content file, and a label is associated with each cluster. The method further includes calculating a topic purity score for each cluster, and selecting a first cluster and a second cluster from the hierarchy of clusters, wherein the topic purity score of the first cluster and the second cluster are less than a purity threshold. The method also includes creating a third cluster by combining the content files included within the first cluster and the second cluster, determining a parent category of the first cluster and the second cluster, wherein the parent category is at a level within the hierarchy higher than a level of the first cluster and the second cluster, and associating a label of the parent category with the third cluster.