Two-Level Document Clustering Algorithm for Search Result Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for clustering web-related documents often produce clusters that include too many documents or duplicates, leading to over-clustering and duplication issues.
Innovation Solution
A two-level clustering algorithm that combines first-level clusters based on domain matching and classification criteria, preventing the inclusion of multiple clusters from the same domain in a second-level cluster, thereby avoiding over-clustering and duplicate content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional clustering methods are used, then documents can be grouped together, but the clusters include too many documents or duplicated content
Solution Approach 1:
The patent applies segmentation by dividing the clustering process into two distinct levels: first-level clustering groups documents by domain, while second-level clustering combines these domain-level clusters based on classification criteria. This segmentation prevents any single cluster from containing documents from multiple domains, thereby eliminating duplicate content and reducing the number of documents per cluster while improving overall cluster quality.
2Ease of manufacture
If documents from the same domain are clustered together, then clustering is simplified, but duplicate content and over-clustering occur
Solution Approach 1:
The patent introduces a new dimension to the clustering process by adding domain-level clustering as a preliminary step before content-based clustering. This dimensional change allows the system to first organize documents by domain (first-level clustering) and then combine these organized groups based on classification criteria (second-level clustering), thereby preventing duplicate content while maintaining clustering simplicity.
3Productivity
If multiple first-level clusters from the same domain are combined, then more documents are processed, but over-clustering and duplication issues arise
Solution Approach 1:
The patent segments the clustering process into two levels where first-level clustering organizes documents by domain and second-level clustering combines these segmented groups. This segmentation ensures that even when processing large volumes of documents from multiple domains, the system maintains cluster accuracy by preventing combination of clusters from the same domain, thereby eliminating over-clustering and duplication issues.
Data Source
AI summary
A computer implemented method for clustering documents that includes receiving at least a first first-level cluster of documents and a second first-level cluster of documents, wherein the first first-level cluster includes one or more first documents associated with a first domain, the first documents satisfying a first first-level classification criterion, and the second first-level cluster including one or more second documents associated with a second domain, the second documents satisfying a second first-level classification criterion. The method also includes creating, by a processor of the computer system, a second-level cluster by combining the first first-level cluster with the second first-level cluster when a) the first first-level cluster and second first-level cluster satisfy a second-level classification criterion; and b) the first domain does not match the second domain.