Flattening Cluster Hierarchy for Document Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document clustering methods, such as agglomerative and partitional clustering, face challenges in efficiently managing large document repositories due to high computational complexity and time requirements, particularly in document review environments where thousands to millions of documents need to be analyzed quickly.
Innovation Solution
A hybrid approach that initially uses partitional clustering to create a cluster hierarchy and then selectively flattens branches for re-clustering using an agglomerative method, allowing for faster initial processing and improved cohesiveness of clusters where needed, significantly reducing overall processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If agglomerative clustering is used to cluster documents, then cluster cohesiveness is improved, but processing time increases significantly
Solution Approach 1:
The patent segments the document clustering process into two distinct phases: first applying partitional clustering to create an initial hierarchy, then selectively applying agglomerative clustering only to specific branches that need improvement. This segmentation allows the system to benefit from the speed of partitional clustering while applying the precision of agglomerative clustering only where necessary, thus resolving the contradiction between processing time and cluster cohesiveness.
Solution Approach 2:
The patent applies local quality by selectively flattening and re-clustering only certain branches of the cluster hierarchy that exhibit poor cohesiveness, rather than applying agglomerative clustering uniformly across all documents. This localized application of the more computationally intensive agglomerative method optimizes cluster quality in problem areas while maintaining overall processing efficiency.
2Productivity
If partitional clustering is used to create cluster hierarchy, then processing speed is improved, but cluster cohesiveness deteriorates
Solution Approach 1:
The patent applies preliminary action by first using partitional clustering to create an initial cluster hierarchy that provides a good starting point with reasonable speed. This preliminary structure is then refined by selectively applying agglomerative clustering to improve cohesiveness where needed, thus achieving both speed and quality through a two-stage approach.
Solution Approach 2:
The patent introduces dynamics by making the clustering approach adaptive: it starts with partitional clustering for speed, then dynamically identifies and applies agglomerative clustering to specific branches that require improvement. This dynamic, multi-phase approach allows the system to optimize both processing speed and cluster cohesiveness based on the specific needs of different document sets.
3Device complexity
If selective flattening of cluster hierarchy branches is performed, then computational complexity is reduced, but clustering accuracy may be affected
Solution Approach 1:
The patent applies local quality by selectively flattening only those branches of the cluster hierarchy that exhibit poor cohesiveness or meet specific criteria, rather than flattening the entire hierarchy. This selective approach reduces computational complexity while maintaining clustering accuracy in branches that do not require intervention.
Solution Approach 2:
The patent incorporates feedback mechanisms to evaluate cluster quality and determine which branches need flattening and re-clustering. By using feedback from cluster cohesiveness metrics, the system can intelligently identify problem areas and apply computational resources only where needed, thus maintaining accuracy while reducing overall complexity.
Data Source
AI summary
In an automatic electronic discovery search tool, documents can be clustered into a cluster hierarchy according to a first clustering approach. Once a hierarchy tree is created, portions of the tree can be flattened for application of a second superior clustering approach. Clustered portions may be used in a document review tool or further filtered according to specified criteria. Clusters may be filtered or used in a document review tool.