Graph Taxonomy Extraction with Adaptive Pruning for Fast Document Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document classification systems face challenges in efficiently processing and labeling large volumes of documents due to high computational complexity and resource requirements, making real-time processing and labeling of millions of documents impractical, especially in cloud-hosted applications.
Innovation Solution
Implementing unsupervised hierarchical clustering on graphs to extract taxonomies from document corpora, using similarity and co-occurrence matrices, and applying community detection algorithms with adaptive pruning to ensure modularity thresholds, reducing computational complexity and enabling rapid labeling and classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional document classification methods are used, then classification accuracy can be maintained, but processing time and computational complexity increase significantly
Solution Approach 1:
The patent segments the document classification task into multiple stages: (1) building a taxonomy hierarchy through hierarchical clustering of documents, (2) extracting features and computing similarity matrices, (3) classifying new documents against the pre-built taxonomy. This segmentation allows the system to process documents rapidly by reusing the pre-computed taxonomy structure rather than performing complex analysis on each document individually.
Solution Approach 2:
The patent performs preliminary actions by pre-building the complete taxonomy hierarchy and computing similarity matrices for the training corpus before classifying new documents. The hierarchical clustering and feature extraction are completed in advance, creating a ready-to-use classification structure that can quickly assign labels to new documents without repeating the complex computational processes.
2Measurement precision
If hierarchical clustering algorithms are applied to large document corpora, then taxonomy extraction quality improves, but processing time increases
Solution Approach 1:
The patent divides the large document corpus into manageable clusters through hierarchical clustering, organizing documents into a tree-like taxonomy structure with multiple levels. This segmentation allows the system to process documents in batches and maintain high extraction quality while reducing overall processing time through structured organization.
Solution Approach 2:
The patent implements dynamic pruning of the taxonomy hierarchy at different levels based on criteria such as minimum cluster size and similarity thresholds. This dynamic adjustment allows the system to adapt the granularity of the taxonomy to the specific document corpus, extracting high-quality taxonomies while avoiding unnecessary processing of overly detailed or irrelevant cluster levels.
Data Source
AI summary
Aspects of the present disclosure provide systems, methods, apparatus, and computer-readable storage media for extracting taxonomies based on hierarchical clustering on graphs related to a corpus of documents and using said taxonomies for classifying and labeling documents. The hierarchical clustering of graphs include the adaptive pruning of nodes at each hierarchy based on betweenness centrality of nodes to form clusters that have modularity score exceeding a minimum modularity threshold.