Document Clustering via Block Header Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document clustering methods fail to effectively utilize document structure for clustering OCR output, particularly in documents lacking hierarchical structural information, such as scanned paper documents, leading to impracticality in using structural information for clustering.
Innovation Solution
A method and system that encode documents by identifying text content blocks, extracting block headers, and building a condensed dictionary from training documents to generate encodings for test documents, allowing for similarity computation and clustering based on both content and structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If tree representation and edit distance methods are used to detect document similarities, then document structure can be taken into account, but these methods are not suited to documents which lack hierarchical structural information such as OCR output
Solution Approach 1:
The patent segments documents into text content blocks with block headers, creating a hierarchical structure from flat OCR output. This segmentation allows structure-based clustering while maintaining compatibility with documents that originally lacked hierarchical information.
Solution Approach 2:
The patent transforms one-dimensional text sequences into two-dimensional block structures by identifying block headers and organizing text content blocks. This dimensional transformation enables structure-based comparison methods to work effectively on OCR documents.
2Measurement precision
If hybrid clustering methods are used that take both content and structure into account, then clustering accuracy improves, but the methods are dedicated to specific types of structured format such as XML and HTML
Solution Approach 1:
The patent creates a universal block header extraction approach that works across different document types including OCR output, XML, and HTML. The method identifies block headers based on positional and structural characteristics rather than format-specific features, enabling broad applicability.
Solution Approach 2:
The patent changes the parameters used for structure detection from format-specific features (XML tags, HTML elements) to universal features (block position, header length, text density). This parameter transformation allows the same clustering algorithm to work across diverse document formats.
3Productivity
If limited structural information from OCR output is used for clustering, then processing speed is maintained, but the structural information varies considerably from page to page even when pages are from the same business workflow
Solution Approach 1:
The patent performs preliminary action by building a condensed dictionary of block headers from training documents before clustering. This pre-processing step captures regularities and trends in block header patterns, enabling consistent structure-based clustering without sacrificing processing speed.
Solution Approach 2:
The patent uses feedback from training documents to improve clustering performance. By analyzing block header patterns in training data and building a condensed dictionary, the system learns from variations in OCR output and adapts to capture consistent structural information across different pages.
Data Source
AI summary
A method for encoding documents includes building or otherwise providing a condensed dictionary including identifiers for block headers identified in text blocks extracted from a collection of training documents. For at least one test document a set of text content blocks is identified. For each of the text content blocks in the set, a block header is identified. Each block header in the training and test documents includes a sequence includes no more than a predetermined maximum number of characters. An encoding of the test document is generated, based on the identifiers of the block headers identified in the test document that are in the condensed dictionary.