Deduplicated Data Block Sorting for Unstructured Document Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing unstructured document processing techniques are inefficient due to document-based text analytics, which require separate processing iterations for each document, leading to increased resource consumption and reduced efficiency in analyzing large volumes of data.
Innovation Solution
The proposed method involves identifying deduplicated data blocks, sorting them by frequency metrics, and applying text analytics to the highest frequency blocks, allowing results to be applied across multiple documents in a single iteration, thereby accelerating document processing and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If document-based text analytics are used to process each document separately, then processing completeness is maintained, but resource consumption increases and processing efficiency decreases
Solution Approach 1:
The patent segments documents into deduplicated data blocks, sorting them by frequency metrics. High-frequency blocks are processed first through text analytics, and results are applied to multiple documents simultaneously. This segmentation approach maintains processing completeness while reducing redundant operations across documents.
Solution Approach 2:
The patent performs preliminary deduplication and frequency analysis of data blocks before applying text analytics. By identifying and prioritizing high-frequency blocks in advance, the system prepares processing sequences that maximize result reusability across multiple documents, thereby reducing overall resource consumption.
2Measurement precision
If separate processing iterations are performed for each document, then individual document accuracy is maintained, but processing time increases
Solution Approach 1:
The patent merges the processing of multiple documents by applying text analytics results from high-frequency data blocks to all documents containing those blocks. This combining approach maintains analytical accuracy for each document while eliminating redundant processing iterations, significantly reducing total processing time.
Solution Approach 2:
The patent discards redundant processing of identical data blocks across multiple documents and recovers computational efficiency by reusing text analytics results. High-frequency blocks are processed once and their results are recovered and applied to all relevant documents, maintaining accuracy while reducing time loss.
3Reliability
If all data blocks are processed with equal priority, then comprehensive analysis is achieved, but resource allocation becomes inefficient
Solution Approach 1:
The patent applies local quality by differentiating the processing priority of data blocks based on their frequency metrics. High-frequency blocks receive priority processing and their results are reused across multiple documents, while low-frequency blocks are processed with lower priority. This differentiated approach maintains comprehensive analysis reliability while significantly improving resource utilization efficiency.
Solution Approach 2:
The patent changes the parameter of processing priority from uniform to frequency-based differentiation. By sorting data blocks according to frequency metrics and processing high-frequency blocks first, the system optimizes resource allocation while ensuring comprehensive analysis coverage through iterative processing of remaining blocks.
Data Source
AI summary
Techniques are described relating to unstructured document processing. An associated computer-implemented method includes identifying a plurality of deduplicated data blocks associated with a collection of unstructured documents. The method further includes sorting the plurality of deduplicated data blocks in descending order based upon at least one block frequency metric, selecting a highest sorted unprocessed deduplicated data block, applying text analytics to the selected deduplicated data block, and applying at least one result of the text analytics to any document among the collection of unstructured documents including the selected deduplicated data block. The method is terminated responsive to satisfaction of at least one stopping condition.