Parallel Document Indexing via Segmented Work Items
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of electronic discovery cases poses a challenge due to the limitations of single-machine indexing capabilities, which struggle to efficiently process large volumes of unstructured and hierarchical electronic data.
Innovation Solution
A data processing system comprising a primary processor, secondary processors, and a messaging module that parallelizes document indexing by creating work items, processing embedded data based on characteristics, and exchanging status messages to distribute and manage data processing across multiple cores, enabling load-balancing and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If document indexing is performed on a single machine, then the system structure is simple, but the processing speed and efficiency are limited
Solution Approach 1:
The patent divides the document indexing system into multiple independent processors (primary processor and secondary processors), each capable of processing different portions of the data set simultaneously. The data set is segmented into multiple data portions that can be processed in parallel, thereby increasing overall indexing speed while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent transitions from single-machine sequential processing to multi-processor parallel processing by adding the dimension of concurrent execution. Multiple processors operate simultaneously on different data portions, transforming the indexing process from a one-dimensional sequential operation to a multi-dimensional parallel operation, thus significantly improving productivity
2Productivity
If multiple processors are used for parallel indexing, then processing efficiency increases, but system complexity increases
Solution Approach 1:
The patent introduces a primary processor that acts as an intermediary between the data source and secondary processors. The primary processor receives the complete data set, segments it into manageable portions, and distributes these portions to secondary processors. This intermediary role simplifies coordination by centralizing the segmentation and distribution logic, thereby managing system complexity while enabling parallel processing
Solution Approach 2:
The patent implements feedback mechanisms where secondary processors report their processing status and results back to the primary processor. This feedback loop enables the primary processor to monitor progress, manage load balancing, and coordinate the aggregation of results from multiple secondary processors, thus managing system complexity through structured communication protocols
3Loss of time
If a single processor handles all data, then data consistency is maintained, but processing time increases
Solution Approach 1:
The patent segments the large data set into multiple smaller data portions that can be processed simultaneously by different secondary processors. Each processor handles its assigned portion independently, which reduces the time each processor needs to operate while maintaining data consistency through the coordinated management of the primary processor that ensures all segments are processed uniformly
Solution Approach 2:
The patent combines the results from multiple secondary processors back together through the primary processor. After secondary processors complete their independent processing of data portions, the primary processor merges these results into a unified index structure, thereby maintaining data consistency across all processed segments while having benefited from the reduced processing time of parallel execution
Data Source
AI summary
A system and method for parallelizing document indexing in a data processing system. The data processing system includes a primary processor for receiving a list of data having embedded data associated therewith, at lest one secondary processor to process the data as provided by the primary processor, a data processor to determine a characteristic of the embedded data and process the embedded data based upon the characteristic, and a messaging module to exchange at least one status message between the primary processor and the at least one secondary processor.