Parallel Indexing Load-Balancing via Dynamic Work Item Creation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic document indexing systems face inefficiencies when archiving large volumes of unstructured data, as they often break down electronic items into separate components, leading to unpredictable processing loads and idle processors due to the complexity of embedded data structures, which slows down indexing processes.
Innovation Solution
The system employs multiple processors that can dynamically allocate processing power and create new work items as needed, allowing for internal load-balancing and efficient indexing of complex data structures, even when components are stored separately across multiple destinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple processors are used for indexing, then indexing speed is improved, but processor idle time increases due to unpredictable work item sizes
Solution Approach 1:
The system dynamically adjusts work item allocation based on actual processing needs. Secondary processors evaluate the complexity of embedded data structures during indexing and request additional work items or subdivide complex items, transforming the static workload distribution into a dynamic adaptation process that matches processor capacity to actual task complexity in real-time
Solution Approach 2:
Secondary processors autonomously determine whether to index data locally or create new work items based on their current workload and the complexity of embedded structures. Each processor self-manages its task queue and makes independent decisions about workload distribution without requiring centralized control, enabling automatic load balancing across the processor pool
2Quantity of substance
If electronic items are broken into separate components, then storage efficiency is improved, but indexing complexity increases due to unpredictable embedded data structures
Solution Approach 1:
The indexing system segments the indexing task into discrete work items that can be independently processed. Each work item corresponds to a specific electronic item or component, allowing processors to handle individual segments without being overwhelmed by the entire archive structure. This segmentation transforms the complex global indexing problem into manageable local tasks
Solution Approach 2:
The system handles nested embedded data structures by recursively creating work items for each level of nesting. When a processor encounters an embedded structure (e.g., an attachment within an email within an archive), it automatically generates nested work items that follow the hierarchical structure, allowing systematic processing of deeply nested components without losing track of the overall structure
3Ease of operation
If work items are pre-divided by a primary processor, then task distribution is simplified, but system responsiveness decreases due to inability to adapt to actual data complexity
Solution Approach 1:
The primary processor performs preliminary actions by creating initial work items from the archive structure and distributing them to secondary processors. This preliminary division provides a starting point for indexing while maintaining the flexibility for secondary processors to further subdivide work items or create additional tasks based on actual data complexity encountered during processing
Solution Approach 2:
The system implements feedback loops where secondary processors report back to the primary processor about the complexity of embedded structures they encounter. This feedback enables the primary processor to adjust future work item creation and distribution strategies, adapting to the actual characteristics of the archive being indexed rather than relying solely on pre-defined distribution patterns
Data Source
AI summary
The invention relates to electronic indexing, and more particularly, to the parallelization of indexing. Systems and methods of the invention index data archives by breaking a job into work items and sending the work items to multiple processors that can each determine whether to index data associated with the work item or to create a new work item and have a different processor index the data. This gives the system an internal load-balancing that results in indexing jobs during which no processor stands idle while another processor indexes data of unexpected complexity.