Cross-phase parallelization for mass data modification tools
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mass data modification tools experience unnecessary delays and non-optimal runtimes due to sequential processing of phases, where each phase completes before moving on to the next.
Innovation Solution
A cross-phase parallelization and optimization platform that generates a worklist with metadata for process phases, identifies long running tasks, and executes predecessor tasks in parallel with next tasks in the same phase, allowing for smooth transitions between phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If each phase runs in parallel processes to complete operations quickly, then processing speed within each phase is improved, but the overall runtime increases because the tool must wait for all processes of a phase to complete before moving to the next phase
Solution Approach 1:
The system performs preliminary identification of long-running tasks in subsequent phases before the current phase completes. By preparing and staging these tasks ahead of time, the system can immediately launch them in parallel with the current phase's remaining work, eliminating the sequential wait time between phases while maintaining proper dependency tracking through the worklist metadata.
Solution Approach 2:
The system transitions from a single-dimensional sequential phase execution model to a multi-dimensional parallel execution model. By introducing cross-phase parallelism where tasks from different phases can execute simultaneously in different process threads, the system adds a temporal dimension to phase execution, allowing overlapping of phase boundaries while maintaining logical phase structure through the worklist.
2Reliability
If the tool waits for completion of all processes in a phase before moving to the next phase, then phase dependency requirements are satisfied, but unnecessary delays occur reducing productivity
Solution Approach 1:
The system performs preliminary identification and staging of long-running tasks in subsequent phases before the current phase completes. By preparing these tasks ahead of time with their dependency information stored in the worklist, the system can immediately launch them in parallel with the current phase's remaining work, eliminating the sequential wait time between phases while maintaining proper dependency tracking.
Solution Approach 2:
The system maintains continuous useful action by preventing idle periods between phases. While the current phase is still executing remaining short-running tasks, the system continuously executes long-running tasks from the next phase in parallel process threads. This eliminates gaps in productive work while maintaining phase dependency integrity through the worklist metadata that tracks predecessor-task relationships.
Data Source
AI summary
Arrangements for cross-phase parallelization and optimization are provided. A worklist including metadata associated with executing a plurality of process phases may be stored in a database table. An initial phase of the plurality of process phases may be selected. Using the worklist, a long running task associated with a next phase may be identified, and the long running task may include a predecessor task. Based on determining that a process is available in the initial phase, the available process in the initial phase may be used to execute the predecessor task required for the next phase. A next task in the initial phase may be selected. After executing the predecessor task, the long running task may be executed in parallel with an execution of the next task using a remaining available process in the initial phase. The cross-phase parallelization procedure may be repeated for the next phase.