Workflow Engine Parallel Execution Checkpoint Merge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workflow platforms face challenges in managing synchronous workflows, particularly in distributed computing environments, where determining if a client has timed out or is waiting for results can lead to inconsistent and inefficient experiences due to the need for locks and potential network issues.
Innovation Solution
The workflow platform enables parallel execution of workflow steps without requiring locks, using checkpoint phases and a consensus algorithm to merge updates based on idempotency keys and record version numbers, ensuring consistent and efficient processing even in cases of network disruptions or retries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a lock is acquired by a first client to execute workflow steps, then workflow synchronization is ensured, but another client is blocked for an undefined duration causing inefficiency
Solution Approach 1:
The patent extracts the locking mechanism from the workflow execution process. Instead of using locks to ensure synchronization, the system uses a checkpoint-based approach where each client executes workflow steps independently and updates are merged at checkpoint phases. This removes the blocking mechanism that caused inefficiency while maintaining synchronization through the checkpoint merge process.
Solution Approach 2:
The patent implements preliminary actions by having clients register their intent to execute workflow steps before actually executing them. The system tracks these registrations and uses them during checkpoint phases to determine which updates need to be merged. This preliminary registration allows the system to coordinate updates without blocking clients during execution.
2Reliability
If a lock is held to prevent concurrent workflow execution, then data consistency is maintained, but the system cannot determine if a client has timed out or is still waiting
Solution Approach 1:
The patent introduces checkpoint phases as intermediaries between workflow step executions. Instead of directly managing client state and timeout detection, the system uses checkpoint phases as intermediary events where all clients synchronize and their updates are merged. This intermediary mechanism simplifies state tracking because the system only needs to track which clients have registered updates by checkpoint phase, not their continuous execution state.
Solution Approach 2:
The patent implements periodic checkpoint phases at regular intervals in the workflow execution process. These periodic checkpoints provide natural synchronization points where the system can assess which clients have completed workflow steps and merge their updates. This periodic structure makes state tracking simpler because the system only needs to track client status relative to the nearest checkpoint phase.
3Productivity
If parallel execution is enabled without locks, then resource utilization improves, but merging updates requires a consensus algorithm
Solution Approach 1:
The patent segments the workflow execution process into distinct phases: workflow step execution between checkpoints and checkpoint phases for merging updates. This segmentation allows parallel execution during the workflow step phase while confining the complex merge operations to the checkpoint phase. The segmentation isolates the consensus algorithm to specific time points rather than requiring it continuously.
Solution Approach 2:
The patent merges all client updates at checkpoint phases using a consensus algorithm. Instead of continuously managing concurrent updates, the system accumulates updates from parallel executions and merges them periodically at checkpoints. This merging approach allows parallel execution to proceed freely while consolidating the complexity of conflict resolution to discrete merge events.
Data Source
AI summary
Techniques are disclosed for a workflow engine performing a checkpoint phase to determine a result of a merge of workflow updates. The techniques may include the workflow engine determining to allow parallel execution of updates of a workflow. The workflow engine may receive first checkpoint data associated with a first update. The workflow engine may also receive second checkpoint data associated with a second update. The workflow engine may initiate the checkpoint phase that includes comparing one or more data elements of the first checkpoint data with respective data elements of the second checkpoint data; and determining the result of the merge of the workflow updates based on the comparing. The workflow engine may then complete the checkpoint phase and transmit the result to at least one user device for presentation.