Computational Workflow Engine for Parallel Task Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workflow engines are limited to sequential processing of software application tasks, leading to higher execution times and unnecessary delays, especially in complex workflows, and lack the ability to dynamically break or reuse workflows.
Innovation Solution
The implementation of a computational workflow engine that utilizes graph-based data structures and specific algorithms for traversing the graph to enable both sequential and parallel processing, along with dynamic reuse of workflows using circuit breaker and short circuit techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential processing is used in workflow engines, then task execution follows a simple linear flow, but execution time increases and processing efficiency decreases
Solution Approach 1:
The workflow is segmented into independent tasks represented as nodes in a directed acyclic graph (DAG). Each node can be executed independently or in parallel with other nodes, breaking the monolithic sequential flow into discrete, manageable units that can be processed concurrently when dependencies allow.
Solution Approach 2:
The workflow engine dynamically determines execution order and parallelism based on task dependencies defined in the DAG. The engine can adaptively switch between sequential and parallel execution modes, optimizing the execution path based on completed tasks and remaining dependencies, rather than following a fixed linear sequence.
2Productivity
If workflow engines process tasks sequentially, then system complexity remains low, but execution time and delays increase significantly
Solution Approach 1:
The workflow engine acts as an intermediary that manages task scheduling and execution. It maintains the DAG structure and dependency relationships, automatically determining which tasks can execute in parallel and which must follow sequentially, thereby managing complexity internally while presenting simplified task definitions to users.
Solution Approach 2:
The system changes the parameter of task execution from strictly sequential to parallelizable by introducing dependency-based scheduling. Tasks are assigned execution states (pending, executing, completed) and the engine dynamically adjusts the execution parameter based on dependency satisfaction, enabling parallelism without requiring complex user intervention.
3Adaptability or versatility
If traditional workflow engines are used, then implementation is straightforward, but the ability to dynamically break or skip workflow flow is limited
Solution Approach 1:
The workflow structure and all possible execution paths are predefined in the DAG before runtime. Dependencies, successors, and termination conditions are established in advance, allowing the engine to dynamically determine execution paths based on completed tasks without requiring complex runtime decision-making or workflow modification.
Solution Approach 2:
The engine supports skipping intermediate tasks by directly transitioning to successor nodes when certain conditions are met or when tasks are marked as optional. This allows selective bypassing of workflow steps while maintaining the overall DAG structure and dependency integrity, enabling flexible workflow adaptation.
4Adaptability or versatility
If workflows are created for each specific process, then customization is high, but maintenance burden increases and reuse becomes difficult
Solution Approach 1:
The DAG-based workflow structure serves as a universal template that can represent multiple different business processes by configuring different nodes, edges, and termination conditions. A single workflow graph can be reused across multiple processes by selectively activating different paths or modifying task parameters, reducing the need to create entirely separate workflows for each process.
Solution Approach 2:
The system enables copying and reusing workflow subgraphs or individual nodes across different processes. Common task patterns and workflow segments can be defined once and instantiated multiple times with different configurations, reducing maintenance burden and ensuring consistency across similar processes.
Data Source
AI summary
A method for processing a software application workflow using a workflow engine includes receiving a workflow configuration defining the workflow including a plurality of nodes and a plurality of connections between the nodes, each of the nodes associated with a corresponding software processing task. The method includes determining a first set of nodes of the plurality of nodes to be executed based on the workflow configuration and the plurality of connections. The method includes causing execution of the software processing tasks associated with the first set of nodes, resulting in an execution result. The method includes determining a second set of nodes of the plurality of nodes to be executed based on the workflow configuration, the plurality of connections, and the execution result. The method includes causing execution in parallel of the tasks associated with the second set of nodes.