Distributed Workflow Engine Stateless Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional workflow management systems face challenges in coordinating and tracking processing steps across multiple computing systems, often resulting in time-consuming and costly operations, especially when dealing with extended processes that exceed execution limits, leading to degraded performance and failures.
Innovation Solution
A distributed workflow management system that utilizes a workflow engine, a non-relational database, and a queue service to execute tasks based solely on the current state, without relying on process execution history, allowing for scalable and limitless process duration and execution without scaling limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional workflow management systems are used to coordinate processing steps across multiple computing systems, then workflow coordination and tracking are achieved, but the system becomes time-consuming and costly with degraded performance for extended processes
Solution Approach 1:
The workflow management system is segmented into independent microservices that can be deployed and executed across multiple computing systems. Each microservice handles specific workflow tasks independently, allowing parallel execution and eliminating the performance degradation associated with monolithic traditional systems. This segmentation enables the system to maintain reliability while reducing execution time through distributed processing.
Solution Approach 2:
An event-driven intermediary layer is introduced between workflow tasks and computing systems. Events are published to a message broker that routes them to appropriate microservices, decoupling the coordination logic from execution logic. This intermediary enables asynchronous processing, allowing the system to coordinate complex workflows without blocking, thereby reducing overall execution time while maintaining coordination reliability.
2Loss of information
If traditional workflow management systems track processing steps with execution history, then complete process tracking is achieved, but the systems experience execution limits and degraded performance for extended processes
Solution Approach 1:
The execution history tracking function is extracted from the core workflow engine and implemented as a separate event logging mechanism. Events are published to an event stream that persists historical data independently of the active workflow state. This extraction allows the system to maintain complete process tracking information without allowing history data to constrain or limit ongoing process execution, enabling unlimited process duration while preserving full execution history.
Solution Approach 2:
Instead of maintaining a single source of truth for workflow state that becomes burdened by execution history, the system creates immutable copies of workflow events in an event stream. Each event is a copy of the state at a point in time, preserved independently. This copying approach allows the system to retain complete execution history for audit and analysis purposes while the active workflow can proceed without being constrained by historical data, thus extending process duration limits.
3Reliability
If workflow management systems use centralized coordination across multiple computing systems, then workflow coordination is achieved, but scalability is limited and performance degrades
Solution Approach 1:
The centralized workflow coordination is segmented into distributed microservices that autonomously coordinate through event-driven communication. Each microservice instance can handle workflow tasks independently, and the system can dynamically add or remove service instances based on load. This segmentation transforms the scalability constraint into an advantage, allowing the system to scale horizontally across multiple computing systems while maintaining coordination reliability through standardized event interfaces.
Solution Approach 2:
The workflow management system transitions from static centralized coordination to dynamic distributed coordination. Microservices can be dynamically instantiated, scaled, and terminated based on demand. The event-driven architecture allows the system to adapt its coordination capacity in real-time, with new service instances automatically joining the distributed coordination network. This dynamic approach enables the system to scale adaptively across computing systems while maintaining reliable workflow coordination through event-based communication protocols.
Data Source
AI summary
A workflow management system having a queue service, a non-relational database, and a workflow engine. The queue service is configured to receive a task from a computing device. The non-relational database is configured to store data. The workflow engine has a workflow definition defining a plurality of operating states for an application. The workflow engine in communication with the queue service and the non-relational database, the workflow engine is configured to receive tasks from the queue service, query the non-relational database, receive query data from the non-relational database, and execute tasks based on task data, process data, and the workflow definition.