Decentralized Workflow Execution via Virtual Machine Triggering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional centralized workflow schedulers require dedicated processing resources throughout the workflow execution, leading to inefficiencies in resource allocation and management, especially in cloud environments where jobs with varying resource requirements are executed sequentially.
Innovation Solution
A decentralized workflow execution system where each virtual machine generates workflow code that initiates the next job in the sequence upon completion, utilizing a resource manager to allocate and deallocate system resources dynamically, allowing different jobs to execute on different virtual machines with varying resource sets, and storing workflow data for access by subsequent jobs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized workflow scheduler is used to monitor and initiate jobs, then workflow execution control is maintained, but dedicated processing resources are consumed continuously throughout the workflow lifecycle
Solution Approach 1:
The patent extracts the centralized workflow scheduler from the system, removing the dedicated processing resources that were continuously consumed for monitoring and job initiation. Each job instead contains embedded logic to trigger the next job in the sequence, eliminating the need for continuous centralized control while maintaining workflow execution reliability.
Solution Approach 2:
The system implements self-service by embedding job triggering logic directly within each job definition. Completed jobs automatically initiate their successors without external intervention, allowing the workflow to self-manage its execution sequence without requiring dedicated scheduler resources.
2Reliability
If a centralized workflow scheduler is used to monitor job progress, then job completion detection is reliable, but resource allocation efficiency decreases
Solution Approach 1:
The centralized monitoring function is extracted and replaced by distributed completion signals from individual jobs. Each job notifies the system of its completion status, eliminating the need for continuous centralized monitoring while maintaining reliable job completion detection through event-driven notifications.
Solution Approach 2:
The mechanical centralized monitoring system is replaced with an event-driven architecture where job completion triggers are automatically generated and processed. This substitution eliminates the need for continuous polling or active monitoring, improving resource allocation efficiency through automatic event-based detection.
3Ease of operation
If dedicated processing resources are allocated throughout the workflow lifecycle, then workflow management is simplified, but resource utilization flexibility is reduced
Solution Approach 1:
The system transitions from static dedicated resource allocation to dynamic resource provisioning. Resources are allocated on-demand based on actual workflow needs, with the ability to scale and adapt resource requirements for each job in the sequence, thereby improving resource utilization flexibility while maintaining ease of operation through automated management.
Solution Approach 2:
The platform achieves universality by enabling a single resource pool to serve multiple workflow jobs with varying requirements. The system can dynamically allocate different resource configurations to different jobs in the workflow sequence, allowing one infrastructure to handle diverse workloads flexibly without requiring dedicated resources for each job type.
Data Source
AI summary
Techniques for de-centralized execution of workflows are disclosed. A system executes a sequence of jobs in a workflow. Each job includes application code to perform a defined set of functions. Each job includes wrapper code at one or both ends of the application code. The system executes the sequence of jobs by iteratively spinning up a virtual machine, loading workflow data to be used by application code, executing the application code, and storing modified workflow data. The virtual machine executing one job triggers the next virtual machine to execute the next job in the workflow, without scheduling the next job by a centralized job scheduler. Upon spinning up the next virtual machine, a virtual machine executing a preceding job shuts itself down.