Dynamic Subtask Creation in Workflow Processing Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed processing platforms face inefficiencies in managing complex workflows due to limitations in static workflow definitions, which struggle to dynamically interrelate tasks and require multiple DAGs to account for various scenarios, leading to resource strain and delayed execution.
Innovation Solution
Implementing dynamic workflow generation and execution mechanisms that allow workers to create and execute sub-tasks based on task parameters and available resources, enabling flexible task division and execution by multiple workers, reducing the need for multiple static workflows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If static workflow definitions are used to manage complex workflows, then workflow management is simplified, but the system cannot dynamically adapt to different scenarios and requires multiple static workflows
Solution Approach 1:
The patent implements dynamic workflow generation where workers can create sub-tasks dynamically based on task parameters and available resources. The workflow definition is no longer static but evolves during execution, allowing the system to adapt to different scenarios without requiring multiple predefined static workflows. This resolves the contradiction by making the workflow management system itself dynamic while maintaining simplicity through automated generation.
Solution Approach 2:
The patent divides workflows into tasks and sub-tasks, allowing complex workflows to be segmented into manageable units. Workers can dynamically create and manage sub-tasks based on the complexity of the current task, segmenting the workflow into appropriate granularities without requiring the entire workflow to be predefined. This segmentation enables adaptability while reducing overall management complexity.
2Adaptability or versatility
If multiple static DAGs are used to account for various scenarios, then workflow coverage is improved, but resource strain increases and execution is delayed
Solution Approach 1:
Instead of maintaining multiple static DAGs for different scenarios, the system dynamically generates the workflow structure during execution. Workers assess task parameters and available resources in real-time, creating only the necessary sub-tasks and DAG structure for the current scenario. This eliminates the resource strain and execution delays associated with managing multiple static DAGs while maintaining comprehensive scenario coverage.
Solution Approach 2:
The system performs preliminary assessment of task parameters and resource availability before generating the workflow. Workers evaluate what sub-tasks are needed based on the current state, allowing the workflow generation to be both dynamic and efficient. This preliminary action prevents unnecessary workflow creation and optimizes execution speed.
3Ease of operation
If workers execute tasks independently, then execution flexibility is improved, but coordination and resource management become more difficult
Solution Approach 1:
The patent implements a feedback mechanism where workers report their execution status, resource consumption, and completion of sub-tasks back to the workflow service. This feedback loop enables the system to automatically coordinate resources and manage complexity while maintaining worker independence. The workflow service uses this feedback to dynamically adjust workflow assignments and resource allocation, resolving the coordination complexity issue.
Solution Approach 2:
The workflow service acts as an intermediary between independent workers and the resource management system. It receives task definitions from the task service, assigns them to workers, collects execution results, and manages resource allocation. This intermediary layer simplifies coordination complexity while preserving execution flexibility, as workers operate independently but through a coordinated framework managed by the workflow service.
Data Source
AI summary
A method for dynamically generating and executing tasks can include includes executing a worker execution stream, where the worker execution stream includes multiple execution threads associated with a workflow of the workflow service, receiving, by the worker execution stream, from a workflow service, a definition of a task, and responsive to determining that the definition of the task satisfies a predefined criterion, dividing the task into a set of sub-tasks. The method further includes generating a definition of a sub-task workflow for the set of sub-tasks, and causing the workflow service to distribute, based on the definition of the sub-task workflow, the sub-tasks of the set to one or more workers for execution.