Step-Enabled Workflow Buffering for Resource-Limited Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies when processing large workflows due to the need for substantial resource allocation for all tasks to run simultaneously, which can lead to resource constraints and high costs, especially in cloud computing environments.
Innovation Solution
Implementing a step-based workflow processing method using buffering functions like first-in-first-out (FIFO) to allow tasks to operate one at a time or in groups, enabling efficient use of resources by transitioning tasks between running and non-running states based on data thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional workflow management systems are used, then workflow execution can be maintained, but the systems lack the capability to dynamically adapt to changing business requirements and cannot effectively manage complex, non-linear workflows
Solution Approach 1:
The workflow management system is segmented into discrete, reusable workflow steps that can be independently configured and combined. Each workflow step represents a modular unit that can be selected and arranged to create different workflow paths, enabling dynamic adaptation without requiring system-wide changes.
Solution Approach 2:
The system implements dynamic workflow management where workflow definitions can be modified at runtime. The workflow engine evaluates conditions and dynamically determines which workflow steps to execute next, allowing the workflow to adapt to changing business requirements without manual reconfiguration of the entire system.
2Productivity
If manual workflow management is used, then system complexity is reduced, but productivity and efficiency deteriorate due to time-consuming manual interventions
Solution Approach 1:
The workflow management system operates autonomously by automatically selecting and executing appropriate workflow steps based on predefined conditions and current workflow state. The system self-manages workflow progression without requiring manual intervention, thereby improving productivity while maintaining manageable complexity through rule-based automation.
Solution Approach 2:
The system incorporates feedback mechanisms where workflow execution results are automatically evaluated against predefined conditions. This feedback drives automatic decision-making about subsequent workflow steps, enabling efficient automation while keeping the system manageable through clear, rule-based logic.
3Loss of information
If comprehensive workflow tracking is implemented, then workflow visibility and control are improved, but information processing overhead increases
Solution Approach 1:
The system maintains workflow state information through structured data representations and metadata associated with each workflow step. Rather than complex tracking mechanisms, the system uses standardized data copies and state representations that enable efficient information retrieval and processing while maintaining comprehensive workflow visibility.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Aspects of the disclosure provide methods and an apparatus including processing circuitry configured to receive workflow information of a workflow. The processing circuitry generates, based on the workflow information, the workflow including a first buffering task and a plurality of processing tasks that includes a first processing task and a second processing task. The first processing task is caused to enter a running state in which a subset of input data is processed and output to the first buffering task as first processed subset data. The first processing task is caused to transition from the running state to a non-running state based on an amount of the first processed subset data in the first buffering task being equal to a first threshold. Subsequently, the second processing task is caused to enter a running state in which the first processed subset data in the first buffering task is processed.