Cloud Workflow File Execution With Containerized Engine Wrappers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a gap in workflow execution support on cloud computing environments, as existing cloud capabilities often fail to execute workflows created by desktop-based applications seamlessly, leading to inefficiencies in data processing and management.
Innovation Solution
A cloud execution ecosystem is implemented, comprising a control plane and data planes that facilitate the execution of workflows adjacent to data sources, using a containerized execution engine wrapper to manage and execute workflow files, enabling support across various cloud environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If workflows are executed on cloud computing environments, then data processing scalability and accessibility are improved, but execution reliability and compatibility with desktop-based workflows deteriorate
Solution Approach 1:
The patent introduces a workflow engine as an intermediary component that translates desktop-based workflow definitions into executable cloud tasks. This mediator layer ensures compatibility between desktop workflow creators and cloud execution environments, resolving the reliability issue while maintaining cloud scalability.
Solution Approach 2:
The system segments workflow execution into distinct components: workflow definition (desktop), workflow engine (translation layer), and cloud task execution (cloud environment). This segmentation allows each component to be optimized independently, ensuring reliable execution across different platforms.
2Stability of the object's composition
If a centralized workflow execution system is used, then workflow management consistency is improved, but system complexity and resource requirements worsen
Solution Approach 1:
The patent transitions from a centralized monolithic architecture to a distributed hierarchical architecture with multiple dimensions: control plane for management, execution plane for running workflows, and worker nodes for actual computation. This dimensional separation maintains consistency while reducing single-point complexity.
Solution Approach 2:
The workflow engine serves multiple functions: translating workflow definitions, validating execution parameters, managing resource allocation, and coordinating task execution. This multi-functionality reduces the need for separate specialized components, thereby reducing overall system complexity.
3Productivity
If workflows are executed closer to data sources, then processing performance is improved, but infrastructure complexity and setup requirements worsen
Solution Approach 1:
The system enables self-service workflow execution where users can submit workflows without manually configuring complex infrastructure. The workflow engine automatically handles resource allocation, data location, and execution coordination, making performance optimization accessible without increasing user-facing complexity.
Solution Approach 2:
The patent allows dynamic adjustment of execution parameters such as data location, resource allocation, and execution timing. By changing these parameters rather than reconfiguring infrastructure, the system achieves performance optimization without increasing structural complexity.
Data Source
AI summary
A data analytics system that includes a control plane for executing workflows is described. The data analytics system receives workflow files from client devices and stores the workflow files in respective storage accounts associated with the respective client device profiles. A virtualized container separate from the control plane executes the workflow file by retrieving the workflow file directly from a storage account or via an application programming interface provided by the control plane. Results generated by the virtualized container are stored in a corresponding storage account for subsequent access, such as by a client device from which a workflow execution request was received.