Serverless Workflow Execution With ML-Driven Multi-Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workflow management systems in cloud-based computing environments face inefficiencies such as increased costs, slower runtime, and potential data loss due to manual configuration of cloud resources and the limitations of serverless computing environments, which lack persistent state.
Innovation Solution
A serverless workflow enablement and execution platform (SWEEP) that utilizes machine learning and optimization techniques to optimize task execution configurations, including selecting cloud service providers and managing data storage, with a Scalable Cloud Storage Interface (SCSI) for intelligent data management across multiple cloud providers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual configuration of cloud resources is used, then users can control resource allocation, but execution time increases and system efficiency decreases
Solution Approach 1:
The system employs machine learning models that automatically configure and optimize cloud resource allocation without requiring manual user intervention. The ML models analyze workflow requirements and autonomously select optimal cloud service providers, allocate resources, and manage execution parameters, thereby eliminating the time-consuming manual configuration process while maintaining or improving execution efficiency.
2Ease of manufacture
If serverless computing is used, then resource provisioning is automated and cost efficiency improves, but persistent state management becomes difficult leading to potential data loss
Solution Approach 1:
The system introduces an intermediary persistent storage layer that sits between the stateless serverless computing environment and the workflow data. This storage layer automatically manages data persistence, ensuring that intermediate results and state information are preserved across function invocations. The ML models coordinate with this storage layer to maintain data integrity while preserving the cost-efficiency and automation benefits of serverless computing.
3Adaptability or versatility
If cloud service providers are selected manually, then users can choose specific providers, but the selection process is time-consuming and may lead to suboptimal performance
Solution Approach 1:
The system implements a feedback-driven approach where machine learning models continuously learn from execution performance data across multiple cloud service providers. The models analyze metrics such as execution time, cost, and reliability to automatically select optimal providers for different workflow tasks. This feedback mechanism enables the system to adapt to changing conditions and provider performance while eliminating manual selection time, maintaining flexibility through automated decision-making.
4Productivity
If optimization techniques are applied to resource allocation, then execution efficiency improves, but system complexity increases
Solution Approach 1:
The system employs a universal machine learning framework that handles multiple optimization functions including provider selection, resource allocation, and cost management through a single integrated platform. The ML models serve multiple purposes: they optimize current workflow execution, learn from historical data, and provide recommendations for future improvements. This multi-functional approach achieves high resource allocation efficiency while managing system complexity through consolidation rather than proliferation of separate optimization systems.
Data Source
AI summary
The present disclosure provides computing systems and methods that optimize the execution of workflows that include computational tasks (e.g., which may take the form of functions or containers). In general, the proposed systems and methods can be referred as to or embodied within a serverless workflow enablement and execution platform (also referred to herein as a workflow management system). The serverless workflow platform can facilitate performance of a large-scale computational workflow. In particular, the serverless workflow platform can facilitate performance of serverless workflows that are executed on serverless execution platforms.