Serverless Workflow Orchestration With Multi-Cloud Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workflow management systems in cloud-based computing environments face inefficiencies such as increased costs, slower runtime, and potential data loss due to manual configuration of cloud resources and limitations in data storage, especially in serverless environments.
Innovation Solution
A serverless workflow enablement and execution platform (SWEEP) that utilizes machine learning and optimization techniques to optimize task execution configurations, including selecting cloud service providers, timing, and resource allocation, with a Scalable Cloud Storage Interface (SCSI) for intelligent data management across multiple cloud providers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual configuration of cloud resources is used, then user control over infrastructure is maintained, but system setup time increases and efficiency decreases
Solution Approach 1:
The system automatically configures cloud resources and executes workflows without requiring manual user intervention. The workflow management system self-manages infrastructure provisioning, resource allocation, and task execution, eliminating the need for users to manually configure cloud resources while maintaining full control through automated orchestration
Solution Approach 2:
The system pre-configures cloud resources and establishes execution environments before workflow tasks need to run. Infrastructure is provisioned in advance based on workflow definitions, ensuring that when workflows execute, all necessary resources are already in place and ready, eliminating setup time delays
2Productivity
If manual configuration of cloud resources is used, then infrastructure control is maintained, but execution time increases
Solution Approach 1:
Cloud resources are pre-provisioned and pre-configured before workflow execution begins. Execution environments are set up in advance, so when workflows start running, infrastructure is already ready rather than being provisioned during execution, significantly reducing runtime
Solution Approach 2:
The system continuously monitors workflow execution and cloud resource performance, using feedback to dynamically optimize resource allocation and execution timing. This enables the system to adjust resource distribution in real-time to minimize execution time while maintaining efficiency
3Ease of operation
If serverless computing is used, then infrastructure maintenance is relieved, but persistent data storage capability is lost
Solution Approach 1:
The system introduces a persistent storage layer as an intermediary between serverless compute functions and data storage. This storage layer provides persistent data storage capability while maintaining the serverless computing model's benefits. The intermediary storage system bridges the gap between stateless compute and persistent data requirements
Solution Approach 2:
The cloud resource management system serves multiple functions: it manages serverless compute execution, provides persistent data storage, handles resource provisioning, and coordinates workflow orchestration. By making the infrastructure management system multi-functional, the patent achieves both serverless ease of operation and reliable data persistence
4Productivity
If cloud resources are provisioned for workflow execution, then computational tasks can be executed, but system complexity increases
Solution Approach 1:
The workflow management system automatically handles all infrastructure provisioning, resource allocation, and coordination tasks without requiring manual user configuration. The system self-manages the complexity of cloud resource management, presenting a simplified interface for workflow execution while handling underlying complexity automatically
Solution Approach 2:
The system divides workflow execution into discrete, manageable segments or tasks that can be independently provisioned and executed. By segmenting the workflow into individual units with specific resource requirements, the system simplifies the complexity of managing large-scale cloud infrastructure through modular, granular control
Data Source
AI summary
The present disclosure provides computing systems and methods that optimize the execution of workflows that include computational tasks (e.g., which may take the form of functions or containers). In general, the proposed systems and methods can be referred as to or embodied within a serverless workflow enablement and execution platform (also referred to herein as a workflow management system). The serverless workflow platform can facilitate performance of a large-scale computational workflow. In particular, the serverless workflow platform can facilitate performance of serverless workflows that are executed on serverless execution platforms.