Cloud Cluster Allocation for Interruptible Big Data Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large volumes of big data using multiple machines is costly for enterprises, especially when utilizing interruptible cloud machines that can be interrupted by higher-paying parties, leading to unreliable job completion.
Innovation Solution
A system and method to identify optimal and equivalent machine configurations, prioritize cluster options based on interruption rates and pricing, and iteratively request allocations until successful, utilizing a job controller to manage cloud-based ephemeral clusters for efficient data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If interruptible cloud machines are used to reduce cost, then cost is reduced, but reliability of job completion deteriorates
Solution Approach 1:
The system pre-calculates and pre-orders multiple cluster options with different configurations and pricing levels before the data processing job starts. By preparing alternative cluster allocations in advance (including both interruptible and uninterruptible options), the system can quickly switch to backup clusters if interruptions occur, thus maintaining job completion reliability while still utilizing cost-effective interruptible machines as the primary option
Solution Approach 2:
The system creates a cushion of reserved computing resources by pre-allocating backup cluster options at different price points. When using interruptible machines, the system has already cushioned against potential interruptions by having pre-approved alternative cluster configurations ready to be activated, thereby protecting job completion reliability while maintaining cost efficiency
2Reliability
If multiple machine configurations are maintained on premises to ensure job completion, then reliability is improved, but cost increases
Solution Approach 1:
The system creates a universal cluster allocation framework that can dynamically utilize multiple cloud service providers and multiple machine configurations through a single integrated platform. Instead of maintaining dedicated physical machines for each scenario, the system establishes a multi-functional allocation mechanism that can switch between different cloud clusters (interruptible and uninterruptible) based on job requirements and availability, reducing the need for redundant on-premises infrastructure
Solution Approach 2:
The system creates virtual copies of cluster configurations across different cloud service providers. Rather than maintaining physical duplicate machines on premises, the system establishes virtual cluster representations that can be activated on-demand from various cloud sources, providing the same computational capability with reduced infrastructure costs
3Reliability
If uninterruptible cloud machines are used to ensure reliable job completion, then reliability is improved, but cost increases
Solution Approach 1:
The system implements dynamic cluster allocation that adapts to changing job requirements and cloud availability conditions. Rather than statically assigning uninterruptible machines to all jobs, the system dynamically evaluates each job's interruption tolerance and adjusts cluster allocation in real-time, switching between interruptible and uninterruptible options as needed to optimize the reliability-cost tradeoff
Data Source
AI summary
A tangible, non-transitory, machine-readable medium, comprising machine-readable instructions that, when executed by one or more processors, cause the one or more processors to determine an optimal cluster option for a data processing job from a set of machine configurations, wherein the optimal cluster option comprises an optimal configuration and a corresponding optimal number of machines having the optimal configuration, determine, from the set of machine configurations, one or more equivalent cluster options comprising equivalent configurations and a corresponding equivalent number of machines having the equivalent configuration to the optimal configuration, generate an ordered cluster options list that prioritizes the optimal cluster option and the one or more equivalent cluster options based upon long-term interruption rates, short-term availability, expected pricing or any combination thereof, and request, based on the ordered cluster options list, allocation of a cluster to implement the data processing job.