Heterogeneous Auto-Scaling Cloud Big Data Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based big data services, heterogeneous clusters composed of varying instance types face challenges in provisioning and optimization due to the unavailability of desired machine types and varying costs, especially in shared resource environments like Amazon Web Services, where users need to manage and optimize costs effectively while ensuring optimal performance.
Innovation Solution
A method for heterogeneously auto-scaling cloud-based big data clusters that determines provisioning requirements based on a primary instance type, assigns weights to other instance types, and dynamically adjusts instance types to minimize costs and maximize processing efficiency, using a normalized instance count to ensure optimal resource utilization across different instance types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If users purchase spot instances at lower prices to reduce costs, then cost savings are achieved, but processing speed and availability may be compromised
Solution Approach 1:
The system dynamically changes instance type parameters based on workload requirements and pricing conditions. It selects from multiple instance types with different performance-cost ratios, adjusting the mix of spot and on-demand instances, and choosing different instance families (compute-optimized, memory-optimized, general-purpose) to optimize both cost and processing speed for different job types
2Productivity
If heterogeneous instance types are used to optimize cost and performance, then resource utilization improves, but cluster provisioning complexity increases
Solution Approach 1:
The system uses instance weights as normalization parameters to convert heterogeneous instance types into a unified scaling metric. Each instance type is assigned a weight reflecting its relative processing power compared to a baseline type, allowing the auto-scaling controller to manage diverse instances through a single normalized instance count parameter
Solution Approach 2:
The system creates a universal provisioning interface that works across multiple instance types and cloud providers. The weight-based normalization creates a common language for specifying cluster size regardless of the underlying instance heterogeneity, making the system portable and provider-agnostic
3Loss of energy
If spot instances are used to maximize cost savings, then cost efficiency improves, but instance availability and stability deteriorate
Solution Approach 1:
The system dynamically adjusts the proportion of spot versus on-demand instances based on current pricing conditions, workload urgency, and cluster needs. The auto-scaling controller monitors spot instance availability and can shift provisioning strategies in real-time, increasing on-demand instances when spot instances become unavailable or increasing spot usage when prices are favorable
Solution Approach 2:
The system changes the mix of instance types and purchase options as a dynamic parameter. It can adjust the spot instance utilization ratio, switch between different instance families, and modify the normalized instance count allocation across different types based on real-time conditions, optimizing the reliability-cost tradeoff
Data Source
AI summary
The present invention is generally directed to systems and methods of provisioning, and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.