Monotone Distribution Scheme for Stateful Cluster Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cluster scaling techniques often block access or require costly offline data copying, making it challenging to resize stateful clusters while maintaining access, especially in distributed storage systems.
Innovation Solution
Implementing a monotone distribution scheme to logically and physically redistribute data across additional nodes within the cluster, allowing continuous access during scaling, and using a remote key-value storage system for incremental backups to facilitate faster recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional cluster scaling techniques are used, then cluster size can be changed, but access to data is blocked or requires costly offline copying
Solution Approach 1:
The patent segments data into partitions distributed across multiple nodes using a monotone distribution scheme. This allows the cluster to be scaled by adding or removing nodes while maintaining data accessibility, as each node holds a specific partition and the distribution scheme ensures continuous access paths are preserved during scaling operations.
Solution Approach 2:
The patent performs preliminary actions by pre-establishing a monotone distribution scheme that maps data partitions to nodes before scaling occurs. This pre-planned distribution strategy enables the system to handle scaling operations without blocking access, as the redistribution follows a predetermined pattern that maintains accessibility invariants.
2Adaptability or versatility
If traditional cluster scaling techniques are used, then cluster size can be changed, but offline data copying is required which increases time and resource consumption
Solution Approach 1:
The patent implements dynamic scaling by allowing the cluster configuration to change in real-time without requiring offline copying. The monotone distribution scheme enables dynamic redistribution of data partitions as nodes are added or removed, making the scaling process continuous and eliminating the need for time-consuming offline operations.
Solution Approach 2:
The patent maintains continuous data access and processing during scaling operations through the monotone distribution scheme. The system performs scaling and data redistribution concurrently with ongoing workloads, ensuring that the useful action of data processing continues without interruption while the cluster size changes.
3Productivity
If more nodes are added to the cluster, then processing capacity increases, but data redistribution complexity increases
Solution Approach 1:
The patent uses parameter changes through the monotone distribution scheme to manage data redistribution. By changing the distribution parameters based on the desired cluster size and current configuration, the system can efficiently redistribute data partitions when nodes are added or removed, maintaining scalability without excessive complexity.
Data Source
AI summary
A stateful cluster may implement scaling of the stateful cluster while maintaining access to the state of the stateful cluster. A scaling event for a stateful cluster may be detected, and in response the stateful cluster may be adjusted to include a different number of nodes. The state of the cluster may then be logically distributed among the different number of nodes according to a monotone distribution scheme. The adjusted node may then service access requests according to the monotone distribution scheme. Prior to making the adjusted storage cluster available for servicing access requests, the nodes from the original cluster may still service access requests for state.