Storage Device Grouping for Data Consistency in Large Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing clusters, increasing the number of nodes to enhance scalability leads to a higher likelihood of disk failures, which can result in data loss, and existing solutions either constrain the number of nodes or increase replication factors at significant cost, without effectively managing data loss probabilities.
Innovation Solution
Implementing a multi-objective storage device grouping technique that forms groups of storage devices to manage data loss parameters, such as disk failure rates and replication factors, to maintain an acceptable data loss probability while allowing for cluster expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of nodes in a cluster is increased to enhance scalability, then the computing and storage capabilities are improved, but the likelihood of disk failures increases leading to higher data loss probability
Solution Approach 1:
The patent segments the cluster into multiple storage device groups, where each group independently stores datasets and their replicas. This segmentation isolates failure domains, so that a disk failure in one group does not propagate to other groups, thereby maintaining data availability even as the overall cluster size increases
Solution Approach 2:
The patent dynamically adjusts the number of replicas and group configurations based on cluster size and failure probability parameters. As the cluster grows, the system modifies replication parameters and group structures to maintain acceptable data loss probability thresholds, rather than using a fixed replication strategy
2Productivity
If data is distributed across more disks to reduce rebuild time, then the rebuild efficiency is improved, but the likelihood of a second disk failure before rebuild completion increases
Solution Approach 1:
By segmenting storage devices into groups with limited sizes, the patent ensures that rebuild operations are confined to smaller subsets of disks. This limits the exposure window during which a second failure could occur, while still achieving acceptable rebuild times through optimized group configurations
Solution Approach 2:
The patent may maintain more replicas than the minimum required for basic redundancy, particularly in larger clusters. This excessive replication strategy extends the safe window for rebuild operations by providing additional copy sources, thereby reducing the risk of data loss during rebuild even as cluster size increases
3Reliability
If the replication factor is increased to mitigate data loss probability, then the data consistency is improved, but the computing, storage, and networking resource requirements increase substantially
Solution Approach 1:
The patent applies segmentation by organizing storage devices into groups where replication is optimized at the group level rather than cluster-wide. This allows the system to achieve data consistency within each group with fewer replicas, reducing overall resource requirements while maintaining reliability
Solution Approach 2:
The patent implements local quality optimization by applying different replication factors to different storage device groups based on their specific characteristics, such as failure rates and performance requirements. This localized approach avoids the uniform high replication overhead across the entire cluster, reducing overall resource consumption while maintaining data consistency
Data Source
AI summary
Storage device groups are formed with respect to data consistency policies and/or quantified probabilities. A method embodiment commences upon identifying a computing system having a plurality of storage devices that are accessed by a plurality of computing nodes. A user interface serves for collecting policies, data loss parameters, and data rebuild parameters. Based on the policies and/or values of the data loss parameters, and values of the data rebuild parameters, sets of storage device groups are formed to achieve particular data loss and rebuild time properties. Data storage containers such as files or virtual disks that hold persistent datasets are assigned to respective storage device groups that are appropriate to the nature of the dataset. The objective pertaining to an acceptable likelihood of data loss as well as the objective of an acceptable time for rebuild are achieved as a result of assignments of certain storage devices into a group.