Erasure Coding Scale-Out via Adjacent Sub-Fragment Layout
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure coding mechanisms do not scale effectively with the expansion of storage clusters, leading to challenges in maintaining data protection and capacity efficiency as the number of nodes changes, resulting in increased workload and resource utilization during re-protection processes.
Innovation Solution
The system employs a re-protection component that adjusts the erasure coding scheme by dividing and re-encoding data fragments across additional nodes, optimizing the storage layout and reducing resource usage by storing data and coding fragments as groups of adjacent sub-fragments and combining them to generate new coding fragments, thereby adapting to changes in cluster size and improving capacity use efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional erasure coding mechanisms are used in expanding storage clusters, then data protection is maintained, but workload and resource utilization increase significantly during re-protection processes
Solution Approach 1:
The patent segments data fragments into groups of adjacent sub-fragments and stores them in a structured layout across nodes. This segmentation allows the system to process and move only specific sub-fragments during re-protection rather than entire data fragments, reducing the workload during cluster expansion while maintaining data protection through erasure coding.
2Reliability
If conventional erasure coding mechanisms are used in expanding storage clusters, then data protection is maintained, but resource utilization increases during re-protection processes
Solution Approach 1:
The patent performs preliminary actions by pre-organizing data fragments into groups of adjacent sub-fragments with a specific layout structure before cluster expansion occurs. This preliminary organization enables more efficient re-protection operations during scaling, reducing the computational resources and energy required when nodes are added to the cluster.
3Adaptability or versatility
If erasure coding schemes are scaled to match new cluster size, then capacity use efficiency is enhanced, but network and drive traffic increase during re-protection
Solution Approach 1:
The patent applies local quality by storing data sub-fragments and coding sub-fragments in specific local positions within nodes, creating a structured layout where adjacent sub-fragments are grouped together. This local organization enables selective movement of only necessary sub-fragments during re-protection, reducing overall network and drive traffic while allowing the erasure coding scheme to scale efficiently with cluster size.
Data Source
AI summary
Overhead associated with data re-protection during scaling out and/or scaling up of a cloud storage system can be reduced. During initial data protection (e.g., prior to a change in the cluster size), data fragments and coding fragments can be stored as a set of adjacent data sub-fragments and a set of adjacent coding sub-fragments, respectively. In response to determining that the cluster has expanded, a portion of the set of adjacent data sub-fragments can be moved to the new nodes that have been added to the cluster. Further, the set of adjacent coding sub-fragments can be combined to generate a new coding fragment that can be stored within the new cluster. Accordingly, the amount of system resources utilized during re-protection can be significantly reduced.