Erasure Code Fragment Rebalancing in Scale-Out Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage area network (SAN) and network-attached storage (NAS) architectures are inadequate for supporting large multi-petabyte data storage and protection, necessitating the use of object storage technology like Elastic Cloud Storage (ECS) that employs erasure coding for data protection, but lacks a universal mechanism for efficient data rebalancing during cluster expansion or contraction.
Innovation Solution
A system and method for universal data rebalancing in scale-out/scale-up cloud-based object storage systems, which involves determining non-integer values representing portions of erasure-coded fragments to be transferred between nodes, using floor and ceiling functions to ensure balanced fragment distribution across nodes, thereby maintaining data availability and system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional SAN/NAS architectures are used for data storage, then data protection schemes like mirroring or parity protection can be employed, but the system cannot support large multi-petabyte capacity levels
Solution Approach 1:
The patent changes the fundamental parameter of data protection from traditional mirroring/parity schemes to erasure coding, which divides data into fragments and distributes them across multiple nodes. This allows the system to scale to multi-petabyte capacities while maintaining data protection through mathematical redundancy rather than simple duplication.
2Quantity of substance
If erasure coding is used for data protection in scale-out storage systems, then large capacity levels can be supported, but the system lacks a universal mechanism for efficient data rebalancing during cluster expansion or contraction
Solution Approach 1:
The patent calculates the target number of fragments per node before rebalancing operations begin, determining the ideal distribution state in advance. This preliminary calculation enables the system to efficiently guide the rebalancing process by knowing the destination state, reducing operational complexity during cluster expansion or contraction.
Solution Approach 2:
The patent segments the rebalancing process into individual fragment transfer operations, where each fragment is moved independently from source nodes to target nodes based on calculated requirements. This segmentation allows for fine-grained control and efficient parallel execution of rebalancing operations across the distributed cluster.
3Quantity of substance
If nodes are added to expand cluster capacity, then storage capacity increases, but fragment distribution becomes unbalanced requiring complex rebalancing operations
Solution Approach 1:
The patent implements a self-service rebalancing mechanism where the system automatically calculates fragment distribution requirements and executes transfers without manual intervention. The rebalancing algorithm autonomously determines which fragments to move and where to place them, reducing operational complexity while maintaining balanced fragment distribution across expanded cluster capacity.
Data Source
AI summary
A universal mechanism is utilized for data rebalancing in a scaled-out data storage cluster. A value (l) representing a number of erasure coded fragments of each data portion that are to be moved to a newly added node can be calculated. Initially, the number of erasure coded fragments moved per data fragment is determined based on the greatest integer that is less than or equal to l and remainders are accumulated. When accumulated reminders equal or exceed l, the number of erasure coded fragments moved per data fragment is determined based on the lowest integer that is greater than or equal to l. A value of accumulated reminders is then decreased by l. Accordingly, system-level imbalances can be avoided and data availability, data robustness, and/or overall system performance can be increased.