Erasure Coding Re-encoding for Geo-Distributed Storage Rebalancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Geographically diverse data storage systems employing erasure coding face challenges in efficiently managing storage space and data redundancy, particularly when adding new zones, as existing solutions often result in high computational and network resource burdens during re-protection and rebalancing processes.
Innovation Solution
The system introduces a method to scale out geographically diverse data storage by altering the erasure coding scheme, such as from a 4+2 EC scheme to a 5+2 EC scheme, allowing for re-encoding of coding chunks on existing zones rather than transferring all data chunks, thereby reducing resource demands and optimizing storage overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional re-protection and rebalancing methods are used when adding new zones, then data protection is maintained, but computational and network resource consumption increases significantly
Solution Approach 1:
The patent segments the rebalancing process by identifying and prioritizing different types of data chunks. Instead of uniformly rebalancing all chunks, it segments them into coding chunks (which provide protection) and data chunks (which store actual data), and further segments coding chunks into those that should be prioritized for rebalancing versus those that can be deferred. This selective segmentation reduces the overall computational and network resources required while maintaining data protection guarantees.
Solution Approach 2:
The patent applies local quality by treating different zones and different chunks differently based on their specific characteristics. It identifies zones with excessive coding chunks and zones with insufficient coding chunks, then selectively rebalances only the necessary chunks from source zones to destination zones. This localized approach ensures data protection is maintained in each zone without requiring global rebalancing of all data, thereby reducing overall resource consumption.
2Productivity
If all data chunks are transferred during scaling events, then storage rebalancing is achieved, but storage overhead and time consumption increase
Solution Approach 1:
The patent performs preliminary analysis before the actual rebalancing operation. It first identifies which zones have excessive coding chunks and which zones need coding chunks, then pre-determines the optimal rebalancing plan. This preliminary action allows the system to prepare the rebalancing strategy in advance, avoiding the need to transfer all data chunks and reducing the actual execution time of the scaling operation.
Solution Approach 2:
The patent applies partial action by transferring only the necessary coding chunks rather than all data chunks. It identifies the minimum set of chunks that need to be moved to achieve proper rebalancing and data protection, leaving unnecessary transfers out of the operation. This partial approach significantly reduces storage overhead and time consumption while still achieving the required rebalancing state.
3Ease of operation
If comprehensive rebalancing is performed during zone addition, then optimal storage distribution is achieved, but system complexity and resource demands increase
Solution Approach 1:
The patent changes the parameters of the rebalancing process by introducing priority levels and selective criteria for chunk transfer. Instead of using a uniform rebalancing parameter for all chunks, it modifies the rebalancing behavior based on chunk type (coding vs. data), zone status (excessive vs. insufficient coding chunks), and protection requirements. These parameter changes simplify the overall process by providing clear decision rules that reduce system complexity while achieving optimal storage distribution.
Data Source
AI summary
Rebalancing as a result of re-encoding a code chunk in response to scaling out of a geographically diverse storage system employing erasure coding technology is disclosed. After a scaling out event, a new erasure coding scheme can be selected. An old coding chunk generated according to an old erasure coding scheme can be re-encoded into a new coding chunk according to the new erasure coding scheme and based on a data chunk not previously protected by the old coding chunk. The re-encoding can be selected to diversify distribution of chunks, resulting in rebalancing occurring as part of re-encoding. In an embodiment, the new coding chunk can be generated in a new zone from the scaling out event. In another embodiment, the data chunk can be moved to the new zone from the scaling out event.