Distributed Erasure Coding for Unbalanced Zone Loads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed erasure coding in cloud storage systems faces high capacity overhead when load is unbalanced across zones, as existing methods are insufficient for recovering data from multiple failures without excessive network burden and inefficient resource utilization.
Innovation Solution
Implementing a k+m erasure coding scheme that produces partial coding chunks for under-loaded zones, replacing missing data chunks with virtual chunks during encoding, and distributing these chunks across zones to reduce capacity overhead and enable efficient data recovery from multiple failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed erasure coding is implemented in cloud storage systems, then data recovery from multiple failures is enabled, but capacity overhead grows unacceptably high when load is unbalanced across zones
Solution Approach 1:
The patent segments the erasure coding process into distinct phases: a first encoding phase that executes immediately when data is written, and a second encoding phase that executes later when load balancing opportunities arise. This segmentation allows the system to maintain data protection while deferring capacity-intensive operations to times when load conditions are more favorable, thereby reducing overall capacity overhead.
Solution Approach 2:
The system performs preliminary encoding actions during the first encoding phase to establish basic data protection, then prepares for additional encoding in the second phase when load conditions permit. This preliminary action ensures that data is protected from the outset while allowing the system to optimize capacity usage through subsequent encoding operations that take advantage of load balancing opportunities.
2Quantity of substance
If distributed erasure coding is implemented with perfect load balancing, then capacity overhead is minimized, but system complexity increases to maintain balance
Solution Approach 1:
The system employs self-service mechanisms where zones autonomously monitor their own load conditions and independently execute encoding operations when their local conditions are favorable. This eliminates the need for complex centralized load balancing control, as each zone serves itself by performing encoding operations when it has available capacity, thereby reducing system complexity while still achieving load balancing benefits.
Solution Approach 2:
The patent implements dynamic encoding where the system adapts its encoding behavior based on real-time load conditions. Encoding operations are not performed rigidly according to a fixed schedule or central directive, but dynamically adjusted based on each zone's current state. This dynamic approach allows the system to achieve load balancing without complex control mechanisms, as encoding naturally occurs when and where capacity is available.
Data Source
AI summary
A method for creating distributed erasure coding chunks in a distributed storage system with unbalanced load is disclosed. The operations comprise configuring the distributed storage system into at least k+m zones, wherein each zone accumulates at least l primary backup chunks of original data chunks replicated from different remote zones, and wherein l<k, and preparing the distributed storage system for recovery from a failure of 1 to m zones of the at least k+m zones including, in each zone, encoding the at least l primary backup chunks to create m coding chunks using an erasure coding having parameters k+m, wherein in each zone in which fewer than k primary backup chunks have been accumulated, at most k−l predetermined virtual chunks are used to create partial coding chunks.