Distributed Blob Reclamation Using Erasure Code Parity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data center protection methods, such as replication and erasure codes, are inefficient and costly, especially in cloud environments where large-scale failures are common, and require significant WAN bandwidth and latency for recovery.
Innovation Solution
A distributed data storage system that uses local and global erasure codes to manage blob occupancy across multiple data centers, allowing for efficient garbage collection and space reclamation by maintaining local and global parity groups, minimizing write amplification and maintaining data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple data centers, then data protection against data center loss is improved, but storage cost and network capacity requirements increase significantly
Solution Approach 1:
The patent changes the redundancy parameter from full replication (100% duplicate storage) to erasure coding where only a portion of data needs to be stored at each location. With codes like 6+6P or 7+5P, data is divided into fragments with parity information distributed across data centers, allowing reconstruction from any sufficient subset, thereby reducing total storage capacity while maintaining data protection.
Solution Approach 2:
The patent segments data into code stripes with multiple elements distributed across different data centers. Each data center stores only a subset of the encoded data elements rather than complete copies, enabling data recovery through mathematical reconstruction from available fragments, thus reducing overall storage requirements while maintaining reliability.
2Quantity of substance
If erasure codes are used for data protection, then storage efficiency is improved compared to replication, but write performance and recovery speed deteriorate due to synchronous encoding and WAN latency
Solution Approach 1:
The patent performs preliminary local encoding of data into erasure code stripes at the source data center before distribution. This preliminary action creates all necessary code elements locally, allowing immediate local storage while only the distribution phase requires network communication. This separates the encoding computation from the network transfer, improving write performance.
Solution Approach 2:
The patent implements partial synchronous encoding where only critical metadata and coordination information require synchronous confirmation across data centers, while the actual data encoding and storage can proceed asynchronously. This partial approach maintains storage efficiency while reducing the performance impact of network latency.
3Reliability
If synchronous erasure coding is used across data centers, then data integrity is improved, but network bandwidth consumption and round-trip latency increase for every write operation
Solution Approach 1:
The patent performs preliminary local encoding and storage of erasure code elements at the source data center before distributing to other data centers. This preliminary action completes the integrity-critical encoding phase locally, eliminating the need for synchronous network round-trips for every write operation while maintaining data integrity through subsequent asynchronous distribution.
Solution Approach 2:
The patent implements partial synchronous confirmation where only essential coordination metadata requires network verification, while the bulk data encoding and storage proceeds asynchronously. This reduces network bandwidth consumption significantly while maintaining sufficient data integrity through the mathematical properties of erasure codes.
4Quantity of substance
If erasure code stripes are distributed across data centers, then storage capacity utilization is improved, but recovery time increases due to WAN data transfer requirements
Solution Approach 1:
The patent performs preliminary distribution of erasure code elements across data centers during normal write operations, so that code elements are already positioned at appropriate locations before failure occurs. This preliminary distribution eliminates the need for time-consuming data transfer during recovery, as all necessary elements are already locally available at each data center for immediate reconstruction.
Solution Approach 2:
The patent optimizes the distribution of code elements so that each data center holds specific elements that are most useful for local recovery scenarios. This local quality optimization ensures that when failures occur, the necessary recovery elements are already present locally rather than requiring WAN transfers, significantly reducing recovery time while maintaining efficient storage utilization.
Data Source
AI summary
One embodiment provides a system including a computer processor, a computer-readable hardware storage device, and program code embodied with the computer-readable hardware storage device for execution by the computer processor to implement a method that includes receiving a selection of a first blob for reclamation from a first data center. The first blob includes multiple erasure code groups. A first message is sent to a second data center indicating the first blob is to be reclaimed. A global reclamation complete message is received from the second data center. The global reclamation complete message indicates a second blob in the second data center has been reclaimed. The first data center and the second data center each maintain local blob occupancy information.