Asynchronous Parity Map for Distributed Storage Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in protecting against data center losses, particularly in cloud environments, where erasure codes suffer from high network bandwidth usage and long recovery times due to the need for synchronous processes and centralized controllers.
Innovation Solution
A data storage system with a dynamic parity map and content map structure that allows asynchronous data replication and parity computation across multiple data centers, enabling local and global protection without a centralized controller, and utilizing a buffer for temporary data storage before loading into storage units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous erasure code replication is used across data centers, then data protection against data center loss is achieved, but network bandwidth overhead and round-trip latency increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching parity data locally at each data center before it is needed for recovery. When data is written to a data center, the corresponding parity is immediately computed and stored locally rather than being computed synchronously across all data centers. This allows the system to achieve data protection without the network bandwidth overhead of synchronous replication, as the parity computation and storage happens asynchronously and locally.
2Productivity
If erasure codes are spread across data centers with centralized control, then storage efficiency improves, but recovery time increases due to WAN data transfer requirements
Solution Approach 1:
The patent applies segmentation by dividing the recovery process into local and remote components. Each data center maintains its own parity data locally, enabling immediate local recovery operations without requiring coordination with other data centers. Only when local parity is insufficient does the system reach for remote parity, minimizing WAN data transfer. This segmentation of recovery operations dramatically reduces recovery time compared to centralized erasure code approaches that require all data to be available before recovery can begin.
Solution Approach 2:
The patent applies preliminary action by pre-positioning parity data locally at each data center. This ensures that when recovery is needed, the necessary parity data is already available locally rather than needing to be fetched from remote data centers over the WAN. This preliminary local storage of parity data enables fast recovery operations while maintaining the storage efficiency benefits of distributed erasure codes.
3Reliability
If three data centers with full replication are used, then protection against concurrent failures is achieved, but storage and network capacity requirements multiply
Solution Approach 1:
The patent applies parameter changes by transitioning from full replication (where each data center stores complete copies of all data) to erasure coding (where data is divided into fragments and parity is distributed). This changes the fundamental parameter of how redundancy is achieved: instead of storing complete copies, the system stores fragments and computes parity mathematically. This reduces the storage capacity requirement from 3x the original data (full replication) to approximately 1.2-1.5x the original data (erasure coding), while maintaining protection against concurrent failures.
Data Source
AI summary
One embodiment provides a method for storing data in a data storage system. The method comprises copying a data set into a buffer in a data storage center in response to receiving a write command for the data set; updating a content map; and sending to a second data storage center a parity write command for the data set. The method continues with copying the data set into a buffer in the second data storage center; updating a second content map; and sending to the first data storage center an acknowledgement indicating the parity write command is complete. Then in response to receiving the acknowledgement, updating the first content map; moving the data set to a storage unit in first data storage center; and acknowledging the data set is protected. Finally, releasing the data set from the first buffer; and updating the first map content.