Local Parity Coding for Reliable Storage With Lower Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale cloud-based data centers face challenges in maintaining data durability and reliability due to hardware and software errors, leading to storage system unavailability, which existing solutions address inadequately, especially in scale-out systems where node failures require network-wide data rebuilding, increasing traffic and latency.
Innovation Solution
Implementing network-efficient data durability coding by fragmenting data across multiple fault domains and using erasure coding algorithms to generate parity fragments, allowing for data reconstruction without relying on network-wide durability coding, thus reducing network traffic and enhancing data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network-wide data durability coding is implemented to protect against node failures, then data reliability is improved, but network traffic increases due to data rebuilding operations
Solution Approach 1:
The patent segments data durability protection into two independent levels: (1) network-wide erasure coding that distributes data fragments across multiple nodes, and (2) node-level local parity data that provides independent protection against storage device failures. This segmentation allows each level to handle specific failure scenarios without requiring network-wide operations, thereby reducing unnecessary network traffic while maintaining comprehensive data reliability.
Solution Approach 2:
The patent implements preliminary action by pre-generating and storing local parity data at each node before any failure occurs. This local parity data is created in advance using erasure coding algorithms, enabling nodes to immediately reconstruct lost data from local resources without needing to retrieve data from other nodes across the network, thus eliminating the need for network-wide data rebuilding operations.
2Reliability
If network-wide data durability coding is used to ensure data access after node failure, then data reliability is improved, but rebuild time increases due to network operations
Solution Approach 1:
The patent divides the data protection mechanism into network-wide fragment distribution and node-level local parity storage. This segmentation enables fast local reconstruction from parity data stored at the same node, avoiding time-consuming network operations for common failure scenarios while maintaining the ability to handle more complex failures through network-wide erasure coding when necessary.
Solution Approach 2:
The system performs preliminary action by pre-computing and storing local parity data at each node before failures occur. This advance preparation allows immediate data reconstruction from local resources, dramatically reducing rebuild time compared to network-wide operations that would require data retrieval and transmission across the network infrastructure.
3Reliability
If erasure coding is implemented across the entire network to protect against failures, then data reliability is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent segments the erasure coding implementation into two layers: network-wide erasure coding that distributes data fragments across multiple nodes, and node-level local parity generation that creates redundant data locally. This segmentation ensures that parity data can be reconstructed from local storage without requiring network bandwidth, eliminating the bandwidth consumption associated with network-wide parity operations while maintaining data durability through the combination of both layers.
Data Source
AI summary
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. In one example, this disclosure describes a method that includes generating a plurality of data fragments from data to enable reconstruction of the data from a subset of the plurality of data fragments; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; and generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; and storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node.