Hierarchical Erasure Coding for Low-Traffic Data Rebuilds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale cloud-based data centers face challenges in maintaining data durability and reliability due to hardware and software errors, leading to data unavailability, and existing solutions like replication and erasure coding often incur high network traffic and overhead.
Innovation Solution
Implementing network-efficient data durability coding by storing data fragments across multiple fault domains and using erasure coding algorithms to generate secondary fragments, allowing for efficient recovery of lost data without relying on network-wide durability coding, thereby reducing network traffic and enhancing data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network-wide data durability coding is implemented, then data reliability is improved, but network traffic increases
Solution Approach 1:
The patent segments data durability protection into two levels: (1) network-wide erasure coding that distributes data fragments across multiple nodes, and (2) node-level secondary erasure coding that protects individual fragments locally. This segmentation allows nodes to recover from certain failures independently without triggering network-wide data reconstruction, thereby reducing network traffic while maintaining data reliability.
Solution Approach 2:
The patent introduces a hierarchical dimension to data durability coding by implementing protection at both network level and node level. This multi-dimensional approach enables selective recovery operations where minor failures are handled locally at the node dimension, avoiding the need to activate the network-wide dimension, thus reducing unnecessary network traffic.
2Reliability
If data is stored across multiple nodes with network-wide durability coding, then data reliability is improved, but rebuild time increases
Solution Approach 1:
By segmenting the recovery process into node-level and network-level operations, the patent enables faster rebuild times for node failures. When a node fails, the system first attempts local recovery using secondary fragments stored at the same node. Only when local recovery is insufficient does the system escalate to network-wide reconstruction, thereby minimizing the time lost during rebuild operations.
Solution Approach 2:
The patent applies preliminary action by pre-storing secondary data fragments at each node using local erasure coding. This preliminary preparation ensures that when a failure occurs, the node can immediately attempt recovery using locally available fragments without waiting for network-wide coordination, significantly reducing rebuild time.
3Reliability
If additional parity information is stored at each node, then data reliability is improved, but storage overhead increases
Solution Approach 1:
The patent applies partial action by implementing erasure coding selectively at the node level rather than redundantly across the entire network. Each node stores secondary fragments for its assigned primary fragments, providing localized protection without duplicating the full parity information across all nodes. This partial implementation reduces overall storage overhead while maintaining adequate reliability for handling node-level failures.
Data Source
AI summary
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. In one example, this disclosure describes a method that includes generating a plurality of data fragments from a set of data to enable reconstruction of the set of data from a subset of the plurality of data fragments; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; and generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; and storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node.