Network-Efficient Data Durability Coding via Node-Level Parity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale cloud-based data centers face challenges in maintaining data durability and reliability due to hardware and software errors, leading to data unavailability and increased network traffic during recovery processes.
Innovation Solution
Implementing network-efficient data durability coding by storing data fragments across multiple fault domains and using erasure coding algorithms to regenerate lost data without relying on network-wide durability coding, thereby reducing network traffic and enhancing data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network-wide data durability coding is implemented, then data reliability is improved, but network traffic increases during rebuild operations
Solution Approach 1:
The patent segments data durability operations into two levels: network-wide erasure coding for distributing data fragments across multiple nodes, and node-level local parity coding for protecting individual nodes. This segmentation allows each level to handle specific failure scenarios independently, reducing the need for network-wide rebuild operations and thus reducing network traffic while maintaining data reliability.
Solution Approach 2:
The patent implements preliminary action by pre-generating and storing local parity fragments at each node before actual data loss occurs. When a storage device fails, the node can immediately reconstruct lost data using these pre-computed local parity fragments without needing to contact other nodes or generate parity data during the rebuild process, thereby eliminating network traffic for parity generation.
2Reliability
If data is stored across multiple fault domains, then data reliability is improved, but system complexity increases
Solution Approach 1:
The patent divides the storage system into multiple fault domains (network-wide level and node-level), each with its own durability mechanism. This segmentation allows the system to achieve high reliability through multiple layers of protection while keeping each layer's complexity manageable and independent.
Solution Approach 2:
The patent implements a universal data durability framework that operates at multiple levels simultaneously. The same erasure coding methodology is applied both network-wide and at individual nodes, creating a multi-functional system that handles different failure scenarios with a unified approach, reducing overall system complexity despite the multi-level architecture.
3Loss of time
If local parity fragments are generated and stored at each node, then rebuild time is reduced, but storage overhead increases
Solution Approach 1:
The patent applies partial action by generating and storing only the necessary local parity fragments at each node, rather than implementing full redundancy. The local parity coding is designed to protect against single-device failures at each node, providing sufficient protection for the intended use case while minimizing storage overhead. This partial protection approach achieves fast rebuild times without the excessive storage costs of full replication.
Data Source
AI summary
This disclosure describes techniques that include implementing network-efficient data durability or data reliability coding on a network. In one example, this disclosure describes a method that includes generating a plurality of data fragments from data to enable reconstruction of the data from a subset of the plurality of data fragments; storing, across a plurality of nodes in a network, the plurality of data fragments, wherein storing the plurality of data fragments includes storing the first fragment at a first node and the second fragment at a second node; and generating, by the first node, a plurality of secondary fragments derived from the first fragment to enable reconstruction of the first fragment from a subset of the plurality of secondary fragments; and storing the plurality of secondary fragments from the first fragment across a plurality of storage devices included within the first node.