Hierarchical Erasure-Coded Storage Layout for Petabyte Resiliency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data protection mechanisms, such as RAID, are inadequate for petabyte-scale systems due to increased drive capacities and lower quality drives, leading to high storage costs and excessive wear on storage devices, while erasure coding techniques consume significant I/O processing and network bandwidth.
Innovation Solution
A hierarchical wide spreading data storage architecture that implements rateless erasure coding, segmenting data objects into fragments and distributing them across multiple storage nodes for maximum resiliency, using a front-end node to encode and select storage nodes based on storage resiliency, and segment processor units to re-encode and distribute fragments across storage media devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RAID mechanisms are used for data protection, then data redundancy is achieved, but storage costs increase and storage device wear becomes excessive in petabyte-scale systems
Solution Approach 1:
The patent segments data objects into multiple fragments and distributes them across different storage devices. This segmentation allows for more efficient use of storage resources compared to traditional RAID, reducing the total storage capacity needed for the same level of data protection and thereby lowering storage costs while maintaining reliability.
Solution Approach 2:
The patent employs erasure coding techniques that use mathematical parameters to generate redundant data fragments. By changing the redundancy parameters and distributing fragments across devices, the system achieves data protection with reduced storage overhead compared to traditional RAID mechanisms, addressing the contradiction between reliability and storage cost.
2Reliability
If erasure coding techniques are implemented to protect stored data, then storage resiliency is improved, but I/O processing overhead and network bandwidth consumption increase substantially
Solution Approach 1:
The patent performs erasure coding and fragment distribution in advance during the data writing phase. By preprocessing the data into encoded fragments and distributing them across storage devices before read operations, the system minimizes I/O processing overhead during data retrieval, thus improving productivity while maintaining storage resiliency.
Solution Approach 2:
The system distributes encoded fragments across multiple storage devices in a self-organizing manner, where each device stores a portion of the encoded data. This distributed architecture allows the system to handle read requests by retrieving only the necessary fragments, reducing network bandwidth consumption and I/O processing overhead compared to centralized erasure coding approaches.
3Reliability
If erasure coding is used to maintain storage resiliency, then data protection is enhanced, but storage devices experience excessive wear requiring frequent replacement
Solution Approach 1:
By segmenting data into fragments and distributing them across multiple storage devices, the patent reduces the workload on individual devices. This segmentation allows for more balanced I/O operations and reduces wear on each device, extending their operational lifespan while maintaining storage resiliency through the distributed fragment architecture.
4Reliability
If data is distributed across multiple storage nodes for wide spreading, then storage resiliency is maximized, but system complexity increases
Solution Approach 1:
The patent employs homogeneous erasure coding algorithms and uniform fragment distribution strategies across all storage nodes. This homogeneity simplifies the system architecture by using consistent encoding and decoding procedures throughout the distributed system, reducing management complexity while achieving maximum storage resiliency through wide spreading of data fragments.
Data Source
AI summary
Systems and techniques for managing data storage are disclosed. In some aspects, a front-end node responds to a request to write an object by dividing the object into multiple source data segments. The front-end node generates redundancy data for the multiple source data segments using a rateless erasure encoding. The front-end node associates a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment. The rateless erasure encoding further includes defining multiple segment-level fragments within each of the encoded segments. The front-end node transmits each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes. For each of the received encoded segments, the storage node generates one or more protection fragments based on redundancy data generated from the segment-level fragments and stores the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.