Storage Cluster Manifests for Erasure-Coded Segment Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster technologies face challenges in balancing the advantages of replication and erasure coding for data reliability, as replication leads to high bandwidth and storage overhead, while erasure coding incurs processing overhead and complicates segment identification and recovery after hardware failures.
Innovation Solution
A storage cluster that combines object replication and erasure coding, using a manifest file to manage segments and enable efficient reconstruction and recovery without the need for an extra control database, allowing switching between replication and erasure coding based on object properties or client instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to store digital objects, then data reliability is improved, but storage overhead and bandwidth consumption increase significantly
Solution Approach 1:
The digital object is divided into multiple segments, which are then distributed across different nodes in the storage cluster. This segmentation enables the system to use erasure coding instead of full replication, reducing storage overhead while maintaining data reliability through the mathematical properties of erasure codes that allow reconstruction from any sufficient subset of segments.
Solution Approach 2:
The system changes the storage parameter from replication factor to erasure coding parameters (k, m values). By adjusting these parameters, the system can optimize the balance between storage efficiency and data reliability, storing only k+m segments instead of multiple full replicas, thereby reducing storage overhead while maintaining the ability to recover data.
2Quantity of substance
If erasure coding is used to store digital objects, then storage footprint is reduced, but processing overhead and segment identification complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing the erasure coding parameters (k, m values) and segment identifiers alongside the encoded segments. This preliminary organization of metadata enables efficient segment identification and reconstruction without requiring complex real-time calculations, thereby reducing processing overhead during data retrieval and recovery operations.
3Quantity of substance
If erasure coding is used, then storage efficiency is improved, but recovery time after hardware failure increases due to segment identification complexity
Solution Approach 1:
The system introduces an intermediary mechanism in the form of a distributed metadata structure stored across the cluster nodes. This metadata contains segment identifiers, encoding parameters, and location information that enables rapid identification and localization of required segments after hardware failure, significantly reducing recovery time while maintaining storage efficiency benefits of erasure coding.
Data Source
AI summary
A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.