Storage Cluster Manifests for Erasure Coding and Disk Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster technologies face challenges in balancing the advantages of replication and erasure coding for data reliability, as replication leads to high bandwidth and storage overhead, while erasure coding incurs processing overhead and complicates segment identification and recovery after hardware failures.
Innovation Solution
A storage cluster that combines object replication and erasure coding, using a manifest file to manage segments and enable efficient reconstruction of digital objects, allowing for seamless switching between storage formats and formats without the need for an extra control database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to store digital objects, then data reliability is improved, but storage overhead and bandwidth consumption increase
Solution Approach 1:
The system dynamically changes the storage parameter from full replication to erasure coding based on object size, converting between different redundancy mechanisms to optimize the balance between reliability and storage efficiency
Solution Approach 2:
Large digital objects are segmented into multiple chunks that can be distributed across the storage cluster using erasure coding, allowing reliable reconstruction without storing complete duplicates of the entire object
2Quantity of substance
If erasure coding is used to store digital objects, then storage footprint is reduced, but processing overhead increases
Solution Approach 1:
The system adjusts the erasure coding parameters (k, l values) based on object size and access patterns, reducing processing overhead for small objects while maintaining storage efficiency for large objects
3Quantity of substance
If erasure coding is used to store digital objects, then storage efficiency is improved, but segment identification and recovery complexity increases
Solution Approach 1:
A manifest file serves as an intermediary that stores metadata about erasure-coded objects including chunk identifiers and recovery information, simplifying the identification and recovery process without requiring complex direct tracking of encoded segments
4Ease of operation
If a separate control database is used to track erasure-coded segments, then segment identification is improved, but system overhead and availability concerns increase
Solution Approach 1:
The manifest file is stored within the distributed storage cluster itself rather than in a separate control database, merging the metadata storage with the data storage system to eliminate single points of failure and reduce system overhead
Data Source
AI summary
A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.