Storage Cluster Manifests for Erasure-Coded Object Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster technologies face challenges in balancing the benefits of replication and erasure coding for data reliability, as replication leads to high bandwidth and storage overhead, while erasure coding incurs processing overhead and complicates segment identification and recovery after hardware failures.
Innovation Solution
A storage cluster that combines object replication and erasure coding, using a manifest file to manage segments and enable efficient reconstruction and recovery without the need for an extra control database, allowing switching between replication and erasure coding based on object properties or client instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to store digital objects in a storage cluster, then data reliability is improved, but storage overhead and bandwidth consumption increase significantly
Solution Approach 1:
The patent changes the redundancy parameter from full replication (100% overhead) to erasure coding with configurable redundancy ratios (e.g., 50% or less overhead). The system allows dynamic adjustment of the erasure coding parameters (k, m values) to balance reliability requirements against storage overhead, transforming the fixed replication approach into a flexible parameterized solution.
2Quantity of substance
If erasure coding is used to store digital objects, then storage footprint is reduced, but processing overhead and segment identification complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the manifest file that contains all segment identifiers, locations, and reconstruction metadata when the object is initially encoded. This eliminates the need for complex real-time segment identification during recovery operations, as the manifest provides a ready-made roadmap for reconstruction, significantly reducing processing overhead during failure scenarios.
3Quantity of substance
If erasure coding is used without a manifest file, then storage footprint is minimized, but segment identification and recovery time increase after hardware failure
Solution Approach 1:
The manifest file is created and stored during the initial object encoding phase, preliminarily organizing all segment information including identifiers, locations, and reconstruction algorithms. When hardware failure occurs, the system can immediately query the manifest to identify missing segments and initiate recovery, eliminating time-consuming discovery processes and significantly reducing recovery time.
4Ease of operation
If a separate control database is used to track erasure-coded segments, then segment identification is simplified, but system overhead and availability concerns increase
Solution Approach 1:
The patent merges the control database functionality into the object metadata structure itself. The manifest file, which is part of the object's inherent data structure, contains all segment tracking information that would otherwise require a separate database. This integration eliminates the need for additional database infrastructure, reducing system overhead while maintaining ease of segment identification.
Data Source
AI summary
A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.