Storage Cluster Manifests for Erasure-Coded Object Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster technologies face challenges in balancing the overhead of replication and erasure coding, with replication consuming high bandwidth and storage while erasure coding incurs processing overhead and requires complex segment identification, especially during hardware failures.
Innovation Solution
A storage cluster that combines object replication and erasure coding, using a manifest file to manage segments and enable efficient reconstruction and recovery without the need for an extra control database, allowing switching between replication and erasure coding based on object properties or client instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to store digital objects, then reliability against hardware failure is improved, but storage overhead and bandwidth consumption increase significantly
Solution Approach 1:
The patent segments digital objects into multiple segments and applies erasure coding to these segments. Each segment can be independently stored and retrieved, allowing the system to reconstruct the original object from a subset of segments even when some segments are lost due to hardware failure. This segmentation approach achieves reliability without requiring full replication of the entire object.
Solution Approach 2:
The patent changes the storage parameter from full replication (100% redundancy) to erasure coding (partial redundancy with mathematical reconstruction). By using erasure coding algorithms, the system can tolerate a certain number of segment losses while storing fewer redundant copies, thereby reducing storage overhead while maintaining reliability.
2Quantity of substance
If erasure coding is used to store digital objects, then storage footprint is reduced, but processing overhead and segment identification complexity increase
Solution Approach 1:
The patent performs preliminary actions by pre-generating and storing manifest files that contain metadata about segment locations, identifiers, and reconstruction information. When hardware failure occurs or when objects need to be retrieved, the manifest file is already available to guide the reconstruction process, eliminating the need for complex real-time identification and search operations.
Solution Approach 2:
The patent introduces manifest files as intermediary structures that mediate between the stored segments and the retrieval/reconstruction process. The manifest file contains essential information about segment locations, identifiers, and erasure coding parameters, serving as a lookup table that simplifies the complex task of identifying and reconstructing objects from scattered segments.
3Ease of operation
If a separate control database is used to track erasure-coded segments, then segment identification is enabled, but system overhead and availability concerns increase
Solution Approach 1:
The patent merges the control database functionality directly into the manifest file, which is itself stored as an erasure-coded object within the same distributed storage system. This eliminates the need for a separate control database infrastructure, reducing system overhead and avoiding the availability concerns associated with separate control systems. The manifest file serves both as data and as the tracking mechanism.
4Quantity of substance
If erasure coding is used for small objects, then storage efficiency is improved, but processing overhead negates the footprint advantage
Solution Approach 1:
The patent applies different storage strategies to different objects based on their local characteristics, specifically their size. Large objects are erasure-coded to maximize storage efficiency, while small objects are stored using replication to minimize processing overhead. This local quality approach allows the system to optimize for the appropriate metric (storage efficiency vs. processing overhead) based on the specific object being stored.
Data Source
AI summary
A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.