Hierarchical Chunking for Distributed Storage Metadata Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in efficiently managing and uploading large objects due to long delays caused by chunking and replication, especially with client and server failures, and high metadata volumes for very large objects.
Innovation Solution
Implementing a hierarchical chunking scheme that splits large objects into chunks, allowing simultaneous upload to multiple storage locations and switching to new shards if one becomes unavailable, reducing global metadata and improving upload availability and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If objects are split into chunks and uploaded sequentially to a single storage location, then upload reliability is maintained through simple error handling, but upload time increases significantly and service availability decreases
Solution Approach 1:
The patent divides the upload process into multiple parallel segments by splitting objects into chunks and distributing them across multiple storage locations (shards). Each shard can be uploaded independently and simultaneously, reducing total upload time while maintaining reliability through the ability to switch shards if one becomes unavailable.
Solution Approach 2:
The patent transitions from sequential upload (one-dimensional time progression) to parallel upload across multiple storage locations (multi-dimensional space-time distribution). This allows simultaneous writing to multiple shards, effectively adding a spatial dimension to the upload process that reduces time loss.
2Productivity
If chunks are aggregated into larger shards for operational efficiency, then storage and management efficiency improves, but metadata volume increases significantly making replication and compaction less efficient
Solution Approach 1:
The patent implements hierarchical chunking where objects are divided into chunks that are further divided into blocks, with metadata organized in a hierarchical structure. This segmentation reduces the metadata volume for each individual chunk while maintaining the ability to aggregate into larger shards for operational efficiency.
Solution Approach 2:
The patent uses a nested metadata structure where chunk metadata contains references to block metadata, and block metadata contains actual data references. This nesting allows efficient organization of metadata at multiple levels, reducing overall metadata volume while maintaining operational efficiency for large objects.
3Stability of the object's composition
If strict shard persistence is enforced during upload, then data integrity is maintained, but upload flexibility decreases and availability is reduced when shards become unavailable
Solution Approach 1:
The patent implements dynamic shard switching during the upload process. Instead of being locked to a single shard, the system can dynamically switch between available shards if one becomes unavailable. This dynamic approach maintains data integrity through proper chunk tracking while significantly increasing upload flexibility and availability.
Solution Approach 2:
The patent changes the state of shard availability from static to dynamic during the upload process. The system monitors shard availability and can change the active shard assignment based on real-time conditions, allowing the upload to adapt to changing availability parameters while maintaining data integrity through proper chunk reference management.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Placement of object replicas in a distributed storage system includes, at a first instance, opening a journal for storage of object chunks. Each journal is associated with a single placement policy. An object is received, which comprises a chunk. The object has a placement policy, and the chunk comprises a plurality of storage blocks. The blocks are stored in a journal that matches the placement policy. Global metadata for the object is stored, which includes a list of chunks for the object. Local metadata for the chunk is stored, which includes a block list identifying each block of the plurality of blocks. The local metadata is associated with the journal. The journal is later closed. The journal is subsequently replicated to a second instance according to the placement policy. The global metadata is updated to reflect the replication, whereas the local metadata is unchanged by the replication.