Multi-layer Encoding for Distributed Storage of Small Data Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies when handling small data objects, as grouping them into container objects leads to increased computing resource usage and bandwidth consumption, especially during operations like compaction and repair, due to the need to process entire container objects even when only a few small data objects are accessed or modified.
Innovation Solution
Implementing multi-layer redundant encoded distributed storage with hierarchical storage levels, using spreading divisors and redundancy levels to split data objects into multiple pieces across storage entities, allowing for efficient retrieval and redundancy without relying on entire container objects, thereby minimizing bandwidth usage and computational expense.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If small data objects are grouped into container objects for distributed storage, then storage efficiency is improved, but computing resource usage and bandwidth consumption increase during operations like compaction and repair
Solution Approach 1:
The patent segments the container object into multiple chunks, where each chunk contains a portion of the data objects. This allows operations to be performed on individual chunks rather than the entire container object, reducing computing resource usage during compaction and repair operations while maintaining storage efficiency benefits of grouping.
2Quantity of substance
If small data objects are grouped into container objects, then storage cost is reduced, but IOPS requirements increase for accessing individual data objects
Solution Approach 1:
By dividing the container object into multiple chunks, the system enables parallel access to different data objects stored within the same container. Multiple IOPS operations can be performed simultaneously on different chunks, improving throughput and reducing the impact on individual IOPS performance while maintaining lower storage costs through grouping.
3Reliability
If entire container objects are processed for individual data object operations, then data integrity is maintained, but bandwidth usage increases
Solution Approach 1:
The patent extracts and processes only the specific chunk containing the target data object during operations like compaction and repair, rather than processing the entire container object. This extraction approach maintains data integrity by working with complete chunk units while significantly reducing bandwidth usage by limiting data transfer to only the necessary portions.
4Speed
If small data objects are stored individually in distributed storage, then access speed is improved, but storage overhead and metadata consumption increase
Solution Approach 1:
The patent merges multiple small data objects into a container object that is then divided into chunks. This combining approach reduces metadata overhead by having a single container metadata structure instead of separate metadata for each small data object, while still maintaining relatively fast access speeds through the chunked structure that allows efficient locating and retrieval of specific data.
Data Source
AI summary
An object is divided into SD1 first-level pieces. Each first-level piece is stored in a first-level container on a first-level storage entity. A redundant encoding of the first-level containers is stored in RL1 additional first-level containers on RL1 additional first-level storage entities. On each of the first-level storage entities, the locally-stored first-level container is divided into SD2 local second-level pieces. Each second-level piece is stored in a second-level container on a second-level storage entity of the specific first-level storage entity. Each first-level storage entities contains SD2 plus RL2 second-level storage entities. A redundant encoding of the second-level containers is stored in RL2 additional second-level containers on RL2 additional second-level storage entities. The stored object is retrievable where any specific RL1 ones of the first level storage entities are not available, and any specific RL2 ones of the second-level storage entities of each remaining first-level storage entity are not available.