Fault-Tolerant Object Storage System for Small Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fault-tolerant enterprise storage systems are inefficient in storing and recovering small objects due to inefficient use of parity data, often requiring the same amount of space as the object and lacking the ability to synthesize parity for small objects, leading to reduced storage efficiency and inability to recover small objects in case of data loss.
Innovation Solution
A fault-tolerant enterprise object storage system that synthesizes parity data for small objects, distributing them across multiple storage entities and parity storage entities, and updates a processed data end offset indicator to ensure recoverability, allowing efficient storage and recovery of small objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional storage systems store small objects using numerous data blocks across multiple storage entities, then fault tolerance is provided, but storage efficiency deteriorates due to inefficient use of parity data and excessive storage space consumption
Solution Approach 1:
The patent segments the parity data update process into incremental updates based on processed data end offsets. Instead of treating the entire aggregate object as a single unit, the system divides parity updates into manageable segments corresponding to individual small objects or groups of objects, allowing efficient storage and recovery operations.
Solution Approach 2:
The system performs partial parity data updates only for the portions of aggregate objects that have changed. By updating parity data incrementally based on processed data end offsets rather than regenerating entire parity blocks, the system avoids excessive computation and storage operations while maintaining fault tolerance.
2Reliability
If parity data is stored across multiple storage entities for fault tolerance, then object recovery is enabled, but device complexity increases due to the need to manage and coordinate multiple storage entities and parity blocks
Solution Approach 1:
The storage entities serve multiple functions: they store both data blocks and parity blocks, and can participate in both normal write operations and recovery operations. The same storage entities that store aggregate objects also store the parity information needed for recovery, eliminating the need for separate dedicated parity storage systems.
Solution Approach 2:
The processed data end offset acts as an intermediary mechanism that coordinates between data writing operations and parity update operations. This offset tracker mediates the synchronization between data blocks and parity blocks across multiple storage entities, simplifying the coordination complexity.
3Reliability
If traditional systems use the same amount of parity data space as object size for small objects, then fault tolerance is achieved, but storage efficiency deteriorates due to disproportionate parity overhead for small objects
Solution Approach 1:
The system changes the parameter of parity data allocation from being object-size-proportional to being aggregate-object-proportional. By pooling multiple small objects into a single aggregate object with shared parity data, the effective parity overhead per small object is reduced dramatically, improving storage efficiency while maintaining fault tolerance.
Data Source
AI summary
Various implementations disclosed herein provide fault-tolerant enterprise object storage system that can store small objects. In various implementations, the fault-tolerant enterprise object storage system writes a small object into an aggregate object that is distributed across a plurality of storage entities. In some implementations, the small object is at least an order of magnitude smaller than the aggregate object, and the small object is within the same order of magnitude of a block unit addressable within each of the storage entities. In some implementations, based on the small object, the storage system updates the parity data associated with the aggregate object in response to writing the small object into the aggregate object. In various implementations, the storage system updates a processed data end offset indicator that indicates that the parity data for the aggregate object includes valid data up to and including the small object.