Hierarchical Distributed Object Storage with Erasure Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed object storage systems face challenges in efficiently managing redundancy and dynamic configuration, particularly in large-scale systems, where adding or removing storage elements requires extensive static configuration and are not well-equipped to handle concurrent failures beyond two storage elements without significant storage overhead.
Innovation Solution
A distributed object storage system with a hierarchical configuration that uses erasure encoding to split data objects into redundant sub-blocks, allowing for efficient storage and retrieval across multiple storage elements, and employs a spreading policy based on unique storage element identifiers to manage redundancy and failures dynamically, enabling easy addition or removal of storage elements without compromising efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to achieve redundancy, then data reliability is improved, but storage capacity overhead increases significantly
Solution Approach 1:
The data object is divided into multiple sub-blocks, which are then encoded and distributed across storage elements. This segmentation allows the system to achieve redundancy without replicating entire data objects, thereby reducing storage overhead while maintaining reliability.
Solution Approach 2:
The system changes the parameter of data representation by encoding data into sub-blocks with specific redundancy properties. By adjusting the encoding parameters, the system can control the balance between reliability and storage overhead, achieving better efficiency than traditional replication.
2Quantity of substance
If RAID systems are used to reduce storage overhead, then storage capacity efficiency is improved, but system flexibility and adaptability deteriorate
Solution Approach 1:
The system dynamically adapts to changes in storage elements without requiring static configuration. When storage elements are added, removed, or replaced, the system automatically reconfigures the distribution of sub-blocks, maintaining both efficiency and flexibility unlike traditional RAID systems.
Solution Approach 2:
The distributed object storage system serves multiple functions: it provides redundancy, handles dynamic configuration changes, and maintains data accessibility. The system can tolerate various failure scenarios and adapt to different storage element types, making it universally applicable and highly flexible.
3Quantity of substance
If the number of storage elements increases to improve scalability, then storage capacity is improved, but the probability of failure increases
Solution Approach 1:
The system continuously monitors the state of storage elements and dynamically adjusts the distribution and recovery of data sub-blocks. When failures occur, the system detects them and automatically redistributes data to maintain the required reliability level, enabling scalable expansion without proportionally increasing failure risk.
Solution Approach 2:
The system pre-encodes data into redundant sub-blocks that can withstand a predetermined number of failures. This beforehand cushioning ensures that even as storage capacity scales up and failure probability increases, the system maintains reliability by having pre-prepared redundancy mechanisms in place.
4Reliability
If extensive static configuration is used to manage redundancy, then reliability is improved, but configuration overhead and complexity increase
Solution Approach 1:
The system performs self-configuration and self-management of redundancy without requiring extensive manual setup. When storage elements are added or removed, the system automatically recalculates and redistributes data sub-blocks, eliminating the need for manual configuration and reducing operational complexity while maintaining reliability.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The invention concerns a distributed object storage system (1) that comprises a spreading module (410) which makes a selection of a number of storage elements (300) being larger or equal to a desired spreading width by means of a spreading policy that comprises rules for selecting storage elements (300) in function of a first hierarchy level identifier (70.1) and/or a second hierarchy level identifier (70.2).