Distributed Object Storage Self-Healing via Erasure Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed object storage systems face inefficiencies in self-healing and redundancy management, leading to data loss and high computing and network bandwidth consumption, especially when handling concurrent failures of storage elements.
Innovation Solution
A distributed object storage system with a self-healing mechanism that uses erasure encoding and a maintenance agent to efficiently retrieve and store redundant sub-blocks, allowing data objects to be decoded from any combination of sub-blocks, thereby maintaining data integrity and reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to achieve redundancy, then data reliability is improved, but storage capacity overhead increases significantly
Solution Approach 1:
The data object is divided into multiple encoded sub-blocks that are distributed across different storage elements. This segmentation allows the system to achieve redundancy without requiring complete copies of the entire data object, thereby reducing storage overhead while maintaining reliability.
Solution Approach 2:
The system changes the parameter of data representation by using erasure encoding to transform the original data into encoded sub-blocks. This parameter change enables the system to achieve the same reliability level with fewer storage resources compared to traditional replication.
2Quantity of substance
If RAID systems are used to reduce storage overhead, then storage capacity efficiency is improved, but system complexity and synchronization requirements increase
Solution Approach 1:
The system employs self-healing functionality that automatically detects and repairs data loss without requiring complex synchronization protocols. The erasure encoding enables automatic reconstruction of lost data from remaining sub-blocks, eliminating the need for manual intervention or complex coordination between storage elements.
3Reliability
If self-healing functionality is added to restore redundancy, then data reliability is improved, but computing power and network bandwidth consumption increase
Solution Approach 1:
The system performs preliminary encoding of data into erasure-coded sub-blocks before storage, which enables efficient self-healing operations. When data loss occurs, the pre-encoded structure allows for targeted retrieval and reconstruction of only the necessary sub-blocks, minimizing computing and network resource consumption during repair operations.
4Quantity of substance
If the number of storage elements increases to expand storage capacity, then storage scalability is improved, but the probability of concurrent failures increases
Solution Approach 1:
The system incorporates erasure encoding that provides a cushion against failures by creating redundant encoded sub-blocks. This beforehand cushioning allows the system to tolerate concurrent failures of multiple storage elements while maintaining data integrity, enabling scalable expansion without proportionally increasing failure risk.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The invention concerns a distributed object storage system (1) comprising a maintenance agent (740) instructing an encoding module (400) to disassemble a repair data object into a specific number of repair sub blocks, this specific number being equal to or greater than said number of missing sub blocks and less then the predetermined number of sub blocks during a regular storage operation.