Distributed Storage Using Rateless Erasure Coding for Reliable Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage solutions, such as RAID and distributed storage systems, face inefficiencies in using storage capacity for redundancy and scalability, particularly in handling failures and data degradation across storage entities with varying capacities.
Innovation Solution
The method employs rateless erasure encoding using online codes to fragment data into blocks, encode them, and distribute them across multiple storage entities, allowing for efficient redundancy and recovery with a controlled level of redundancy that can be scaled, using a director entity to select storage entities and a distribution policy for optimal storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored redundantly by storing the same data multiple times on different entities, then reliability is improved, but storage capacity efficiency deteriorates
Solution Approach 1:
The data set is divided into multiple data blocks which are then encoded into multiple encoded blocks. These encoded blocks are distributed across different storage entities, allowing the system to achieve redundancy without simply copying entire data sets multiple times, thus improving storage efficiency while maintaining reliability.
Solution Approach 2:
The system uses rateless erasure encoding to generate encoded blocks where the encoding parameter k can be adjusted. By changing the encoding parameters, the system can flexibly control the trade-off between storage capacity efficiency and data recovery reliability, allowing optimal configuration based on specific requirements.
2Adaptability or versatility
If storage capacity is increased to handle future growth, then adaptability is improved, but device complexity deteriorates
Solution Approach 1:
The system employs dynamic encoding where the encoding process can adapt to different storage scenarios. The rateless encoding allows the system to dynamically adjust the number of encoded blocks generated and distributed, enabling flexible scaling without requiring complex static configuration for every possible future scenario.
Solution Approach 2:
The rateless erasure encoding algorithm serves multiple functions: it provides redundancy, enables flexible scaling, and works with storage entities of varying capacities. This universal approach simplifies system complexity by using a single encoding mechanism that handles multiple requirements rather than requiring different mechanisms for different scenarios.
3Reliability
If encoding and distribution parameters are optimized for maximum redundancy, then reliability is improved, but productivity deteriorates
Solution Approach 1:
The rateless encoding algorithm allows dynamic adjustment of encoding parameters to balance reliability and productivity. The system can adjust the encoding rate and the number of encoded blocks generated based on specific requirements, enabling optimization between data protection level and encoding speed without being locked into fixed parameter sets.
Data Source
AI summary
The present invention is directed to a method of storing a data set in a storage system. Said storage system comprises a plurality of storage entities. The method comprises the following steps. A step of forming at least one data set block from said data set using a fragmentation entity. A step of encoding said at least one data set block by means of an encoding entity, wherein said encoding entity provides a plurality of encoded blocks based on said at least one data set block at an output of said encoding entity. And a step of distributing said encoded blocks amongst at least two of said storage entities using a distribution entity for storage thereof by said storage entities. Said encoding entity uses a rateless erasure encoding algorithm based on rateless codes for encoding said at least one data set block. The invention is further directed to a storage system and a computer program product.